News

When Danielle Citron began studying online harassment more than a decade ago, her argument that online mobs and coordinated harassment constituted civil-rights violations didn’t make sense to many of her colleagues. Now, the legal scholar works with some of the most important tech companies and lawmakers on finding ways to minimize harassment, which has grown from posting standard lies and insults to revenge porn and “deep fakes.” Earlier this year, Citron received a MacArthur “genius” grant for her work in the field. She spoke with Intelligencer about what’s changed over the last few years and what hasn’t.

When did you start studying online harassment and online interactions?
I began writing about privacy generally in 2006 and then focusing on online harassment in 2007, when we first saw (or at least they were first reporting about) attacks on women from all different walks of life online. Whether it was female law students being attacked on a message board called AutoAdmit that was really popular at the time for all information on college and grad schools, to Kathy Sierra — she was a software developer; she was writing about how to write code that makes you happy. Nothing deeply controversial.

And she was targeted on her blog and then on a few other blogs with sexually threatening, sexually demeaning lies and privacy invasion. And it basically chased her from participating in a big tech conference, giving a talk, [to] then hiding out in her house and really stopping blogging, which was a big part of her public outreach of the work that she was doing. She wrote a lot of books on programming, she was a writer, she was a tech developer, and she basically retreats for like a year. There were female law students from every law school — they were targeted on AutoAdmit.

By other law students?

It was clear that it was definitely some law students, because they were saying, “I saw X-name person at the gym. She was wearing this,” and accompanied by a photo. Or, “This is where she lives. This is where she worked last summer.” And then what would follow were rape fantasies, very specific and particular. Home addresses and defamation. Accusing women of sleeping with their deans to get into law school, of having herpes, like, “Don’t sleep with that bitch. She’s sexually diseased. She has AIDS.”

So at the time, I’m writing about privacy. And all of this is under the veil of anonymity. So, Kathy is targeted by people who are writing under pseudonyms that don’t give you any sense of who they are. Whether it’s Hitler, Jesus, you know what I mean. XOXO. Names that aren’t traceable in that public way to people’s real identities.

Right. It was all like WeedLord420.

Yeah, there you go. But I was thinking of the real names that people used like Cheese Eating Monkey. I’m serious. This is in the lawsuits that two of the female law students, represented by pro bono counsel, [who] sued 39 posters. Because basically what it had become was like a cybermob bent on ensuring that these two female law students couldn’t get jobs after law school.

People would put up posts saying, “Everyone contact the hiring folks at the top 50 law firms and tell them why they shouldn’t hire this bitch. She’s a liar who scammed her way into law school, and isn’t trustworthy, and has sexually transmitted diseases,” et cetera. And people called law firms and sent emails. It was economic and personal reputational sabotage. What happens is, there was this “top-14 hottest girls in law school” list that was generated from AutoAdmit. Women were being named at all these law schools as being the hottest women in law school. And then, of course, people picked up those names and followed through and created a cybermob attack on all these women.

That’s what grabbed me through that. I write about privacy, but I also understood all this made the case for cyber civil rights, that understanding this is a civil-rights problem. The whole idea was to disenfranchise vulnerable people from making a living. And taking advantage of all the things, all these ways in which online tools are so important to our lives, and making sure that people that folks resented were vulnerable, could not take advantage of all those opportunities.

So, I made the case pretty early on that this was a civil-rights problem and I got really pooh-poohed for the idea: “You’re making a mountain out of a molehill. You’re exaggerating. It’s just boys being boys. Relax yourself.” This was from even dear colleagues at the time saying I wanted to break the internet, and it was free speech, and I have to back off.

They were even holding up free speech back then?

Oh yeah, at least for academics, my wonderful friends who pushed back. I always appreciate pushback, but the significant argument was (a) it’s not a civil-rights problem, and (b) it’s the free speech of the harassers and thou shalt not regulate it. Even though I said, “Listen, I don’t want to go outside First Amendment doctrine. These are credible threats. This is knowing defamation.” Like accusing someone of being an actual prostitute and said in a context where it’s clear they’re making a false assertion. It’s not an opinion like “She’s a slut.”

They were saying people were available for sex, and this is where they lived, and they were interested in rape fantasies. Those are provable facts. You can prove them as true or false. Even then I said, “Look, it’s a true threat. It’s defamation. It’s privacy invasion in the sense of someone’s nudes photos posted without consent. And I don’t want to tread on what’s virtually protected free speech. But if it happens offline and we regulate it, if it happens online, we should regulate it.” And so I got the answers, “The internet is different. It’s special, and thou shalt not touch it.

What has changed in the last decade or so?

Isn’t it obvious that when you say you’re going to chop someone’s head off and shove it up their cunt — excuse me, but this is literally the stuff I’m writing about or was writing about at the time — that that’s a threat? The suggestion that that’s not a true threat, I think now you’d probably say to me, “Of course that’s a true threat.” In a context in which you have no idea who it is, it seems really frightening. There’s nothing to suggest it’s a joke.

Yeah. I mean, even if it’s not a true threat, if I were a message-board moderator, I would not let it stand.

We’ve come a long way from being kind of crazy and overreacting. What has been so gratifying in the past 12 years is not only convincing companies to take this seriously and working with the safety folks at Facebook and Twitter and Microsoft, but also working really closely with law enforcers and lawmakers. I worked for two years with then–[California] AG Kamala Harris on what she called cyberexploitation. After my book came out, she read it and had me in to talk to her executive team. And then I worked with them for two years before she became a senator. And ever since then, I’ve worked with her office and many other folks on the Hill on a wide range of issues involving harassment, cyberstalking, threats.

In working with government officials, do you still get a lot of free-speech advocates?

Oh, definitely. I feel like in my work with lawmakers, we are very careful about First Amendment challenges. We think about our drafting suggestions with First Amendment concerns, values, and challenges in mind. But nonetheless, as careful as we can be in our advice, certain advocacy groups like the ACLU just say, “If you want to criminalize speech, you can’t. End of story.” Which isn’t actually true. We have like 21 crimes that we think of that are crimes involving words.

But of course, we have special ways to think about criminalization involved in terms of making sure the law is very specific and not vague. And you’ve got to think really hard about First Amendment doctrine. But it’s not off the table by any means. But that’s the role they play, and that’s an important role. Right?

Right. There’s got to be a check on it.

Of course. That is, of course I appreciate civil libertarians. You know what I mean? I work really closely with groups like EFF [Electronic Frontier Foundation]. It’s not as if I love those folks; it’s that I appreciate the work they do. But it has to be tempered, and it is, by discussion.

The internet has become a lot more centralized over the last decade or so. And Facebook and Twitter and whatnot, I mean they’ll never stop getting criticized for not doing enough about harassment, it seems. So I’m wondering how you think about megaplatforms in regard to this topic, platforms that seem almost too big to effectively regulate.

Okay. I think, gosh, the idea of they’re too big to regulate strikes me as just a false idea. I don’t think you’re suggesting, Brian, that that’s true. But —

No, but I’m saying they operate on a scale at which they’re like, “Something will always slip through the cracks.”

Oh, I see what you’re saying. Right. “At our scale, you can’t always be happy about our responses.” I’ve worked really closely with the safety people [at these companies]. It’s my sense that at that scale, you’re never going to have perfect compliance with your own rules and practices. You can always do better. My sense is that for the people I work with — Nathaniel Gleicher, or Monika Bickert, Antigone Davis, folks at Facebook who have very important jobs but they’re not the ultimate deciders, so to speak, but they are in charge of policy — is that they want to do better. They always want to do better. My sense is always we can always do better and let’s talk about how. It’s gratifying, certainly. The reason why I don’t get paid for my work is so that I can always then criticize. I’m unpaid, which allows me to, as an academic, speak my mind.

I think you’re right that the objection of “We’re never going to get it right, so don’t complain,” it’s there. But I’m not persuaded by it. The folks who are not in the C-suite, but folks who are making policy and on the ground at these companies, I think they know they’re fallible and can always do better. This is why I bother doing it. I feel if I wasn’t useful, I would just swing from the outside. But because I always feel like there’s real listening and efforts at improving at every stage, I think it’s worth helping.

Facebook, for example, is trying to create a single rule set for the entire Earth. Do you ever run into people saying, “Oh, we can’t implement that rule. Because while it makes sense in the U.S., it might not make sense somewhere else.”

And I criticize them for that approach in my scholarship. I don’t think there’s a one-size-fits-all rule for hate speech in the U.S. The definition of what we mean by hate speech and why we care about hate speech, I think, should differ depending on the context. A society that’s resilient and strong, and also one that has a First Amendment, is different from a society that’s really fragile. Whether it’s Rwanda or Myanmar, the challenges on the ground are really different.

A one-size-fits-all rule is cheaper because you don’t have to have piecemeal policy and execution of policy, but that is really wrongheaded because societies and cultures are different. I also think a huge mistake is just rolling out your product in a country where you don’t know the problems on the ground.

That’s what gave us Myanmar and the Rohingya. You just roll out your product and you don’t think that and don’t consider the possibility that there’s a really vulnerable minority that are going to be subject to ethnic cleansing. There’s burgeoning hate speech leading to violence that you don’t even know about, because you don’t have content moderators who speak the language. That’s just wholly irresponsible, manifestly so.

What’s different in the U.S. is different from Canada, is different from Mexico, is different from … right? Given how important these platforms are, how pervasive they are, and how integral they are in everyday life — meaning all aspects of our lives not just to spout off, not just the public square, but everything we do — I think it’s irresponsible to have a one-size-fits-all approach.

Do you think Facebook will ever get to that epiphany, or are they already there?

I guess it depends on the topic. There’s some things that having a one-size-fits-all [approach that] I think actually totally works. Just take nonconsensual pornography. They have a no-nudity policy anyway. So my deep worry is about people’s nude photos being used without their consent and shared on the platform. They’ve been really aggressive and wonderful about how to proactively deal with some of these problems. That is a one-size-fits-all approach tied to child pornography and exploitation. And that sort of makes sense, because the approach is largely similar across boundaries. In Wales and the U.K., in India. We say, “Okay, you don’t want your nude photo posted without consent.” This luckily falls within their anti-pornography rules.

But having a hate-speech policy, that’s different. The four biggest platforms in 2016 signed a memo of understanding with the European Commission to ban within and take down within 24 hours hate speech, understood as just speech that incites or that demeans protected groups. So, that’s pretty broad, demeaning speech. I actually want them to be much more transparent about what they’re actually doing. I feel like on hate speech, we need far more transparency for me to even have an educated guess. That’s not one of the areas that I consult with them on.

Last year, Facebook announced a tool where users could submit their nude photos to Facebook so it would index them and fingerprint them, which from an outside perspective seems insane to me, but I was wondering what your take on it was.

Let me explain why I don’t think that’s insane. I’m on Facebook’s Nonconsensual Intimate Imagery Task Force. Its a group of folks advising them, from advocacy groups including the National Network to End Domestic Violence, CCRI, and others. The reason why it’s not crazy is because what we hear from victims of nonconsensual pornography is so often, people threaten them with the posting. They haven’t yet done it, but the person will say, “I’m going to post this on Facebook unless you go out on a date with me.”

It’s a great relief to victims that victims can provide an image that Facebook then hashes, and it will help prevent it being reposted on the platform. [Hashing is a computer function that turns a file into a unique, algorithmically generated string of letters and numbers. A file with a hash identical to a submitted photo could not be uploaded to Facebook.] Now, at the time that this was announced, there were really smart computer-security folks who were worried that this could lead to theft or leaks. That then undermines the project. But they’ve been working really hard, from my understanding, on those issues involving security to make this process work. Once you’ve hashed an image, you can’t reengineer it back from the hash. So the database of hashes isn’t what’s vulnerable, it’s the initial uploading to Facebook.

The reason why we helped them with it is because actually victims were coming to us asking advocates for help. It didn’t come from Facebook, like, “This is this great idea we have.” It really was advocacy groups saying to Facebook, “This is what victims are saying.” It’s not total relief of your worries, but at least it takes something off your plate to worry about as a victim.

Do you know if the tool is getting a lot of use?

It’s getting use. Do I know the actual numbers right now? I don’t know. I did at some point know that people were using it. It really was an effective policy and people were taking advantage.

Good. Yeah, I just remember seeing that initial announcement. And from a privacy standpoint, given Facebook’s reputation, it seems like it’s dicey. I was like, “Hmmmm.” But it’s good to know it’s actually working as intended.

Right, and privacy folks like myself were part of the conversation. I’ve written about that market response as a pro-social, on balance, thing. I wrote a piece called “Sexual Privacy” in the Yale Law Journal that just came out, and is the start of my next book, about how market responses have promise. Not just law, but market responses. That project was one I pointed to as a helpful response.

Have you heard from other platforms about similar systems or information sharing?

That’s an interesting point, because they do it with the hashes of extremists in terrorist material, these major platforms. Not just the four dominant ones, but many more have subscribed to the database of hash of determined terrorist images that they’ve proactively filtered or removed. And I have not heard of a similar project with regard to NCP [nonconsensual pornography]. It doesn’t mean it’s not happening, I just don’t know about it. It would be important, I think.

Have you been following what’s been going on with Representative Katie Hill?

I think what is so distressing about it is that, of course, it lines up with everything that I’ve been studying, which is the targeting of a woman, shaming her for her sex and sexual relationship, and then have it cost her probably more than people in more privileged spots would have. We have a president who has been accused of rape who is above the law. And a lot of folks who have been accused of far worse nonconsensual touching and assault that just proceed as if, “Okay, I can handle and withstand these kinds of allegations,” and continue on as lieutenant governor, continue on as senator, as representative.

What I think what is so depressing is that not only is it an ex-husband who wants to really destroy the life and career of his ex-wife — it’s worked. He’s also shamed and embarrassed and hurt, harmed really reputationally in all sorts of ways, that young staffer, the young campaign worker that he was also having a relationship with. So, who loses are these few victims of nonconsensual pornography. It’s true they [news sites that posted the photos] blacked out the nipples, but it’s an egregious violation of sexual privacy no matter what. You’re exposing someone’s body in a way that shows them in the most intimate moments in which they did not consent.

Do you think it’s illegal for those news outlets to have posted those photos?

There are laws on the books in California and D.C. In California, you need to show an intent to cause emotional distress. You have to make out a case that the husband, beyond a reasonable doubt, intended to cause emotional distress.

All of these statutes have exemptions for matters of public interest. And so, as to the husband, is he doing it as a matter of public interest? Absolutely not. As to the journalists, there are, of course, arguments that the matter of public interest is what makes it newsworthy is that here we are, we have someone running for Congress in a relationship with someone who has less power, someone working on her campaign.

But the truth of the matter is there are texts that speak to their relationship, and it was seen as totally consensual. “Do we need to publish the photos?” is a question that Mary Anne Franks and I raised in one of our law review articles [“Criminalizing Revenge Porn”] on the topic of revenge porn. For it to qualify as a matter of public interest — we’re talking about the photos, not just the conversation about the relationship. There is at least an argument that it’s not newsworthy. Seeing the photos isn’t newsworthy.

But again, do I think they’re going to bring criminal charges? I would doubt it as to the publishers. I think there’s an argument on both sides as to matter of public interest. But they shouldn’t get a free pass. The truth of the matter is it seems like this is all an effort to make this female Democrat resign, and it worked. She resigned.

Are you worried about seeing it happen more?

Yes, I am. Imagine a deep-fake sex video; it’s very hard to distinguish between real and fake. Let’s say the technology is really advancing in three months, and again used against a woman running for an office, and she drops out even if it’s not real. I worry about it in all sorts of ways. It’s part of the broader civil-rights story I’ve been telling for a long time.

You also study deep fakes. I’m curious what your perspective on that is, because I feel like there’s a lot of worry about deep fakes as a tool to start World War III with fake news. But that doesn’t seem like the most effective use of them, if I’m understanding it correctly.

About a year and a half ago, I think it was Bobby Chesney, had written a series of articles on deep fakes both in California Law Review and Foreign Affairs. I testified before the Senate Intelligence Committee this summer. So we’ve done both public writing, and I’ve testified about the national security and privacy implications of deep fakes. Your question is, where should we be most concerned about the rise of deep fakes? And as a practical matter, it ties right into my broader research, which is 98% of deep fakes that are appearing online are deep fake sex videos. And 99% of deep fake sex videos involve women, and usually it’s female celebrities. Taking their faces and inserting their faces into porn, and basically making you be a sexual object in ways that you didn’t choose. There’s nothing wrong with pornography as long as you chose it yourself.

We are seeing it used, not just for female celebrities but for average people and subjecting them to humiliation, reputational harm, emotional harm. A fake sex video of the famous journalist Rana Ayyub engaged in a sex act basically was on half of the phones in India because it was spread through WhatsApp groups. And she was then beset by a cyber mob and basically couldn’t leave her home for weeks and weeks and weeks. She stopped writing. People threatened to rape and to kill her. It’s not just someone like Rana Ayyub who is provocative in the sense that she wrote about government abuses, human rights abuses, especially of Muslim minorities in India, it’s also just the everyday person appearing in a deepfake sex video and the harm and the impact is broad. It could be hard to keep a job and get a job if it’s in a search of your name. It’s terrifying, because it takes your sexual identity and exposes it in ways you didn’t choose.

But, I think I have to say Bobby Chesney and I are pretty damn worried about the 2020 election and a well-timed deep fake that could shift and tip an election. What I’ve seen, and you’ve seen it too, is politicians basically turning around and saying what’s real is fake. We call that the liar’s dividend. Like when President Trump said of the Access Hollywood tape, “That was fake.” Oh, come on. But the idea of a liar’s dividend, having deep fakes work for liars, have it work for criminals, I worry about that, too. It’s more evidence for folks who want to convince us that we live in a post-fact environment. It’s actually destruction of democracy.

I think a lot of the 2020 fear around deepfakes is someone will create one and then it’ll just go viral on its own, and everyone will be convinced by the video. I think the real issue is what you talked about, which is some sort of elite opinion maker endorses it and that gives it more mileage than it might’ve otherwise gotten.

What’s interesting is that you don’t even really need a deep fake to have that happen. You know, like the Nancy Pelosi video [which slowed down footage and audio of her to make it appear as if she was slurring her words]. And really, who’s the bug? It’s us. It’s human beings. We are so easily hacked it’s scary. We think it’s a video. We have this visceral reaction. And if it confirms our beliefs, we spread it on. If it’s salacious, forget about it. We really help it go viral. And so, even a really poor production like the Nancy Pelosi slowed down video got a lot of mileage. And if you time it well enough, and someone prominent endorses it, as you were saying, then it has great potential to shift how people behave. You can turn an election, especially if it’s timed right. It’s all in the timing, I think, for these deep fakes.

An October Surprise deep fake.

Right, it’s not that it’s my greatest concern, but I worry when you don’t have a test case. It doesn’t have to just be an election, it could be the night before an initial public offering. To change the direction of the market or change what a CEO does for a really consequential decision, it doesn’t have to be just an election. The stakes are significant societally as well as, of course, as we’ve seen with individuals.

Is there anything that the average person can do to figure out what’s a deep fake, or is it just sort of hopeless?

I don’t think it’s hopeless. A journalist with time and energy can debunk a deep fake, not based on the technology but on asking questions. “Well, where were you?” And are there facts that suggest that person wasn’t remotely near where … you know what I’m saying?

Individuals, I think we do need some real literacy and education. Before you spread information online you should to say to yourself, “This is pretty provocative. Should I pass this on? This might not be true.” We can talk to ourselves, because we’re the problem. We’re the ones making them go viral. I hope we, human beings, before we start spreading things, we think before we act. Platforms amplify, they supercharge our worst instincts. But we don’t have to let them. Nothing is inevitable in this space. Individuals are responsible for what goes on behind their computer.

Are there generational differences in internet usage? I at least want to believe that older people are more gullible than I am, but maybe that’s not actually true. I’m wondering if you’ve seen any appreciable differences there.

I don’t know if you’ve read Yochai Benkler’s new book called Disinformation, about basically the spread of misinformation or disinformation to influence campaigns and propaganda, but where bad information thrives is actually on Fox News. I mean, I’m not talking about deep-fake sex videos. I’m talking about obvious falsehoods throughout the world.

It’s more often people in the older-than-55 set. There’ve been studies about how an older and more right-wing group of folks, they’re spreading falsehoods at a rate much higher than they’re spreading accurate stories. And it’s human beings and not bots doing it. Yochai’s new book talks about how that disinformation is spread more often in right-wing echo chambers. And it really starts with Fox. It’s not that it’s coming from Facebook and these clickbait sites. It’s coming from Fox News, and then more often spread on social-media platforms by people who are, as you said, not your generation, Brian. There is an age differential. You’re not wrong about that.

Do you think there’s any way of breaking that at all?

I have this new piece in Michigan Law Review about cybermobs and death videos, disinformation. Law, and social norms, and education, and companies, they’re all part of the same conversation. None can do it all. But the fact of the matter is that people over 55 [are] spreading disinformation. We need to education them. It’s not like we have schools for it.

I worry that the people most in need of educating either are resistant or aren’t interested, and there are no passive hubs for them. Because the younger people are taught digital literacy in school and then in university. I have two college students, so I know they’re pretty skeptical readers.

Yeah. If I were an older person, I would not want to be told to go back to school, either.

I do a bunch of work with the ADL, and they are constantly discussing literacy that’s not just kids, but it’s also all about in companies. There’s a way to get education in the mix without it being so obvious. And that companies and platforms, they’re our educators as well, or they could be if they cared. And convincing them to care is tough. Why? Because their business model is clicks. The fact that we do not have strong privacy rules, that we don’t have strong limits on behavioral advertising or any disincentives in this space, is a huge part of the problem. I always think all these conversations about disinformation, we have to have a conversation about privacy regulation. If we don’t get at the business model, we’ll never end these problems.

I went to a Facebook event maybe at least a year ago -— I forget exactly when — where they were talking about what they’re doing to fight fake news or whatever. One of the things was a pop-up in the News Feed about becoming more media literate. I asked them, “How many people are actually clicking that?”

Sure. What’d they say?

They said, “We don’t have the numbers, but if you follow up, we’ll get back to you.”

Oh, please. They so have them.

I’m guessing it’s in the single or low double digits. At some point, the idea of teaching giving people the resources and expecting them to use them on their own seems a little naïve to me.

I’m not suggesting that as a cure-all, but it has to at least be part of the discussion. Their financial incentives are to keep us clicking and sharing. And if it’s the salacious and the outrageous, then there are no barriers in the laws, then they’re going to keep doing it. It’s just rational for them to escalate. I mean, from their own financial perspective it’s rational, and from their shareholders’ perspective. If we had privacy rules that restricted what we could collect, how we could use it and share it and sell it, that would change the landscape of incentives. My view of all these issues at the content layer are informed by my view of Wild West of data collection, use and sharing practices, and the conspicuous lack of regulation.

Once you get platforms to not focus on maximizing engagement, that just sort of changes the calculus of how these things work overall.

Right, yeah. Totally.

How do you feel about Section 230 [the regulation that shields platforms from responsibility for the behavior of their users]?

I’m a longtime student of and someone who has criticized 230, but with caution. Ben Wittes and I have proposed keeping the immunity but conditioning it on reasonable content moderation practices. Right now, there’s a provision of Section 230 which is about under-filtering or failing to remove. It’s really a free pass, even though the statute was designed to encourage what the statute called good Samaritans cleaning up the internet. But the statute itself doesn’t condition under-filtering on being a good Samaritan. I think we should go back to the original purpose and reintroduce that into the wording and say, “Sure, you get the immunity so long as writ large you engage in reasonable content moderation practices.” There are like now four sites that basically their business model is deep-fake sex videos. Their business is such that people upload deep fake sex videos, and then they make the advertising money. Should they enjoy immunity from liability? Absolutely not.

Think about 8chan having a strong sense that there were people on that site, totally unmoderated, who are inciting violence against vulnerable groups. And very specifically, like calling on other people to kill Hispanics and Jews and whatever, and leading to violence. This idea that you say, “I don’t do anything. Whatever happens is too bad, so sad,” is absurd.

My last question for you is, you recently won a MacArthur “genius” grant, so what are you going to do with that?

I’m going to write my next book. The timing could not have been better. I was in the midst of thinking about, I want to write a book about sexual privacy. My plan for the summer and for next year was to write in earnest. And I needed just really the time to do it. I started a new job at Boston University Law School this semester, and I’ve been teaching a lot. I always give talks every week, and so I’m on the road or I’m teaching, and I haven’t really had much time for writing. So, what’s great is that the grant frees me up to write next year. I can take some time off from teaching. Still retain my job, but buy myself out, so to speak. So, I’m so excited. I’m so honored I don’t even know what to say, really. This sort of thing happens and you’re like, “Are you kidding? What?” It’s amazing.

Yeah, it’s cool. I would love to win one of those.

Yes. I feel like if I can, anyone can. It’s really cool.

This interview has been edited and condensed for clarity.

Comments are closed.