Who Should Stop Unethical A.I.?
The shadow of suspicion that now falls over much of A.I. research feels different in person than it does online. In early 2018, I attended Artificial Intelligence, Ethics, and Society, a conference in New Orleans, where a researcher presented a model that uses police data to guess whether a crime was gang-related. (I covered the event for Science.) The presenter took pointed questions from the audience about the possible unintended consequences of his research—could suspects be mislabelled as gang members?—before declaring, in exasperation, that he was just “a researcher.” Wrong answer. (“No one is ‘just an engineer’ if what you’re doing is going to result in a carceral outcome,” Hanna told me.) An audience member stormed out, reciting, in a German accent, a song about the Nazi rocket scientist Wernher von Braun: “Once the rockets are up, who cares where they come down?”
In 2018, a group of researchers wrote a blog post for the Web site of the Association for Computing Machinery (A.C.M.)—the largest computer-science society in the world, with a hundred thousand members—titled “It’s Time to Do Something: Mitigating the Negative Impacts of Computing Through a Change to the Peer Review Process.” The scholars recommended that the ethical scrutiny which was being applied improvisationally, online and in person, be systematized. “Peer reviewers should require that papers and proposals rigorously consider all reasonable broader impacts, both positive and negative,” the researchers wrote. Brent Hecht, a computer-science professor at Northwestern University and one of the authors of the post, told me that it caused “a bit of a splash.” He went on, “Folks hadn’t thought, at least in any sort of systemic way, about using their peer-review abilities in the way that we suggest.”
Tech ethicists call technologies that can be used for both good and ill “dual-use.” Pretty much all technologies are dual-use to some degree: a hammer can hit a nail or break a bone. Still, some tools, such as napalm, are better adapted for uses we might find disagreeable. The A.C.M. bloggers suggested that, when negatives appear to outweigh positives, peer reviewers should require researchers to discuss means for mitigation, perhaps through other technologies or new policies. “Computer-security conferences, interestingly, have had a history of asking for ethics statements,” Katie Shilton pointed out, when we discussed this idea. In their calls for papers, the USENIX Security Symposium and I.E.E.E. Symposium on Security and Privacy require authors to discuss in detail the steps they’ve taken, or plan to take, to address any vulnerabilities that they’ve exposed.
An occasional justification for publishing dangerous or creepy research is that sunlight is the best disinfectant; by this logic, scientists should share their work even if it’s alarming, and even when they don’t know how to mitigate harms. In 2018, a paper in the Journal of Personality and Social Psychology described an algorithm that predicts, well above chance, sexual orientation from facial photos—essentially, a kind of automated gaydar. The research is frequently cited as A.I. gone wrong. But, at the time, Michal Kosinski, a professor of organizational behavior at Stanford University and a co-author of the paper, told me that the aim of the work was to sound a warning: a repressive government could be using similar methods already. “My worry is that this is unavoidable, however offended we are by it and whatever we want to do about it,” Kosinski said. “I think the genie is out of the bottle and has been for many years, and we have to accept this and think about how to address these issues.”
Michael Kearns, who co-authored “The Ethical Algorithm” with Aaron Roth, accepts this argument—to a degree. “What I wouldn’t want to see is a steady drum beat of papers doing intrusive things of this form, and it becoming a cliché ritual justification that, ‘I’m just pointing these things out.’ ” Shilton, for her part, argues that a research paper might not be the best venue for such sensitive research: a computer scientist might “work with the media, without providing so much how-to.” Last February, meanwhile, a paper presented at the Artificial Intelligence, Ethics, and Society conference pointed out that mitigation works differently in the worlds of computer security and A.I.: the disclosure of a security vulnerability tends to benefit security experts, because software patches can be designed and deployed quickly, but in A.I. the reverse is true. Algorithms alter our social systems, not just our technical ones; it’s hard to patch a government that’s become addicted to surveillance, or a public that can no longer trust what it reads, sees, or hears.
It’s not inconceivable that I.R.B.s might play a role in shaping A.I. research. Ben Zevenbergen, a research scientist at Google, who was an academic at Oxford and Princeton University when we spoke, told me that, although I.R.B.s are prohibited from considering “long-range effects,” the consequences of new A.I. technologies may be felt quickly enough to evade such a provision. Several other scholars have suggested that the definition of a “human subject” may be flexible. If an algorithm learns to recognize faces using a public database of photos, then perhaps the people in the photos should be considered subjects. Perhaps they should be consulted, or have their consent obtained, before the research proceeds. “Obviously, researchers are incentivized to pretend that there are no human subjects involved, because, otherwise, things like informed consent become issues, and that’s the last thing you want to do when you’re processing data with millions of data points,” Zevenbergen said. Still, the objections encountered by the speech-to-face researchers, for example, might have been raised earlier, had the people behind the portraits been consulted.
Companies and governments, of course, don’t only conduct A.I. research; they also deploy the technology. For A.I. that is being used by private companies, “the natural point of enforcement would be the regulatory agencies,” Kearns told me. “But, right now, they are playing a serious game of catch-up. They don’t understand the technologies that they’re regulating anymore, or its uses, and they have no means of auditing it.” In a report for the Brookings Institution, Kearns and his co-author, Aaron Roth, proposed that regulators should be allowed to run experiments on companies’ algorithms, testing for, say, systematic bias in advertising. Reforms within academic computer science matter, but they are only part of the picture.
For now, A.I. research is mostly self-regulated—a matter of norms, not rules. “The fact that these papers do come up on Twitter nontrivially often” has made an impression, Hecht said. “The vast majority of researchers don’t want to be the subject of these types of discussions.” Last year, I participated in an online workshop organized by Partnership on A.I., a nonprofit coalition founded by several of the biggest tech firms. In the workshop, which was focussed on encouraging more responsible research in the field, we discussed alternative release strategies: sharing new work in stages, or with specific audiences, or only after risks have been mitigated. Meanwhile, an online document evoked the spectre of social opprobrium: “Visualize your research assistant approaching your desk with a look of shock and dread on their face two weeks after publishing your results. What happened?”
Still, at some conferences, new norms are being formalized. Last year, for the first time, the Association for Computational Linguistics asked reviewers to consider the ethical impacts of submitted research. The Association for the Advancement of Artificial Intelligence has decided to do the same. NeurIPS now requires that papers discuss “the potential broader impact of their work . . . both positive and negative.”
Predictably, the new NeurIPS requirement was hotly debated among computer scientists. One particular response stood out: Joe Redmon, a star graduate student who, in 2016, developed a pioneering object-recognition algorithm called YOLO (“You Only Look Once”), revealed that he had stopped doing computer-vision research altogether, because of its military and surveillance applications. (His three papers on the YOLO system have been cited more than twenty-five thousand times.) Redmon’s decision wasn’t necessarily a surprise. “‘What are we going to do with these detectors now that we have them?” he asked in 2018, in his paper on YOLOv3. “A lot of the people doing this research are at Google and Facebook. I guess at least we know the technology is in good hands and definitely won’t be used to harvest your personal information and sell it to . . . wait, you’re saying that’s exactly what it will be used for?? Oh.”
Not all computer scientists think as Redmon does. This past December, at a NeurIPS workshop, researchers presented the results of a survey about the new “broader impact” statements, conducted among their peers. Some respondents considered the new requirement a joke (“If I liked writing fiction I would be writing novels”), while others appreciated it as a chance to “reflect.” Iason Gabriel, the philosopher who leads the NeurIPS ethics-review process, told me that the statements he’s read are surprisingly good. “They actually tend to be much better quality than you would expect from a purely technical audience,” he said. (Another paper from the same workshop criticized frequent “failures of imagination.”) NeurIPS has now implemented a second layer in the review process: any reviewer or area chair can flag a paper for review by a panel of three reviewers with expertise in weighing social impact. In 2020, out of about ten thousand submissions, reviewers flagged a few dozen; four papers that were technically strong were rejected based on feedback from the ethical reviewers. Some people on Twitter protested the intrusion of ideology into engineering; it’s likely that more would have spoken up but feared backlash. One outspoken professor emeritus received praise from anonymous accounts. “Thank you for your courage standing up against the woke,” one observer tweeted.
Without a representative poll, it’s hard to quantify the community’s views. “Sometimes there’s this hypothesis within the domain of A.I. that there’s a silent majority very hostile to ethics,” Gabriel said. But “the majority reaction that I found was actually something slightly different,” he went on. “There was almost a sense of relief among a lot of these researchers—that these things could finally be spoken about, and that they weren’t just dealing with it as a kind of personal moral crisis. It’s something that’s being addressed through systemic reform.” Shilton, the chair of the SIGCHI ethics committee, concurred. “In the last ten years, I have stopped having to justify myself to computer scientists,” she said. “Instead, they say, ‘Oh, that’s an important thing to be working on,’ which is lovely and very nice. Something has happened, with Facebook and A.I. and bias and fairness and racism. That has crystalized an awareness.”
Scientists have been known to exercise caution ahead of time: in 1941, for example, researchers retracted papers they’d submitted to Physical Review on plutonium, holding them until the end of the Second World War. The American Society for Microbiology has a code of ethics forbidding research on bioweapons. But, historically, the regulation or self-regulation of science has often followed regrettable incidents. The National Research Act, which paved the way for today’s I.R.B. system, was passed in 1974, after research abuses such as the Tuskegee Syphilis Study caused public outcry. From 2002 to 2009, two psychologists worked with the C.I.A. to develop torture techniques; they later faced censure from colleagues (and a lawsuit from the A.C.L.U.). In 2014, after several lab-safety lapses, the U.S. government paused its funding for certain so-called gain-of-function research projects aimed at increasing the power of SARS, MERS, and flu viruses. (The funding later resumed.) A.I. hasn’t yet had its Hiroshima moment; it’s also unclear how such a decentralized and multipurpose field would or could respond to one. It may be impossible to align the behavior of tens of thousands of researchers with diverse motives, backgrounds, funders, and contexts, operating in a quickly evolving area. And yet, all the same, we’re now seeing rules, norms, and principles bubble up.
Hecht, who helped write the Association for Computing Machinery blog post that called for a more organized ethics process, predicts that increasing numbers of researchers, contemplating what their babies could grow up to become, will begin avoiding certain research topics. “If we’re more transparent about the impacts, it can make authors say, ‘You know, I really don’t want to be up there having a debate with the audience,’ or, ‘I don’t want to talk about how this work can be used negatively—I’m just going to do something else.’ ” He recalled an encounter he had with a young researcher presenting a poster at a conference before the pandemic. The work “had clear negative impacts that were not engaged with,” Hecht said. “And the student sort of said, ‘Yeah, I know. I don’t want to do this. I don’t want my research to succeed.’ ” Hecht laughed. “That was, um, something to reflect on.”
This post has been updated to include Ben Zevenbergen’s affiliations at the time of his interview.
Gloss