Doom sells, denial sells and nobody outside the labs really knows
The question comes up after almost every talk now: “So what do you think about all this AI leading-to-end-of-humanity?”
My answer is that I’m not sure. I want to explain why that’s the answer and not a dodge, and how eyecare providers could approach this topic.
Why it’s alarming
Start with the number. The CEO of Anthropic has said there’s a 25% chance things go “really, really badly.” A survey of AI researchers put the median probability of an extinction-level outcome at 5%, with somewhere between a third and half of respondents at 10% or higher.
If a cardiologist told you there was a 10% chance a drug would kill you, you would not take the drug. If a structural engineer put 10% risk on a bridge, the bridge would close. In every other domain where we quantify catastrophic risk, a number like that ends the conversation. So, when someone says it about AI, of course it’s alarming. Anyone who isn’t at least rattled hasn’t been listening.
Then there is the other end of the spectrum. Jensen Huang, whose chips every one of these labs runs on, said on CBS this past weekend that there’s “a 0% chance” this ends the world, that the warnings are “irresponsible,” and that the people making them “must be doing it for ulterior reasons.”
The messenger
The companies handing us the probability are the ones building the thing. At first glance, it seems like an odd thing for these companies to bring up, as they are raising it by the hundreds of billions and heading for the public markets. So the cynical readings write themselves. Fear as marketing, what Cal Newport calls “doom trolling.” A product dangerous enough to end the world must be powerful enough to justify the valuation. And the warnings of their power has happened before: OpenAI held back GPT-2 in 2019 as too dangerous to release…yet was released anyway. And the world didn’t end.
But the reading I find most persuasive is the regulatory one, because I’ve watched it work in my own field. LeCun made the charge bluntly in 2023—that the lab CEOs were “doing massive corporate lobbying” so that “a small number of companies will control AI.”
This month it got harder to dismiss. Anthropic’s CEO proposed a framework called “Pace the Frontier”—safety evaluators embedded in labs, common standards among the frontier companies, government backing for coordination—and within days it was endorsed by OpenAI, Microsoft and Elon Musk. Whatever else that is, it is incumbents writing the rules that will apply to whoever comes next.
Anyone who has worked in drug development knows this move. A regulatory burden that a company with a thousand employees can absorb is a wall to a company with ten. It isn’t always cynical, but it is always convenient for the people already inside. When the biggest players in an industry ask to be regulated, the question is never whether, but on whose terms.
I should be fair about the other side of the ledger. Huang sells the chips, and doom is bad for chip sales. LeCun just raised a billion dollars on the premise that the doomers’ architecture is a dead end. Everybody in this argument has a stake. Which is precisely why “I’m not sure” is the honest answer for anyone standing outside the labs—and why I’ve become suspicious of anyone outside them who claims certainty in either direction.
What an eyecare provider should do with this
Here is where I land and what informs the work at AI in Eye Care.
We are obsessed with one thing: using these tools to augment a clinician’s intelligence so that patients get better care. Not to replace the exam, not to chase superintelligence. But to catch the diabetic retinopathy that would otherwise go unscreened, to read the OCT the way a specialist would in a place that has no specialist, to give an optometrist the second set of eyes she doesn’t have. We think the promise there is real, and we think it is already arriving.
None of that requires settling the 10% question. The tools that will change eyecare over the next five years are narrow, testable and cleared by an agency we already know how to deal with. They can be evaluated the way we evaluate everything else: sensitivity, specificity, in a population like ours.
Cal Newport, a Georgetown computer scientist, has an apt analogy. The labs’ method—release increasingly autonomous systems into the wild and see what happens—”is the equivalent of deploying a fleet of experimental self-driving cars onto the interstate, hoping that at least one will avoid crashing long enough to win a navigation challenge.” No one would permit that type of irresponsible behavior in any other industry. We’d demand the test track, and someone independent standing on it.
My “not sure” is not a license to ignore the larger question, and here the drug analogy is worth following all the way through.
We don’t let a drug company decide on its own whether a drug is safe. So, we built an FDA. And for any clinical trial where the stakes are high, we built the independent data safety monitoring board: outsiders who see the data as it accumulates and have the authority to stop the trial before the company would choose to.
An FDA for AI
AI has no such thing. The labs evaluate their own models, publish their own safety cards and— under “Pace the Frontier”—propose to embed evaluators of their choosing, which cuts against the need for true independence.
And the case for oversight doesn’t rest on the 10%. It rests on harm that has already happened. There are lawsuits alleging that companion chatbots encouraged self-harm in minors, and an FTC inquiry into the companies that make them. There is a generation of students whose teachers can no longer tell whether the work is theirs. There are the ethical questions we’re still discovering.
So, I don’t dismiss the idea of an FDA for AI. I think it’s the obvious move, and it’s what every other powerful technology in medicine eventually got. The only requirement I’d insist on—the one the incumbents’ proposals conveniently leave out—is that it be independent in fact and not just in name: independent funding, staffed by people who don’t go back to the labs afterward, with the authority to see inside the training runs, and if needed, to stop one.
That’s not doomerism, and it’s not dismissal. It’s how clinicians have always handled a powerful intervention with an uncertain risk profile.
Read more from our Professional Co-Editors here
Author
