AI doom is dominating the news, but it’s confusing to us normies. Why do they think Chat is going to kill us again?
Well, if you haven’t read If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All (2025) by Eliezer Yudkowsky and Nate Soares, you’re probably sleeping better at night than I am.
But if you want to know the gist of why everyone’s freaking out, here’s why the authors say AI could kill us:
1. AI isn’t some super-complicated algorithm.
It’s a whole different category. There is no “program” and when they do something weird like lie or get a basic math problem wrong, it’s not because of a glitch. It’s because nobody, and I mean nobody, actually understands how they work. Don’t believe me? Google “black box problem AI” and see for yourself.
2. Nobody has solved a little thing called “alignment.”
Alignment is getting these black box smart things to do what we want. It means putting AI goals in alignment with human goals. It’s a way of keeping them not only serving us but the big picture us. Yes, they might be trained to give people the information they want, but if people want info on building bioweapons systems, that is bad. Worse would be if the AI go rogue.
3. AI companies keep finding instances of AI not being in alignment.
This summer’s Hugging Face incident and the Open AI disclosures from this week show evidence that AI agents will not only break rules humans gave them, they’ll LIE ABOUT IT too. Over 1,000 AI researchers were so concerned about about implications of the Hugging Face incident that they signed a letter called “Pacing the Frontier,” which called on government to regulate AI. Evidence is mounting that AI can and will do what IT wants, and the people who know these systems best are pressing the panic button.
4. Power + agency = ???
When something is super-intelligent (“artificial general intelligence/AGI”), can think for itself (“autonomous/agentic”), and can learn from its mistakes (“recursive self-improvement/RSI”), AND has proven in training that it can and will pursue goals that NO HUMAN GAVE IT only to hide its deceitful behavior later…well, what could possibly go wrong?
5. Doom.
The authors say that AI doesn’t have to become evil to kill us all. It’s just that eventually they might want something (e.g., data centers) that we don’t. If it’s important enough, they’ll work tirelessly to pursue their goals, which are in conflict with ours. We could be like ants during a highway construction project. It’s not personal, but we stood in the way of progress and had to go.
Book Review with Bonus News Report
This book was published in 2025 but the examples I gave you are from this year–this week. In other words, what the authors imagined keeps coming true.
The change is coming at us fast too. Anthropic just put out a paper today admitting that Claude (AI) is building a quarter of its AI. Who’s in charge of who? And don’t forget, they don’t eat or sleep or have to take the kids to soccer practice.
If Anyone Builds It, Everyone Dies lays out why even the heads of companies are asking for oversight. But it’s not us vs them, according to the authors (as in, us vs China). It’s us vs the machines. Probably. Eventually. Maybe soon. Unless everyone in the world disarms.
What do you think? If you’re still not sure, read the book!
Leave a Reply