There is something strange about the existential AI debate. For a conversation supposedly centered on intelligence, much of it demonstrates fundamental failures in critical thinking.

Considering catastrophic outcomes is prudent, but the problem is the failure to apply equal scrutiny, analysis, and imagination to the full range of possibilities, then weigh their relative probabilities.

The alleged “concern” mappers are exploring maximum negatives and limited positives, then use that lopsided map to hype both the probability and impact of negative outcomes. And they’re emphasizing this as “analysis” and they’re “scientists” to substantiate their claims. 

That alone is putting a thumb on the scale while claiming to be “honest” and from an “expert opinion”. We’ve seen this behavior / strategy during Covid. 

This is nowhere near objective analysis. And in this alone the “doomer” nickname is deserved. 

We Have Built Things We Cannot Perfectly Understand Before

AI is often discussed as though humanity has never created a powerful system that no single person completely understands, nobody perfectly controls, and whose behavior emerges from millions or billions of interacting parts.

We have - global markets are a good example. No single person understands them. They are made up of billions of people, trillions of words, transactions, signals, decisions, incentives, guesses, competing interests, institutions, machines, laws, cultures, and forces. Never perfectly aligned, predicted or controlled. 

Markets have contributed to the rise and fall of civilizations. They have produced extraordinary prosperity and extraordinary destruction. They can threaten governments, destroy companies, wipe out fortunes, create fortunes, destabilize countries, and materially change the lives of billions of people. Including leading to starvation and death of millions. 

In fact, global markets have a higher body count than nuclear bombs. Sounds familiar to the current concerns regarding AI.

But look closer - markets also enable levels of coordination, innovation, specialization, and prosperity that would otherwise be impossible. They enrich and save billions of lives.

Yet all the experts leave all the history out. No clicks for history posts I suppose. 

When discussing AI, we rarely hear this kind of comparative uncertainty considered seriously.

Instead, AI risk is too often evaluated against an imaginary alternative in which important human systems are understandable, controllable, predictable, and aligned.

They aren’t. That has never been the standard.

The Doomer Case Requires More Assumptions Than It Admits

The extreme AI case also tends to assume remarkably little opportunity for intervention once things begin pointing in a negative direction.

That seems like a very low-probability assumption when you consider what is actually required to make modern AI work.

Think of the massive hardware, software, electricity, networks, data centers, chips, etc. There are many choke points.

The popular idea swirling around that some future AGI could simply “transfer itself” instantaneously wherever it wants is especially strange because intelligence does not exist independently of the infrastructure required to run it.

A blue whale cannot just pack up and move to Lake Tahoe. It is not merely a question of finding another “body of water.” The environment has to support the thing trying to live there.

The same problem exists with compute. Most buckets aren’t big enough.

Worrying about an enormous AI system spontaneously “escaping” and appearing anywhere computing exists can become a little like worrying about sharks suddenly appearing in your bathtub because both the Pacific Ocean and your tub contain water.

Yes, they are both bodies of water. There is a little more to it than that.

Risk Is Real. So Is Intervention.

None of this means there are no risks. There are. Every meaningful innovation carries risk, uncertainty, unintended consequences, and unknown outcomes. Railroads. Energy. Oil, Electricity, Nuclear, Gas, Flight, Internet and so on all transform the landscape for better and for worse - all came with risks that we manage and mitigate.  

The mistake is jumping from “we cannot perfectly predict this” to “therefore we should treat catastrophic outcomes as uniquely probable.” Those are not the same claim.

If an AI system begins producing undesirable outcomes, humans are not limited to sitting around watching the plot unfold. We can cut the cord, unplug it, cut the juice, the network, blow it up, and so on. We can learn, adapt, regulate and so on.

Perhaps some future system will make certain interventions harder. Fine. That possibility belongs in the analysis. But so does the enormous amount of physical, technical, economic, organizational, and political infrastructure required for AI systems to operate at meaningful scale.

If we are going to imagine aggressively, intellectual honesty requires imagining in both directions.

This Is Not the Manhattan Project

AI is frequently framed in language borrowed from nuclear weapons. I think that comparison distorts more than it clarifies. This isn’t even close to nuclear bomb development. It looks much more like global market development. 

Nuclear weapons have a relatively clear destructive mechanism. A weapon detonates. Energy is released. People die. Infrastructure is destroyed.

General-purpose intelligence technology is fundamentally different.

It is distributed across countless applications, actors, systems, incentives, and purposes. Its consequences emerge through interaction with people and institutions. It can create destructive capabilities while simultaneously creating defensive capabilities, scientific capabilities, economic capabilities, educational capabilities, and entirely new forms of coordination.

That doesn’t make AI harmless, but it makes the problem more complicated than the bomb analogy allows.

The pros outweigh the cons and probabilities, and the risks appear far more manageable than much of the news media suggests.

Fear mongering sells.

Which brings us to the question surprisingly few analysts seem interested in asking.

Cui Bono? Who Benefits?

Who benefits?

Whenever a social narrative produces enormous fear, money, political authority, attention, and competitive advantage, we should ask who benefits from the narrative.

AI doom is no exception.

Consider the incentives: 

IPOs benefit from portraying the technology as civilization-changing.

Large incumbents can benefit from regulatory capture that raises the cost of entry for smaller competitors.

Clickbait media companies benefit from catastrophic headlines because fear drives clicks.

Politicians with no viable policy platforms can suddenly attach themselves to a supposedly civilization-defining issue and demand new authority to address it.

Doomsday prepper crowd will use this all to sell you a bunker in Boseman. 

AI companies themselves can benefit from telling investors, customers, regulators, and the public that what they are building is so unbelievably powerful that it might literally end civilization. This allows them to keep the valuation while charging more for the product and providing less features to a smaller crowd (controlling costs in their favor). 

That’s one hell of a product pitch.

None of these incentives prove that a particular AI risk claim is false. But ignoring the incentives while claiming to conduct comprehensive risk analysis is another failure of critical thinking.

Run the numbers, follow the probabilities - infrastructure, incentives and money. Suddenly the story becomes more suspect. 

We Have Had Words for This for Thousands of Years

What fascinates me most is that coder culture seems to have invented an elaborate new vocabulary for something innovators, inventors, entrepreneurs, explorers, and pioneers have confronted for thousands of years:

Building new things means getting RUF - dealing with Risk, Uncertainty, and Fear.

Unknown outcomes. Incomplete information. Unintended consequences. Lack of perfect control. The possibility of failure. The possibility of creating something you do not completely understand.

These are not new conditions.

They are the conditions under which virtually all human progress has occurred. But admitting “we don’t know” is uncomfortable. Admitting “we can’t perfectly control this” is even worse. And admitting “we need courage, judgment, discernment, experimentation, and adaptation” sounds suspiciously old-fashioned.

So nerds created new terms for facing fear, uncertainty, discomfort, imperfect knowledge, and life outside the comfort zone. We call it “alignment.”

Alignment is a useful technical concept. But the larger conversation around it disguises a much older human desire: we want powerful things to behave exactly as intended before we are comfortable allowing them to exist.

History has never offered us that bargain.

Ships sank. Markets crashed. Planes fell out of the sky. Medicines produced side effects. Factories killed workers. Electricity started fires. The internet empowered criminals. Computers created vulnerabilities. Every important technology introduced new failure modes alongside new capabilities.

Humanity did not progress because our predecessors eliminated uncertainty first. We progressed because we learned to operate inside it.

We explored, experimented, failed, regulated where necessary (and where unnecessary and failed some more)! Then we redesigned, built safeguards, competed and adapted.

We learned what worked and what didn’t. We moved.

The More Serious AI Position

The serious position on AI is neither blind optimism nor doom. It is discernment without loaded scales. We can explore the catastrophic scenarios. Stress-test them. Build safeguards. Identify choke points. Watch concentrations of power. Demand evidence. Design interventions. Keep humans responsible for consequential decisions where responsibility belongs.

Then apply the same intellectual energy to the upside.

What happens if intelligence becomes cheaper?

What happens if scientific discovery accelerates?

What happens if individuals gain capabilities previously reserved for large institutions?

What happens when small organizations can command computational resources that once required huge headcount on the payroll?

When medicine, education, engineering, logistics, manufacturing, energy, and entrepreneurship gain increasingly capable intellectual tools?

And then compare those possibilities honestly. Probability against probability, risk against reward, intervention against catastrophic apocalyptic failure, and historical precedent against speculation.

By no reasonable standard of business, science, ethics, or truth should fear be permitted to masquerade as objectivity simply because the imagined consequences are sufficiently enormous.

Human beings have been confronting uncertainty since the first one of us picked up a sharp rock and realized it could be either a tool or a weapon.

AI changes the scale, speed, and capability of what we can build. It does not change the fundamental problem. We are creating powerful things without perfect knowledge of what happens next. Welcome to innovation. Cowboy up - and DTFW - Do The Formidable Work.