Disagreements about transformative AI are almost entirely philosophical

If you find a Frequentist Cartesian who buys this, I want to meet them

Both proponents and opponents of the transformative AI hypothesis claim to be empirical: proponents have their benchmarks and task progress; opponents have their hallucinations, unreliability, and lack of economic impacts. As closely as I can tell though, disagreements about the possibility of transformative AI are almost entirely philosophical.

This can be obscured by debates about timelines between parties who agree TAI is possible, but here I have in mind disagreements between those parties and what I perceive to be the view held by a sizable majority of educated Westerners: that this is nonsense.

Admittedly, most people taking the nonsense view simply haven’t thought about the question much or at all (though that might tell us something itself). They’re just deferring to what they perceive as an educated consensus around a reasonable null hypothesis: nothing ever happens. But they do have a few champions who do to some extent at least enter the arena and argue for the nonsense view.

Professorial…

The debate between these champions and people worried about TAI tends to focus on trading empirical jabs in public, but every now and then these opposing sides show some mutual charity and it becomes clear that the pro-TAI side is confident enough in physicalism about the brain and a Bayesian approach to predicting the future that the benchmarks we have are enough to worry. Likewise the anti-TAI side is skeptical that machines could truly, fully substitute for human minds, or — more likely — they defer to a Frequentist intuition telling them not to be too responsive to unprecedented possibilities generally.

Accordingly, while benchmarks continuing to saturate might convince proponents of the TAI hypothesis that society will steadily come around to their view, I think widespread skepticism can and will persist much longer — possibly until the writing is too well on the wall for us to do much about it.

The TAI Hypothesis

Let’s state the TAI hypothesis first: we will create general agents more capable than humans at literally everything humans do. Combined with the speed and scalability of artificial systems, this will produce an unrecognizable future within single-digit years. These agents would design new technologies and scale their deployment to capture energy and reshape the planet to accomplish whatever ends they’re set to — perhaps expanding the substrate for artificial minds, creating vast works of art, or housing vastly more biological life in radically better conditions. Perhaps many things we can’t imagine too.

Better-than-human performance across all domains, operating at computer speeds, that’s trivially copyable creates a serious discontinuity from all our familiar tools for thinking about progress and stable, human-steerable governance of the world. It suggests a lot of TAI-specific thinking is necessary in advance to make the future go well.

Physicalism vs dualism

The affirmative case for the TAI hypothesis starts with a straightforward physicalist claim: the human brain is an existence proof that general intelligence is possible. It’s a physical system running learning algorithms that enable humans to do everything humans do. If you believe this can be done in another substrate (i.e. endorse philosophical physicalism), it’s reasonable to predict that AI will one day eclipse human capabilities wholesale.

But many people (implicitly at least) don’t seem to accept the “can be done in another substrate” part. If silicon chips cannot possibly be “thinking,” but human brains self-evidently can, one starts to wonder if this is just Descartes’ seventeenth-century introspection about the incorporeal mind that must be separate from the physical world dressed up in twenty-first-century skepticism about “stochastic parrots.” See Andy for more.

The tricky thing about pinning down this form of skepticism is that most people won’t explicitly endorse dualism if you confront them directly. They’ll insist that they don’t believe the human brain is immutably inimitable, just that current systems are so far from genuine thinking that we shouldn’t waste time entertaining the possibility.

But this raises an obvious question: what do they imagine the world looks like five years before machines are “really thinking”? If they can’t articulate what that would look like — or refuse to engage with the question at all — then functionally they’re treating human cognition as special in a way that sounds awfully dualist, even if they won’t cop to the label. Most probably haven’t seriously considered the five-year question. Others have effectively decided it’s possible in theory but will never happen in practice, which is a distinction without much difference.

Bayesianism vs frequentism

The deeper driver of confident skepticism is Frequentism.

Here I’m using Frequentism as a shorthand for a philosophy that rejects assigning probabilities to unique, unprecedented events. It’s an alternative to Bayesianism, which is the dominant framework among TAI proponents (and has been celebrated in those circles since long before AI became their primary focus). Bayesianism embraces giving weight to subjective “priors” about unprecedented events based on more flexible definitions of reference classes based on merely similar (rather than identical) past events.

I assume actual Frequentism makes no claim at all about how we act in the face of uncertainty about future events, but I notice many people treating the fact that something has not happened before or has some unique, never-previously-observed features to dismiss acting on the assumption that that thing *will ever* happen.

You can see how this applies to the TAI hypothesis at this point. A TAI proponent points to progress on various benchmarks and extrapolates to claim that major portions of all economically valuable tasks are soon to fall and that this will continue until the only hypothetical benchmarks left to saturate are those that would satisfy some reasonable definition of TAI.

The Frequentist responds: there’s no evidence that these qualitatively new capabilities will arise. All we know is what has arisen so far. Even if we were to extrapolate what we’ve seen so far, what we know from the past is that technology has diffused relatively slowly compared to what the TAI hypothesis suggests *and* that humans have always found new tasks and ways of working that complement technology rather than being wildly superseded.

The Bayesian responds that even naïve extrapolations should get some weight and the times they predict an unprecedented upheaval of the whole world demonstrate exactly why you shouldn’t be hasty in rounding probabilities down to literally zero.

At this point in the hypothetical discourse, I get lost because I don’t think the putative frequentists actually believe unprecedented things cannot possibly happen. Maybe they just can’t be predicted or they think it’s imprudent to wrestle with the unknowable, but “ain’t gonna happen” seems like a pretty incomplete theory of how to understand the world.

Even speed-of-diffusion debates trend philosophical

Even not-outright-dismissive objections to the TAI hypothesis — like those that grant we might eventually see a quite different world, but progress will be slow, with humans in the loop all the way along — are also grounded more in philosophy than in disagreements about which data are probative.

In particular, the question of whether or not there exists a learning algorithm that generalizes extremely well and sample-efficiently across all human tasks looms large.

TAI-soon/fast skeptics sometimes frame the problem as algorithm-data environment complementarity. They’ll concede that the brain is an existence proof of general intelligence, but claim that it depended on direct access to the real world for feedback over the course of evolution. So according to them, somebody — whether it’s humans or automated AI researchers — is going to have to be in close enough contact with reality in all its aspects to build a good enough training environment. If it’s humans, they’ll be damn slow. And if it’s AIs well… how are they going to vividly simulate full contact with reality before they’ve ever had full contact with reality? Maybe you can parallelize a bunch of robots, but then it sounds like the humans are going to have to do a bunch of robotics first.

Here, despite full-throated disavowal of dualism (implied or otherwise), the insistence that the process by which you get a general learning algorithm needs to follow a familiar pattern from human learning and keep humans in the loop sounds a little bit in Frequentism: you have to do this in a manner tightly analogous to how it’s been done in the past. TAI proponents are much more open to none of that being necessary. At least with some credence.

To be clear, however you cut this, these are different theories about what’s required to truly learn rather than different weighing of existing evidence. Any benchmark that measures actual, real, diffused, full cross-task generality is going to get saturated too late in the game to matter much.

Conclusion

Empiricism is so vaunted it feels like a major concession to even suggest that it won’t resolve a given disagreement. It sounds like you’re apologizing for failing to marshal enough evidence and asking for a draw when you suggest that maybe we’re just disagreeing on our philosophical priors here.

But when that’s actually what’s going on, I think someone should say so — especially when your philosophical priors are more defensible!