Is This All a Doomed Enterprise?
Thinking in the limit might mean thinking about inevitability
What I’m trying to do with AI is indirect. I want to alert people to the fact they’re coming up to this pivotal moment in history, and help them relate to the gravity of that. I don’t know what strategies and approaches will make sense in the wake of a broader appreciation of our circumstance, but I’m more open to what happens with broad appreciation than I am to what happens without it.
In that light, my question about whether we’re all doomed is not the traditional empirical one: “Will AIs displace and make obsolete flesh and blood humans?,” but a bigger picture framing on whether a good universe is even possible in the long run. Avoiding AI extinction this century wouldn’t amount to much in the scheme of things if it was just a tiny bump on the inevitable road to a bad universe. It might simply preserve our net-negative world for a few more decades before immutable forces of competition and selection land our descendants in an even worse world. It also might not, but this is the thing to figure out.
This is all a prompt for you to look at Joe Carlsmith’s “Can Goodness Compete?“ and a related podcast here.
The question presented there is whether the universe has some predictable ultimate end state based on the types of actors and incentives that get selected for, given enough time for them to emerge and compete, and whether that end state is good.
The problem within that is whether the universe invariably selects for power and the ability to occupy matter and space. If the answer is yes, it seems like “goodness,” as we conceive of it, might not have a place in the limit. This is because any resource dedicated to goodness—things like rest, beauty, or joy—is a resource that could have been dedicated to power, control, or expansion. If you have two systems starting out with equal resources, the one that invariably chooses power and growth will, in the fullness of time, eclipse any system that chooses to allocate even a few resources towards goodness (or indeed anything that’s not power-and-growth).
I’m of at least two minds about this. The crux is whether there is such a thing as “slack.” Slack is the resources left over after you’ve done everything you need to do to secure your position from maximally power-seeking competitors. This could look like a star system-wide benevolent government where interstellar conflict is physically unviable and every strategy for overthrowing the government from within is adequately prepared for. The key is that the costs of unassailable defense and security are not total: there are still resources left over to spend on good things. It’s not obvious that this would be the case. It’s a question of offense-defense balance at technological maturity. It could just be very cheap to destroy social stability as a technical matter, where no matter how much you invest in security, there will still be gaps for virus-like destructive forces to creep in and the only stable equilibrium is perpetual destruction and predation. Kill or be killed.
The hopeful story is that there are systems and strategies that allow powerful good-seeking entities to quash entities that might contest goodness very cheaply, quickly, and efficiently, so as not to tax all of their goodness away.
The less hopeful story is that power-seeking entities are so strongly selected for and incentivized that they’ll continue to emerge and evolve until the tax on goodness grows high enough that we end up in something we’d recognize as the kill-or-be-killed world.
While I don’t have an answer to the ultimate question, I do think it’s interesting to reflect on the fact that we know there is locally counterfactual good that we can do at large scales—at least in terms of what seems familiar to us now. You really can prevent hundreds of children from dying who would have otherwise died. You can also greatly improve conditions for hundreds of thousands, perhaps billions, of farmed animals who wouldn’t have had any respite but for your actions. I happen to think the world is net negative overall because of wild animal suffering, but we are still living in a non-Malthusian dream time where we each control more resources than we strictly need to preserve ourselves. We have options to create more good even if, like AI risk, this is just a tiny detour on the inevitable path elsewhere.
Whether you can extend what’s true on the margin in the dream time to the technological limit is the core question. The immediate choice we face is whether to trade away big-on-a-human-scale opportunities in pursuit of an uncertain, but potentially vastly good future.
> The problem within that is whether the universe invariably selects for power and the ability to occupy matter and space. If the answer is yes, it seems like “goodness,” as we conceive of it, might not have a place in the limit. This is because any resource dedicated to goodness—things like rest, beauty, or joy—is a resource that could have been dedicated to power, control, or expansion. If you have two systems starting out with equal resources, the one that invariably chooses power and growth will, in the fullness of time, eclipse any system that chooses to allocate even a few resources towards goodness (or indeed anything that’s not power-and-growth).
I think this assumes a zero-sum framework. What if the maximally power-seeking strategy is mostly good as well? (I recently read The Evolution of Cooperation, which is an older game theory book that argues for cooperation being the winning strategy in iterated prisoner's dilemmas / certain kinds of multipolar traps, so that's the framework that I'm operating from.)
Hmm idk. I still think that if moral realism is true, it’s likely to be motivating, meaning that you get it in the limit (if you have rational agents, which I think you should expect - as Joe notes, the locusts are gonna need to be really smart to capture the universe). If moral realism is false/ not motivating (I never really understood what is meant by real and not motivating but whatever), it doesn’t matter very much. Nothing does. Especially after you die.
The idea Joe responds with (that weakening that motivation could be selective) just doesn’t make that much sense to me. The typical story for moral realism is that it follows from being rational (word rational is doing a bunch of work but whatever). I find it hard to conceive of a way for it to be good to be less rational in the limit — isn’t that just meta rationality? Shouldn’t that just then be the good stuff?
If you want to say, on the other hand, that moral motivation isn’t required for rationality, I think you’re going to have a really hard time explaining why the moral thing is the thing that you “ought” to do.