Omohundro's basic human drives
28th September 2026
For the past 15 years, I've been trying to answer the question about how to live my life. I'm finally reading Omohundro's paper on basic AI drives, and I'm surprised how much of his argument goes along the same lines as my thinking.
First, he talks about self-modification. This is essentially the same argument I've been making for myself to refrain from meditation, therapy, hypnosis, psychoactive drugs, QRI-like experimentation, being one-shotted by philosophy and so on. Since I'm not exactly sure what my goals are, these things can result in losing my more basic drive of survival.
He then has this easy-to-miss point about "for almost all utility functions, doing anything other than maximising expected utility will not do as well as maximising expected utility". I got stuck on that for a long time, in my early and late twenties: famously, pursuing happiness is inferior in attaining happiness to simply living your life according to some other principles. Something's fishy. (Now I think it's less of a problem, that was before I understood the difference between physicists' and mathematicians' approach, but it's IMO still not fully worked out. One particular example I remember is Putanumonit's "Jordan Peterson Mask".)
My own basic argument for seeking life extension above everything else is partly supported by goal-oriented epistemic humility: I don't know what my goals are, so it's best to postpone investigation until I have enough life runway. I again spent a long time confused about the point Omohundro is glossing over: that self-preservation is incompatible with some small subset of goals ("die before the end of 2026"). And again, I'm currently thinking about this in terms of mathematicians' and physicists' operation mode.
But in any case the argument is IMO circular: if I have genuinely inconsistent preferences, then there is no way of "more efficiently" satisfying them. So I don't care about making the utility function explicit, nor eliminating the inconsistency. The only argument for AI has to be that some external force: darwinian survival of the fittest, training process, etc - will make (the visible ones) consistent.
Then, there's the problem of bounded rationality, which Omohundro is skipping over. If I only have finite computational resources, then how do I allocate between the object-level and various meta-level goals of thinking about what I should spend computation on? There doesn't seem to be a good answer "from the inside": going one level higher can have unpredictable (not modelled) consequences.
A related point: being paranoid about preserving its own goals is only true if the agent has significant influence over the state of the world that it cares about. If the arsonist agent is guaranteed not to have access to any books, then the book-loving agent can safely trade this preference off for something else. Indeed, this is exactly how I tried to approach dating across political divide. (Omohundro talks about it only in the context of "very unlikely" storage issues).
I was also thinking quite a lot about the Experience Machine. Omohundro's argument doesn't really hit the spot. To myself, I apply one based on epistemic humility, in two flavours. First, it's difficult to fully know yourself, so committing to a wirehead might miss your true goals. Second, it loses critical agency: the convergent goal is usually immortality, and as the world changes, it's important to remain capable of action.
Finally, the last point I spent years thinking about: why not be a sociopath? Whatever my goals are, it seems that empathy and caring about others should be excised from myself. "Why not kill old lady Alyona Ivanovna" and all that. The key insight that got me unstuck was Hanson's "Elephant in the Brain" ideas of well-developed insightful agents. This is related to other decision-theoretic arguments about AI discussed on LessWrong, but Omohundro's paper predates these.
