> This too, I am skeptical of. How can you craft a perfect continual learning algorithm without a proper diverse set of problems to learn over? That’s how evolution did it. It’s just not clear to me that the datacenter of automated AI researchers would be able to judge the relative merits of one continual learning algorithm over another without a real problem set to test it on.
Doesn’t this depend on the premise that “there will be many continual learning algos that perform equally well at learning skills in any synthetic environments you can possibly set up, but which nonetheless differ significantly in their ability to learn skills in ‘the real world’”?
This is possible I guess, but it seems unlikely to me.
Why? I think we're very far from synthetic environments that can substitute for the real world, particularly the bits of the real world that are beyond the frontier of science. We can't even simulate the worm C. elegans, which only has 302 neurons.
Great post! I’m glad someone wrote this up into a coherent view, and I think it’s an accurate model of LLMs and their near-term successors (“LLMs relate to most tasks as McKinsey does to running a company” is one of many banger lines).
I think the weakest spot in your argument — and the biggest open question here — is the implicit claim about how much real-world data you’d need to properly test your algorithms, especially continual learning algos (Brendan makes a version of this point, as did Tom Davidson in his tweet thread with Bradley).
If it takes a lot of data — the “turn the whole economy into RL environments” world that Mercor et al. imagine — then that’s great news, and the country in the datacenter won’t get very far on its own. Your argument implicitly takes this stance (though correct me if I’m wrong!), and within the LLM paradigm I think that’s a reasonable expectation.
If it doesn’t take much data — either because RL for LLMs starts to generalize, as Dario expects, or because Grok 6 can test its continual learning algos perfectly well with a couple hundred Optimus robots roaming around Colossus’s backyard for a year — then the argument is much less comforting. I’m not sure how to weigh the evidence here, but I’m not as convinced as you seem to be that we’ll end up needing the world economic RL env (particularly for non-LLM paradigms, e.g. Steve Byrnes’s Brain-like AGI).
One intuition that might differentiate our views: evolution needed lots of time and diverse problems to develop the brain’s algorithms, but we have some big advantages over evolution. Obviously we can take much more strategic actions (e.g. deciding to scale up an architecture by many OOMs within a few years), but perhaps more importantly we have a working example of a sample mega-efficient general intelligence algorithm in our brain. If (and it’s a big if!) we could reverse engineer it even partly (e.g. to where we have chimp-level brain algos), that might greatly reduce the amount of data we’d need to test our AGIs.
That said, it’s not clear at all whether that kind of reverse engineering work is on track to being automated (e.g. maybe it involves lots of neuroscience or taste/generalization), or whether hundreds of humans working on it irl have made that much progress (unclear how to measure that).
So, anecdote from earlier today: I’m trying to teach an AI to get good at a task that requires some slow, real-world input. I wanted to speed up the process, so I asked it to build a non-LLM low-fidelity software simulation of the system, and use the limited real-world data we had collected to validate that simulation. Then I started training the AI against that simulation. It worked. There are a TON of asterisks around this story — this was very much not “real cutting edge AI research in hard domains” on a number of dimensions — so I don’t regard this as proving anything, but I found it at least suggestive, and I’m not sure there’s reasons why this kind of technique wouldn’t work on many/most of the real-world domains relevant to stories about AGI.
China is roughly nine to twelve months behind the frontier, judging from the relative performance of Mythos and DeepSeek V4. If we're not far from properly initiating RSI, this gap is probably enough for American companies to get there first, and once they do, their massive inference compute advantage means the loop runs much faster than anything Chinese labs could match once they catch up to the same starting point.
But if RSI isn't enough to unlock "true" general-domain superintelligence, then getting there first matters much less for ultimately winning the race as whatever advantage the head start confers could be substantially offset by China's superior deployment capabilities, top-down enforcement, and the possibility that the multimodal/world model/embodied bet pays off (see https://www.chinatalk.media/p/how-china-hopes-to-build-agi-through for an excellent overview of China's AGI thinking). So whether a software-only intelligence explosion is possible or not turns out to be a fundamental tension underneath the China–US dynamic too, and it's striking how few of the policy people working in this space seem to be reasoning about it that way.
As a deep learning aficionado without much hands-on experience, I don't have strong intuitions about how sample efficiency evolves. To me, sample efficiency is hard to disentangle from raw intelligence or generalization (think of the kid who gets it after one example versus the one who needs a dozen, which one is smarter?)
But as far I as I understand the software efficiency improvements we've seen (even granting the meme that most of the gains come from data quality) are still working around the transformer architecture, so it's not surprising that fundamental generalization ability hasn't shifted dramatically. But it might shift under an RSI regime. Discovering things like the attention mechanism, or better RL algos (EfficientZero and its descendants), can be done fully in-silicon using metrics that aren't obviously more Goodhart-prone by a machine than by a human researcher: regret, samples-to-threshold, loss curves, and so on. The whole point of RSI, at least as I picture it, is to enable a search over architecture space that specifically targets this generalization ability. Though I'll admit I don't have a great intuition for what automated AI research actually looks like as a step-by-step process (possibly nobody does).
I suppose the takeaway of my comment is that the shape of sample efficiency might also be the variable that decides whether China or the US wins the race. Isn't that crazy?
(x-posting from EAF version since idk if you get notified there)
> How can you craft a perfect continual learning algorithm without a proper diverse set of problems to learn over? That’s how evolution did it. It’s just not clear to me that the data centre of automated AI researchers would be able to judge the relative merits of one continual learning algorithm over another without a real problem set to test it on.
Could you say more about why you don't think you could construct a "proper" diverse set of simulated problems?
I see the argument for why its hard to make simulations that match the real world, but you don't need to match the real world to develop a sample efficient learning algorithm that works in the real world. Evolution devised its learning algorithm in a massively different environment from the one we're in right now, and our learning algorithm generalized fine from things like "hunting gazelles" to things like "going to the moon".
Incidentally, I also think the focus on external deployments is a bit strange. At the moment it seems like AI companies are getting more of their task-specific data by directly working with domain experts rather than by leveraging data from their deployments. You say things like:
> this progress could unfold almost entirely within the confines of data centres, unconstrained by the rate of AI’s wider diffusion into the economy, and take the rest of the world by surprise
And while I agree it'll be important for the data centres to take inputs from the rest of the world (like data from domain experts and some practice with robotics equipment, etc) I don't think those inputs need to be bottlencked by AI's diffusion, and therefore this doesn't necessarily prevent the world from being taken by surprise.
I think there’s a separate objection here from “simulations might generalise better than you think”.
Even if an arbitrarily intelligent simulator cannot generate perfectly representative practice data for the real world, an hitherto-'undiffused' AI would not need a perfect strategy to be dangerous. It would only need a strategy good enough to beat us. The relevant threshold for loss of control may be much lower than “domain-general superintelligence which can reliably run companies, win elections, design drones, etc.” If simulated experience gets an AI much of the way there, and that already makes it vastly better than humans at some strategically important combination of cyber, persuasion, planning, replication and AI R&D, the remaining Goodhart gap may not save us.
I also wonder whether the argument puts too much weight on “practice” surviving genuine RSI. If RSI actually produces much more general intelligence, rather than just a larger bag of domain-specific capabilities, why assume the resulting system still needs extensive practice before becoming exceptionally good at unfamiliar tasks? Intelligence seems partly to consist in extracting more from prior knowledge, transferring abstractions across domains, and figuring things out from very few examples. A sufficiently strong system could therefore be unusually capable on its first attempt, then improve rapidly from a small amount of real-world feedback.
Superintelligence isn’t about producing Einsteins. It is about automating Edisons (who have the peripheral vision to synthesize ideas across different fields). AI will definitely achieve the latter, possibly never the former.
This finally puts into words a lot of the shifting thoughts and notions that have been percolating in my head for the past many months. GenAI/LLMs will be huge, disruptive, maybe destructive. Lots of reasons to work on AI safety, regulation, and policy. But *NOT* ASI x-risk.
> This too, I am skeptical of. How can you craft a perfect continual learning algorithm without a proper diverse set of problems to learn over? That’s how evolution did it. It’s just not clear to me that the datacenter of automated AI researchers would be able to judge the relative merits of one continual learning algorithm over another without a real problem set to test it on.
Doesn’t this depend on the premise that “there will be many continual learning algos that perform equally well at learning skills in any synthetic environments you can possibly set up, but which nonetheless differ significantly in their ability to learn skills in ‘the real world’”?
This is possible I guess, but it seems unlikely to me.
Why? I think we're very far from synthetic environments that can substitute for the real world, particularly the bits of the real world that are beyond the frontier of science. We can't even simulate the worm C. elegans, which only has 302 neurons.
Great post! I’m glad someone wrote this up into a coherent view, and I think it’s an accurate model of LLMs and their near-term successors (“LLMs relate to most tasks as McKinsey does to running a company” is one of many banger lines).
I think the weakest spot in your argument — and the biggest open question here — is the implicit claim about how much real-world data you’d need to properly test your algorithms, especially continual learning algos (Brendan makes a version of this point, as did Tom Davidson in his tweet thread with Bradley).
If it takes a lot of data — the “turn the whole economy into RL environments” world that Mercor et al. imagine — then that’s great news, and the country in the datacenter won’t get very far on its own. Your argument implicitly takes this stance (though correct me if I’m wrong!), and within the LLM paradigm I think that’s a reasonable expectation.
If it doesn’t take much data — either because RL for LLMs starts to generalize, as Dario expects, or because Grok 6 can test its continual learning algos perfectly well with a couple hundred Optimus robots roaming around Colossus’s backyard for a year — then the argument is much less comforting. I’m not sure how to weigh the evidence here, but I’m not as convinced as you seem to be that we’ll end up needing the world economic RL env (particularly for non-LLM paradigms, e.g. Steve Byrnes’s Brain-like AGI).
One intuition that might differentiate our views: evolution needed lots of time and diverse problems to develop the brain’s algorithms, but we have some big advantages over evolution. Obviously we can take much more strategic actions (e.g. deciding to scale up an architecture by many OOMs within a few years), but perhaps more importantly we have a working example of a sample mega-efficient general intelligence algorithm in our brain. If (and it’s a big if!) we could reverse engineer it even partly (e.g. to where we have chimp-level brain algos), that might greatly reduce the amount of data we’d need to test our AGIs.
That said, it’s not clear at all whether that kind of reverse engineering work is on track to being automated (e.g. maybe it involves lots of neuroscience or taste/generalization), or whether hundreds of humans working on it irl have made that much progress (unclear how to measure that).
So, anecdote from earlier today: I’m trying to teach an AI to get good at a task that requires some slow, real-world input. I wanted to speed up the process, so I asked it to build a non-LLM low-fidelity software simulation of the system, and use the limited real-world data we had collected to validate that simulation. Then I started training the AI against that simulation. It worked. There are a TON of asterisks around this story — this was very much not “real cutting edge AI research in hard domains” on a number of dimensions — so I don’t regard this as proving anything, but I found it at least suggestive, and I’m not sure there’s reasons why this kind of technique wouldn’t work on many/most of the real-world domains relevant to stories about AGI.
Great post. Thanks 🙏
China is roughly nine to twelve months behind the frontier, judging from the relative performance of Mythos and DeepSeek V4. If we're not far from properly initiating RSI, this gap is probably enough for American companies to get there first, and once they do, their massive inference compute advantage means the loop runs much faster than anything Chinese labs could match once they catch up to the same starting point.
But if RSI isn't enough to unlock "true" general-domain superintelligence, then getting there first matters much less for ultimately winning the race as whatever advantage the head start confers could be substantially offset by China's superior deployment capabilities, top-down enforcement, and the possibility that the multimodal/world model/embodied bet pays off (see https://www.chinatalk.media/p/how-china-hopes-to-build-agi-through for an excellent overview of China's AGI thinking). So whether a software-only intelligence explosion is possible or not turns out to be a fundamental tension underneath the China–US dynamic too, and it's striking how few of the policy people working in this space seem to be reasoning about it that way.
As a deep learning aficionado without much hands-on experience, I don't have strong intuitions about how sample efficiency evolves. To me, sample efficiency is hard to disentangle from raw intelligence or generalization (think of the kid who gets it after one example versus the one who needs a dozen, which one is smarter?)
But as far I as I understand the software efficiency improvements we've seen (even granting the meme that most of the gains come from data quality) are still working around the transformer architecture, so it's not surprising that fundamental generalization ability hasn't shifted dramatically. But it might shift under an RSI regime. Discovering things like the attention mechanism, or better RL algos (EfficientZero and its descendants), can be done fully in-silicon using metrics that aren't obviously more Goodhart-prone by a machine than by a human researcher: regret, samples-to-threshold, loss curves, and so on. The whole point of RSI, at least as I picture it, is to enable a search over architecture space that specifically targets this generalization ability. Though I'll admit I don't have a great intuition for what automated AI research actually looks like as a step-by-step process (possibly nobody does).
I suppose the takeaway of my comment is that the shape of sample efficiency might also be the variable that decides whether China or the US wins the race. Isn't that crazy?
(x-posting from EAF version since idk if you get notified there)
> How can you craft a perfect continual learning algorithm without a proper diverse set of problems to learn over? That’s how evolution did it. It’s just not clear to me that the data centre of automated AI researchers would be able to judge the relative merits of one continual learning algorithm over another without a real problem set to test it on.
Could you say more about why you don't think you could construct a "proper" diverse set of simulated problems?
I see the argument for why its hard to make simulations that match the real world, but you don't need to match the real world to develop a sample efficient learning algorithm that works in the real world. Evolution devised its learning algorithm in a massively different environment from the one we're in right now, and our learning algorithm generalized fine from things like "hunting gazelles" to things like "going to the moon".
(See also Tom Davidson's post here, especially the section on data bottlenecks to automating most real-world work. https://www.lesswrong.com/posts/icykbWYmdLFYxgvqY/data-bottlenecks-won-t-prevent-an-intelligence-explosion-but#Data_bottlenecks_to_automating_most_real_world_work )
Incidentally, I also think the focus on external deployments is a bit strange. At the moment it seems like AI companies are getting more of their task-specific data by directly working with domain experts rather than by leveraging data from their deployments. You say things like:
> this progress could unfold almost entirely within the confines of data centres, unconstrained by the rate of AI’s wider diffusion into the economy, and take the rest of the world by surprise
And while I agree it'll be important for the data centres to take inputs from the rest of the world (like data from domain experts and some practice with robotics equipment, etc) I don't think those inputs need to be bottlencked by AI's diffusion, and therefore this doesn't necessarily prevent the world from being taken by surprise.
I think there’s a separate objection here from “simulations might generalise better than you think”.
Even if an arbitrarily intelligent simulator cannot generate perfectly representative practice data for the real world, an hitherto-'undiffused' AI would not need a perfect strategy to be dangerous. It would only need a strategy good enough to beat us. The relevant threshold for loss of control may be much lower than “domain-general superintelligence which can reliably run companies, win elections, design drones, etc.” If simulated experience gets an AI much of the way there, and that already makes it vastly better than humans at some strategically important combination of cyber, persuasion, planning, replication and AI R&D, the remaining Goodhart gap may not save us.
I also wonder whether the argument puts too much weight on “practice” surviving genuine RSI. If RSI actually produces much more general intelligence, rather than just a larger bag of domain-specific capabilities, why assume the resulting system still needs extensive practice before becoming exceptionally good at unfamiliar tasks? Intelligence seems partly to consist in extracting more from prior knowledge, transferring abstractions across domains, and figuring things out from very few examples. A sufficiently strong system could therefore be unusually capable on its first attempt, then improve rapidly from a small amount of real-world feedback.
Superintelligence isn’t about producing Einsteins. It is about automating Edisons (who have the peripheral vision to synthesize ideas across different fields). AI will definitely achieve the latter, possibly never the former.
Hi Tom. Great post. Can I crosspost it to the EA Forum, crediting you, and linking to the original?
Hey vasco - sure!
Thanks. Published (https://forum.effectivealtruism.org/posts/Doew7ieJ3XACfovKs/the-goodhart-singularity).
Great article! I have made a similar point here before applied to what it means for vertical ai: https://kaiko.ai/insights-hub/betting-Machine-God
This finally puts into words a lot of the shifting thoughts and notions that have been percolating in my head for the past many months. GenAI/LLMs will be huge, disruptive, maybe destructive. Lots of reasons to work on AI safety, regulation, and policy. But *NOT* ASI x-risk.