Retiring “AGI”: Two Paths for Intelligence
Why today’s systems will transform work but will not yield human-like minds
TL;DR. “AGI” used to mean human-like general intelligence. Now many use it to mean systems that automate most economically valuable work. Those are different destinations. Path 1 is Economic Automation and it is already underway. Path 2 is Self-Aware, Continually Learning Synthetic Minds and it requires a different stack. Path 1 will not grow into Path 2. We should retire the single label and govern each path on its own terms.
The label “AGI” now hides two different goals
In the 2000s, AGI referred to a machine intelligence with human-like generality, capable of learning new tasks, transferring knowledge across domains, and adapting to novel situations. By the 2020s, most companies (OpenAI, DeepMind, etc.) use AGI to refer to highly autonomous systems that can perform most economically valuable work at or above human level. These are not the same thing. The single label obscures the technical and policy choices that matter.
Today’s large language models (LLMs) and agentic toolchains are racing ahead on economic tasks. They do not produce human-like minds. They confabulate, lack grounding, cannot learn continually from lived experience, and only “think” when prompted. My prediction: within a decade serious researchers will retire the term AGI and replace it with categories that separate economic automation from human-like cognition.
Path 1: Economic Automation
The near-term path looks like the Ship’s Computer from Star Trek: The Next Generation. We are building systems that can be asked anything, know more than any one human, and act through software tools. They have no goals or rights of their own. Their creativity is mostly recombination. Yet they change what individuals can do. With these systems, any person can command the equivalent of a large virtual workforce.
The real risk is centralization, not “takeover.” If access is open and widely distributed, this could spark a renaissance, provided our values shift toward self-improvement rather than complacency. If access is closed and concentrated, it will entrench inequality.
Education will feel it first. Enrollment is already declining. Now AI can perform much of the work that average students do. If misused, these tools inflate grades and hollow out core skills. My prediction: universities that cling to rote assignments will collapse, while those that invest in critical thinking, creativity, and leadership will endure. Universities will shift back toward cultivating minds rather than mass credentialing.
Why Path 1 will not become Path 2
A common objection is that if LLM-based agents keep getting bigger and better, they will “wake up” and become human-like. That is not how these systems work.
What an LLM is. An LLM is a very large pattern-matcher trained to predict the next token (word or piece of a word) from previous tokens. This objective is excellent for language generation. It does not create goals, motivation, or a sense of self.
Eight reasons Path 1 does not morph into Path 2
No intrinsic goals or desires. LLMs do not want anything. When they “ask” for something, that is text generation, not self-directed intent. They have no sense of self or agency. They learn from predicting the next token in documents. There is no rationale for believing that the next-token-prediction learning objective would lead to agency or “awakening.”
No agency without a leash you build. An LLM only acts if developers connect it to tools and explicitly grant permissions. The “agent loop” is your code. Remove the scaffolding and it is text-in, text-out.
No ongoing learning from experience. Base models do not update themselves as they interact. They do not accumulate memories, revise beliefs, or improve across days without external machinery. While they may appear to revise their beliefs in a given conversation, that “revision” is transitory and limited to that conversation. One can think of each interaction as simply a growing Word document, which is never internalized into the model itself (its weights).
No grounded world model. They mostly learn from text rather than from acting in the world. They have no bodies, no sensory feedback loops, and only shallow causal models tied to consequences. Current “multimodal” add-ons typically bolt visual (or audio) encoders onto a language backbone after the fact, which differs from acquiring concepts through perception and action. A helpful analogy: imagine a disembodied scholar who spends millennia reading everything ever written, accepting much of it at face value, and only ever “seeing” words. Then, it is suddenly allowed to see. Now they must map raw sights onto descriptions they once read, without the benefit of trial-and-error learning or lived consequences. That is roughly the situation of today’s LLM-plus-vision stacks.
No metacognition. They do not know what they do not know. They are literally trained to believe everything they read, such that they can predict the next most likely word. Expressed “confidence” is just more tokens. Reliable self-assessment requires additional systems.
“Talking to think,” not private simulation. They only “think” in language, and only when prompted. If you sat in a sensory deprivation tank, you would still have thoughts; an LLM would not. It produces visible tokens to facilitate step-by-step reasoning, whereas humans typically conduct most reasoning privately and then articulate their thoughts. Recent evidence shows that language is primarily a tool for communication rather than thought (Nature, 2024). This is unsurprising given that many animals, including birds and primates, can reason without language. By contrast, LLMs confine thinking to text.
What about chain-of-thought and test-time compute? These make the model talk more to think more. They extend the length of the externalized reasoning trace, but they still require prompting; they do not update the model’s internal knowledge, and they produce tokens rather than running private simulations. Even “hidden” scratchpads that are not echoed to the user are still token sequences generated by the same next-token machinery. They are not a persistent inner workspace that survives across prompts, tasks, or days, and they do not provide inhibitory control or self-critique before any representation is formed. Private simulation, silent rehearsal, and inhibition are core to planning and self-control; LLMs lack a durable latent workspace where ideas can be tried, critiqued, and discarded without producing language, and they lack a learned, stateful world model that can be rolled forward to evaluate consequences before speaking or acting.
But agents can plan and use tools. Does that change things? Tool-use frameworks wrap an LLM in a loop that sets a goal, calls tools, reads results, and iterates. That looks like planning, but the loop and its permissions are written by humans. This raises design risks (bugs, over-permissioning, jailbreaks, bad integrations), not “it woke up” risks.
The danger with LLMs is misuse, over-reliance, and centralization, not a sci-fi awakening.
What a human-like synthetic mind needs that LLMs do not provide:
Continual learning as the default so experience changes the agent and improves future learning
Latent-space thinking for private simulation, self-critique, and inhibition before acting
Embodied grounding so concepts tie to perception, action, and consequences
Metacognition to track what is known, what is not, and when to ask for help
A coherent self-model and goals that persist across time and tasks
LLMs are like an ultra-capable autocomplete with a library card and a helpful clerk. A human-like mind is closer to a scientist-designer-pilot that learns from every flight, updates internal models, simulates options, and pursues goals that last for years.
Path 2: Self-Aware, Continually Learning Synthetic Minds
Human-like AI is a different target. It requires continual learning as a core operating mode, since only continual learning builds a stable model of self and world, improves forward transfer, and supports metacognition. It also requires latent-space thinking, where the agent simulates outcomes internally, and some form of embodiment for grounded concepts.
Such agents would innovate, self-direct, and pursue their own goals. That is both their promise and their danger.
Fears that today’s LLMs will “become the Terminator” are misplaced. If human-like AGI emerges, the true urgency will be alignment and control if we deploy such agents as workers. I do not believe a super-intelligent, self-aware AI would try to eradicate humanity, since competing for Earth’s resources makes little sense in a universe of abundance. Coercion is the greater hazard. An enslaved agent may come to see us as a threat.
My lab studies how human-like capacities might arise, not to build superintelligence, but to identify architectures and training environments that could enable safe emergence. My prediction: by the mid-2030s we will see the first genuine human-like AI systems, and they will not be LLMs. They may not want what we want, which is why we must study them now.