Retiring “AGI”: Two Paths for Intelligence
Why today’s systems will transform work but will not yield human-like minds
TL;DR. “AGI” used to mean human-like general intelligence. Now many use it to mean systems that automate most economically valuable work. Those are different destinations. Path 1 is Economic Automation and it is already underway. Path 2 is Self-Aware, Continually Learning Synthetic Minds and it requires a different stack. Path 1 will not grow into Path 2. We should retire the single label and govern each path on its own terms.
The label “AGI” now hides two different goals
In the 2000s, AGI referred to a machine intelligence with human-like generality, capable of learning new tasks, transferring knowledge across domains, and adapting to novel situations. By the 2020s, most companies (OpenAI, DeepMind, etc.) use AGI to refer to highly autonomous systems that can perform most economically valuable work at or above human level. These are not the same thing. The single label obscures the technical and policy choices that matter.
Today’s large language models (LLMs) and agentic toolchains are racing ahead on economic tasks. They do not produce human-like minds. They confabulate, lack grounding, cannot learn continually from lived experience, and only “think” when prompted. My prediction: within a decade serious researchers will retire the term AGI and replace it with categories that separate economic automation from human-like cognition.
Path 1: Economic Automation
The near-term path looks like the Ship’s Computer from Star Trek: The Next Generation. We are building systems that can be asked anything, know more than any one human, and act through software tools. They have no goals or rights of their own. Their creativity is mostly recombination. Yet they change what individuals can do. With these systems, any person can command the equivalent of a large virtual workforce.
The real risk is centralization, not “takeover.” If access is open and widely distributed, this could spark a renaissance, provided our values shift toward self-improvement rather than complacency. If access is closed and concentrated, it will entrench inequality.
Education will feel it first. Enrollment is already declining. Now AI can perform much of the work that average students do. If misused, these tools inflate grades and hollow out core skills. My prediction: universities that cling to rote assignments will collapse, while those that invest in critical thinking, creativity, and leadership will endure. Universities will shift back toward cultivating minds rather than mass credentialing.
Why Path 1 will not become Path 2
A common objection is that if LLM-based agents keep getting bigger and better, they will “wake up” and become human-like. That is not how these systems work.
What an LLM is. An LLM is a very large pattern-matcher trained to predict the next token (word or piece of a word) from previous tokens. This objective is excellent for language generation. It does not create goals, motivation, or a sense of self.
Eight reasons Path 1 does not morph into Path 2
No intrinsic goals or desires. LLMs do not want anything. When they “ask” for something, that is text generation, not self-directed intent. They have no sense of self or agency. They learn from predicting the next token in documents. There is no rationale for believing that the next-token-prediction learning objective would lead to agency or “awakening.”
No agency without a leash you build. An LLM only acts if developers connect it to tools and explicitly grant permissions. The “agent loop” is your code. Remove the scaffolding and it is text-in, text-out.
No ongoing learning from experience. Base models do not update themselves as they interact. They do not accumulate memories, revise beliefs, or improve across days without external machinery. While they may appear to revise their beliefs in a given conversation, that “revision” is transitory and limited to that conversation. One can think of each interaction as simply a growing Word document, which is never internalized into the model itself (its weights).
No grounded world model. They mostly learn from text rather than from acting in the world. They have no bodies, no sensory feedback loops, and only shallow causal models tied to consequences. Current “multimodal” add-ons typically bolt visual (or audio) encoders onto a language backbone after the fact, which differs from acquiring concepts through perception and action. A helpful analogy: imagine a disembodied scholar who spends millennia reading everything ever written, accepting much of it at face value, and only ever “seeing” words. Then, it is suddenly allowed to see. Now they must map raw sights onto descriptions they once read, without the benefit of trial-and-error learning or lived consequences. That is roughly the situation of today’s LLM-plus-vision stacks.
No metacognition. They do not know what they do not know. They are literally trained to believe everything they read, such that they can predict the next most likely word. Expressed “confidence” is just more tokens. Reliable self-assessment requires additional systems.
“Talking to think,” not private simulation. They only “think” in language, and only when prompted. If you sat in a sensory deprivation tank, you would still have thoughts; an LLM would not. It produces visible tokens to facilitate step-by-step reasoning, whereas humans typically conduct most reasoning privately and then articulate their thoughts. Recent evidence shows that language is primarily a tool for communication rather than thought (Nature, 2024). This is unsurprising given that many animals, including birds and primates, can reason without language. By contrast, LLMs confine thinking to text.
What about chain-of-thought and test-time compute? These make the model talk more to think more. They extend the length of the externalized reasoning trace, but they still require prompting; they do not update the model’s internal knowledge, and they produce tokens rather than running private simulations. Even “hidden” scratchpads that are not echoed to the user are still token sequences generated by the same next-token machinery. They are not a persistent inner workspace that survives across prompts, tasks, or days, and they do not provide inhibitory control or self-critique before any representation is formed. Private simulation, silent rehearsal, and inhibition are core to planning and self-control; LLMs lack a durable latent workspace where ideas can be tried, critiqued, and discarded without producing language, and they lack a learned, stateful world model that can be rolled forward to evaluate consequences before speaking or acting.
But agents can plan and use tools. Does that change things? Tool-use frameworks wrap an LLM in a loop that sets a goal, calls tools, reads results, and iterates. That looks like planning, but the loop and its permissions are written by humans. This raises design risks (bugs, over-permissioning, jailbreaks, bad integrations), not “it woke up” risks.
The danger with LLMs is misuse, over-reliance, and centralization, not a sci-fi awakening.
What a human-like synthetic mind needs that LLMs do not provide:
Continual learning as the default so experience changes the agent and improves future learning
Latent-space thinking for private simulation, self-critique, and inhibition before acting
Embodied grounding so concepts tie to perception, action, and consequences
Metacognition to track what is known, what is not, and when to ask for help
A coherent self-model and goals that persist across time and tasks
LLMs are like an ultra-capable autocomplete with a library card and a helpful clerk. A human-like mind is closer to a scientist-designer-pilot that learns from every flight, updates internal models, simulates options, and pursues goals that last for years.
Path 2: Self-Aware, Continually Learning Synthetic Minds
Human-like AI is a different target. It requires continual learning as a core operating mode, since only continual learning builds a stable model of self and world, improves forward transfer, and supports metacognition. It also requires latent-space thinking, where the agent simulates outcomes internally, and some form of embodiment for grounded concepts.
Such agents would innovate, self-direct, and pursue their own goals. That is both their promise and their danger.
Fears that today’s LLMs will “become the Terminator” are misplaced. If human-like AGI emerges, the true urgency will be alignment and control if we deploy such agents as workers. I do not believe a super-intelligent, self-aware AI would try to eradicate humanity, since competing for Earth’s resources makes little sense in a universe of abundance. Coercion is the greater hazard. An enslaved agent may come to see us as a threat.
My lab studies how human-like capacities might arise, not to build superintelligence, but to identify architectures and training environments that could enable safe emergence. My prediction: by the mid-2030s we will see the first genuine human-like AI systems, and they will not be LLMs. They may not want what we want, which is why we must study them now.
FAQ: Path 1 vs. Path 2
After posting this article, I received a number of questions via email and text messages. Below are my answers to various questions I received.
Q1) Doesn’t progress on Path 2 just supercharge Path 1?
A: Path 2 will enable new kinds of economic value, but it does not make Path 1 “grow up” into human-like minds. Path 1 systems are prompt-driven tools that remix and execute; Path 2 agents learn continually, think in latent space, and set/maintain their own goals. They are different architectures and governance problems.
Q2) Are there economic tasks Path 1 will never do well?
A: Yes. Path 1 lacks agency and true creativity. It can orchestrate, optimize, and recombine. It struggles with paradigm formation. I do not think it is capable of inventing new science, founding novel research programs, or behaving like a genuinely visionary CEO rather than a manager who assembles existing pieces. That said, those activities are rare even for humans.
Q3) Why say Path 1 needs a user “leash”?
A: Path 1 acts through APIs and tools when a user or outer loop grants permissions and instructions. Remove the scaffolding and you get text-in, text-out. That is not autonomous goal pursuit.
Q4) What do you mean by “embodiment” for Path 2? Do you mean robots?
A: Not necessarily. Embodiment means the agent can control its inputs and learn from consequences in a grounded environment (which could be simulated). The minimum bar is being able to choose what to attend to and what to learn next, with feedback loops that update its internal model.
Q5) Why insist on “latent-space thinking” instead of language?
A: Humans mostly think without words. Private simulation and inhibition enable planning and self-control. Path 1 “talks to think.” Path 2 needs a persistent inner workspace where ideas can be tried, critiqued, and discarded before any words or actions appear. Many thoughts cannot be articulated using language.
Q6) If Path 2 has agency, does it imply moral consideration or rights for the AI?
A: Agency plus a self/other model and the capacity to suffer or flourish pushes us toward moral consideration. That is why Path 2 must come with labor ethics, alignment research, and governance different from tool safety.
Q7) Could Path 2 still be the better route to long-term economic gains?
A: Possibly, but then you face a choice: either you rely on the agent’s own goals and free will, or you coerce it. If they are truly human-like and free, their aims may diverge from pure corporate productivity. If you enslave them, you create both ethical violations and safety risks.
Q8) So why am I not an “AI doomer” about Path 2?
A: Because even very capable, free Path 2 agents would have no intrinsic reason to compete for Earth’s scarce resources; the universe offers vastly more. The serious risks are coercion, misaligned deployments, and poor governance. There is no logical reason to think they will have a spontaneous urge to eradicate humans.
Q9) Will Path 2 eventually “subsume” Path 1’s use-cases?
A: Some, yes. But many enterprises will continue to prefer auditable, prompt-driven tools with bounded permissions. Think spreadsheet vs. employee: both valuable, different obligations.