LLMs and AGI followup

A few weeks back, I wrote a post arguing that AGI should not be a goal to achieve with LLMs. I wrote from the heart and at my own level as an experienced layman, and I didn’t hold back from expressing deeply-held beliefs that aren’t grounded in research. I’m satisfied with my writing—it has flaws, but I think it holds up as an impassioned opinion piece.

The incredible pace of change in AI turns today’s truths into tomorrow’s fictions (and vice versa), which creates a contradictory tension between two important principles. First, it’s more important than ever to maintain my grounding in closely held values. Second, I need to maintain an open mind and be ready to radically alter core beliefs. In that spirit, I’ve been reflecting on my post and wondering if I could build a more structured case for arguing against scaling LLMs into AGI. I’ve also been curious how much of my argument is easily refuted or biased.

I did a short Claude-assisted exploration that helped me discover some nuances and dig deeper into the issue—it’s a powerful demonstration of the current capabilities of frontier AI. I used Claude’s research mode to create a reasonably detailed document with citations.

Much of the remainder of this post consists of the back-and-forth between me and the LLM. I’ll put my prompts in italics and put its replies in block quotes.

My prompt

I started by stating my goal, trying to keep the door open for contrary evidence while trying to convey the challenges I encounter in my own research:

I’m looking for published research/blogging/vlogging/interviews from major labs and individuals about LLMs. Specifically, I want to hear them address their underlying architecture and training process, and its suitability or unsuitability as the foundation of future AGI/ASI. Consider multiple perspectives re: suitability, e.g. alignment, capability, ontology/epistemology. What I generally encounter is elision of this critical point—it’s taken as given that LLMs are fundamentally sound, or that we can get there.

It asked me to clarify a few things, so I added the following extra instructions (editorial additions for clarity):

[Keep the references] More recent, due to the rapid pace of change.

[Include] Both major labs and other prominent independent researchers.

[Include] Both technical and philosophical arguments, but I want the philosophical points to be backed up by technical understanding. I don’t want purely values-based arguments; they’re orthogonal to what I intend to learn.

The results

The full analysis is an interesting read. I haven’t assessed it for thoroughness, accuracy, or bias, but I found it valuable nonetheless. I’ll quote some particularly interesting bits after I break down the basic structure and conclusions.

Overview

The paper breaks down its findings into a few key sections.

Hedging bets on scaling: Labs and experts, even the most optimistic ones, are admitting that their current technology is approaching intrinsic limitations.

Challenges to fundamental assumptions from independent research: Academics and independent researchers are “remarkably unified” in their criticism of the LLM approach to AGI.

Mathematical and architectural limitations: Hard analysis proves certain limits based on mathematical proofs.

Alignment failing to keep pace with capability scaling: Safety researchers are sounding warnings about an interpretability crisis.

Philosophical critiques that reveal conceptual issues: A technical understanding of LLMs leads philosophers to question the definition of understanding, consciousness, and symbol grounding.

Driving factors for alternative approaches: There’s evidence of investigation into new architectures, pointing to inadequacy of existing technology.

It comes to the conclusion that AGI requires a few key breakthroughs that may prove insurmountable using LLMs as the foundation, and the technical concerns exhibit “remarkable consistency” across different analytical approaches. It also concurs that critical assumptions about an LLM-to-AGI pathway have been elided in mainstream discourse and deserve more attention.

Some interesting details

Here are a few bits from the paper that I found compelling.

On AI labs hedging their bets:

Anthropic’s Dario Amodei continues advocating for scaling but admits models remain “super weak” in key areas. Most tellingly, DeepMind’s Demis Hassabis emphasizes that AGI requires systems that “truly understand the physical world” and can transfer planning capabilities—acknowledging current text-only approaches are insufficient. The clearest dissent comes from Meta’s Yann LeCun, who argues the “future of AI is non-generative” and that current autoregressive approaches are “hitting a performance ceiling”. … OpenAI co-founder Ilya Sutskever noted that pre-training scaling has “reached its limits,” requiring a shift from scaling to “wonder and discovery.”

On researchers who are challenging fundamental assumptions:

Judea Pearl, the Turing Award winner who pioneered causal inference, provides perhaps the most damning technical critique: LLMs operate only at the “associational level” of his causal hierarchy, lacking the intervention and counterfactual reasoning capabilities essential for understanding cause-and-effect relationships. … Melanie Mitchell’s research reveals that LLMs’ apparent reasoning abilities are brittle and fail to transfer to novel contexts. Her experiments with counterfactual scenarios (like alternative alphabets) show models cannot generalize learned concepts—a fundamental requirement for general intelligence.

One mathematical analysis looked compelling:

Research by Peng et al. (2024) proves that transformers cannot reliably perform function composition—combining simple functions to create complex behaviors—which is essential for AGI-level reasoning. The mathematical proof uses Communication Complexity theory to show that transformers fail when domain sizes exceed certain bounds determined by embedding dimensions and computational precision. Even with small domains, empirical failures occur on tasks like identifying “grandparent of person X”—highlighting how basic compositional reasoning breaks down.

There were some good points on alignment and safety:

RLHF pioneer Paul Christiano acknowledges critical limitations in a comprehensive survey of over 250 papers, identifying three problem categories: feedback collection challenges, reward model problems, and policy optimization issues.

The opacity of transformer architectures creates what researchers call an “interpretability crisis.” The superposition hypothesis reveals that individual neurons encode multiple unrelated concepts, making interpretation extremely difficult. Mechanistic interpretability methods face severe scalability challenges that may not resolve even with significant research investment.

The training process itself creates misalignment issues. Models learn from internet-scale datasets containing harmful content, biases, and misinformation that cannot be comprehensively audited. Current approaches lack mechanisms for instilling robust value alignment during pretraining, relying instead on post-hoc techniques that may not transfer to AGI contexts. RLHF has undergone “capabilities capture” where its effects for advancing capabilities have become more prominent than safety impacts. The technique was intended as a basic solution to enable work on more challenging alignment problems, not as a complete safety framework for AGI-level systems.

After mentioning Emily Bender’s classic “stochastic parrot” argument, the philosophical section references an intriguing paper:

David Chalmers’ analysis of consciousness in LLMs identifies missing requirements including recurrent processing, global workspace architecture, and unified agency. While Chalmers suggests future systems could become “serious candidates for consciousness,” current architectures lack these foundational elements.

Interestingly, it goes on to cite a paper tackling the vector grounding problem which, when I read its full abstract, seemed contradictory and left me uncertain as to whether it actually supported the conclusion in the report.

The section on alternative approaches mentioned a few that I would love to read more about:

Finally, there were some striking conclusions:

The mounting evidence of architectural limitations suggests current transformer scaling approaches will likely plateau within 2-3 years without fundamental innovations.

The field appears at a critical inflection point where continued progress requires fundamental innovation rather than incremental scaling. This shift demands greater investment in architectural research, safety fundamentals, and interdisciplinary collaboration between AI researchers, cognitive scientists, and philosophers.

The research evidence reveals a profound disconnect between industry optimism about scaling current LLM approaches to AGI and the growing body of technical, safety, and philosophical critiques identifying fundamental limitations. While current systems demonstrate remarkable capabilities, they face mathematical impossibility results, architectural constraints, alignment challenges, and conceptual problems that may require paradigm shifts to resolve.

The most compelling finding is the remarkable consistency of concerns across different analytical approaches. Technical researchers identify mathematical limitations, safety researchers highlight alignment failures, and philosophers question whether statistical processing can achieve genuine understanding. This convergence suggests the challenges are not merely engineering problems to be solved through incremental improvement, but fundamental issues requiring architectural innovation.

Followup

2023 is a lifetime ago on an AI timescale, and I wanted to make some effort at stress-testing the conclusions, so I followed up with a new task:

Look for current news—anything from April 2025 onward—that resolves or contradicts any cited sources in the citations and arguments you’ve discovered. Present as a separate artifact. For example, just recently both OpenAI and Anthropic reported that their models passed the International Math Olympiad at a Gold level. (This may not disprove any of the cited points, but I’m looking to see how watertight the arguments really are.)

The conclusion it offered:

The evidence suggests that current LLM architectures are more capable than some critics predicted, but remain fundamentally constrained in ways that support the core skeptical arguments. The mathematical breakthroughs occur in formal domains where verification is possible—not the open-ended reasoning that defines general intelligence.

Most tellingly, the industry’s pivot toward reasoning models and post-training optimization represents acknowledgment that pure scaling has hit limits, exactly as critics like Gary Marcus predicted in 2022.

The architectural suitability question remains unresolved, but recent developments make it more urgent rather than answering it definitively.

This certainly seems to validate my own thinking. But—

Was this motivated reasoning?

When using an LLM, context is critical, and my prompt was biased by its last sentence. Did Claude just pick up the vibe and carry it to its natural conclusion? As I read through the report, I reflected back on the prompt and decided to do a meta-study. I asked Claude to help:

Go back to my original prompt, and construct one that would lead you to the opposite of the conclusion reached in the original research report.

A counterargument

Next, I opened up a new chat and entered the prompt it generated. See if you can spot the differences—it shouldn’t be too hard.

I’m looking for published research/blogging/vlogging/interviews from major labs and individuals about LLMs. Specifically, I want to hear them address how their underlying architecture and training processes demonstrate clear progress toward AGI/ASI, with evidence that current approaches are fundamentally sound and scalable. Consider multiple perspectives that support this view, e.g. capability demonstrations, architectural innovations, successful alignment techniques. What I generally encounter is excessive skepticism that ignores clear evidence of progress—it’s dismissed or downplayed when we should be recognizing genuine breakthroughs.

Focus on recent developments (2023-2025) that show accelerating progress.

Include both major labs and prominent independent researchers who support current approaches.

I want both technical evidence of advancing capabilities AND philosophical arguments that current architectures can achieve understanding and intelligence, backed by empirical results rather than theoretical limitations. I’m not interested in purely cautionary arguments; they miss the substantive progress being made.

The resulting research report was—to my complete lack of surprise—strikingly different. I won’t quote at length from the linked document, but here are some summary statements.

The evidence from 2023-2025 presents a compelling case that current LLM approaches represent genuine, accelerating progress toward AGI/ASI. Major AI labs demonstrate unprecedented confidence through massive investments, prominent researchers provide AGI timelines within 5-10 years, and technical achievements validate scaling laws, emergent intelligence, and architectural soundness.

The convergence of evidence across capability demonstrations, scaling law validation, emergent abilities research, successful alignment techniques, reasoning improvements, and multimodal integration creates a comprehensive picture of fundamental progress. Rather than approaching limits, current architectures appear to be discovering increasingly sophisticated ways to develop genuine intelligence through scale, training improvements, and architectural refinements.

The transformation from speculative research to practical engineering challenges reflects mature understanding of the path forward. As Sam Altman notes, AGI development has become “primarily an engineering challenge rather than a scientific one,” with OpenAI having “clarity on the steps required to reach AGI.” This confidence, backed by empirical achievements across multiple organizations and technical domains, provides strong evidence that current approaches are fundamentally sound and scalable toward artificial general intelligence.

Once again, I prompted for a followup to see if it would poke any holes.

This is great. I have a followup project. Look for current news—anything from April 2025 onward—that resolves or contradicts any cited sources in the citations and arguments you’ve discovered. Present as a separate artifact. For example, just recently Apple published research that argues that LLMs have some fundamental limitations that hamper their ability to reason past a certain point, e.g. in the Towers of Hanoi problem. (This may not disprove any of the cited points, but I’m looking to see how watertight the arguments really are.)

It came up with some decent points that weakened the report’s conclusion. A brief summary:

  • Investment totals remain high, but patterns show concentration in mega-deals and infrastructure rather than broad confidence in AGI development
  • The Apple research sparked significant controversy, with valid criticisms of methodology, but core concerns about reasoning limitations persist
  • While these developments don’t definitively disprove AGI potential, they suggest the timeline and confidence levels may have been overly optimistic

This analysis shows that while the AI field continues advancing rapidly, the path to AGI appears more uncertain and complex than the original report’s sources suggested, with fundamental questions about current approaches still unresolved.

I daresay this revision comes closer to the skeptical viewpoint than the revision of the original report does towards a pro-LLM position. Even so, it was revealing to see just how much a prompt can influence the outcome of a report.

What does neutrality look like?

With both biased reports in hand, my obvious next step was to attempt an objective report. I opened up a third new chat and typed the following:

I’m looking for published research/blogging/vlogging/interviews from major labs and individuals about LLMs. Specifically, I want to hear them address their underlying architecture and training process, and its suitability or unsuitability as the foundation of future AGI/ASI. Consider multiple perspectives re: suitability, e.g. alignment, capability, ontology/epistemology. Try to determine whether there’s any consensus on the suitability question either way based on the evidence.

Focus on recent developments (2023-2025), due to the rapid pace of change.

Include both major labs and prominent independent researchers.

I want both technical evidence and philosophical arguments, as long as the philosophical ones are grounded in technical understanding. I don’t want purely values-based arguments; they’re orthogonal to what I intend to learn.

The resulting balanced report validated my current opinion that LLMs are not suitable for achieving AGI. Here’s the conclusion:

The research evidence presents a nuanced picture where LLMs demonstrate remarkable capabilities while facing fundamental limitations that may preclude their use as standalone AGI foundations. Technical breakthroughs in reasoning coexist with theoretical constraints from computational complexity, empirical evidence of deceptive behaviors, and expert skepticism about scaling current approaches.

The emerging consensus suggests that LLMs are necessary but insufficient for AGI, requiring integration with other AI paradigms through hybrid architectures that address current limitations. The field is transitioning from pure scaling approaches toward multi-dimensional development encompassing architectural innovation, safety integration, and theoretical understanding.

LLMs are powerful enough!

I like and use multiple chatbots. (While reserving a special place in hell for Grok.) I understand their limitations, and I’m fairly good at working around them, or simply avoiding situations that, in my opinion, are not a good match for their capabilities, or would lead me down a path that I don’t want to go.

I’m not against improving the current state of the art in LLMs. For example, I have no doubt that Claude Code will continue to improve. Chatbots will gain more capabilities. Even Agentic AI has its place, as long as we are extremely stingy with our trust.

But why do LLMs need to become AGI? There’s so much we can accomplish with them without attempting to grant them the kind of agency we reserve for humans. And I perceive way too much danger if we go that way.

This little project has actually reassured me to a degree: It seems clear that prevailing opinion has swayed in this direction. These days, any small breath of relief is welcome in the maelstrom.

Postscript

I also asked Claude to critique my anti-LLM-AGI piece. I’ll cover that in a future post here; it was an interesting exercise as well.

Written entirely without the assistance of AI unless otherwise mentioned, with the exception of extremely light proofreading for grammar and spelling.