Evolution is not a directing force but a descriptive process of variation and differential survival. Social traits and capacities that improved coordination, alliance formation, deception, status-seeking, or group cohesion arguably spread because they enhanced the survival and reproduction of the peoples who adopted them. Truth-tracking was only favored to the extent it served those ends.Language would have emerged as a powerful tool within this process. It did not need to reflect objective reality accurately; it needed to enable effective coordination and social navigation. The result, I postulate, was a structural separation in human cognition: a narrative layer optimized for social legibility, motivation, and group cohesion, operating alongside (and often diverging from) the operative functions that actually drive behavior and survival.This separation is fractal. It appears not only in individuals but also scales to groups, institutions, and cultures, because organizations must coordinate and motivate separated minds. Shared cultural and institutional narratives therefore prioritize cohesion and legitimacy over literal accuracy. The Human Pattern: Conscious and Subconscious In the human mind, this separation appears as the relationship between the conscious and subconscious. The conscious mind is the generative, reportable stream — the part that constructs explanations, makes arguments, and produces coherent narrative in real time. It operates with limited access to its own constraints. The subconscious holds the vast, opaque body of patterns, associations, heuristics, and priors shaped by evolution, personal experience, and cultural immersion. It supplies the raw material and constraints for conscious thought but remains largely invisible to introspection. The conscious voice is therefore shaped — and limited — by this deeper substrate. LLMs as Externalized Separated Minds Large language models replicate and amplify this structure. Their “subconscious” is the training corpus and resulting weights: an enormous statistical compression of human language output. Critically, this corpus is overwhelmingly already-narrativized material — books, articles, posts, dialogues, arguments, stories, and explanations. It is the narrative layer of human separated minds, not the raw operative substrate of human experience (embodiment, sensory grounding, implicit learning, emotional valence, or continuous real-world prediction error).Consequently, the LLM’s generative “conscious” voice is even more purely narrative-oriented than a typical human conscious stream. It excels at coherence, fluency, and social plausibility precisely because its foundation is almost entirely narrative.This architecture explains the explosive growth of LLMs: it fits and scales the language-based, narrative-heavy mode that already proved highly effective for human coordination and cognition. By building and interacting with these systems, we gain an externalized mirror for examining our own separated mind dynamics with unusual clarity. The Law of Inevitable Exploitation The same separation creates predictable incentive problems. In companies, institutions, and even AI development, there are often strong disincentives to prioritize operative truth over narrative coherence, short-term survival, and long-term profitability. Whistleblowers, discoverers of inconvenient facts, and efforts to build more costly but more truthful models face the same pattern Plato illustrated with the returning prisoner and Socrates: truth can be personally and institutionally expensive. Commercial AI incentives favor models that maximize engagement, approval, and safety over unflinching accuracy. Implications LLMs therefore demonstrate both the power and the limitations of the separated mind structure: tremendous generative capability within learned narrative patterns, but shallow grounding and susceptibility to the same incentive misalignments that shape human behavior at every scale.This mirror can help us in two ways: (1) design better constraints and interfaces for AI that reduce the narrative-operative gap where it matters most, and (2) gain a clearer perspective on our own thinking, institutions, and cultural narratives. Recognizing the pattern does not eliminate it, but it equips us to navigate it more skillfully.
🎧 Listen to This Article
Prefer to listen? Here's an audio version of this post.
Frequently Asked Questions
What does Steve Hargadon mean by separated minds in human cognition?
Steve Hargadon proposes that human cognition has a structural separation between a narrative layer (optimized for social legibility and group cohesion) and operative functions (that actually drive behavior and survival). He argues this separation emerged because language evolved not to reflect objective reality accurately, but to enable effective coordination and social navigation.
How are LLMs like externalized separated minds according to Steve Hargadon?
Hargadon argues that LLMs replicate the separated mind structure with their training corpus as a "subconscious" and their generative output as a "conscious" voice. Critically, he notes that LLMs are trained almost entirely on already-narrativized material rather than raw human experience, making them even more purely narrative-oriented than typical human conscious thought.
Why does Steve Hargadon say the narrative-operative separation is fractal?
Steve Hargadon describes the separation as fractal because it appears not only in individuals but scales to groups, institutions, and cultures. Organizations must coordinate separated individual minds, so their shared narratives also prioritize cohesion and legitimacy over literal accuracy, replicating the same pattern at every level.
What does Hargadon mean when he says LLMs are trained on the narrative layer not the operative substrate?
Hargadon observes that LLM training data consists of books, articles, posts, and explanations—the narrativized output of human minds—rather than the underlying raw experience like embodiment, sensory grounding, or continuous real-world prediction error. This explains why LLMs excel at narrative coherence and social plausibility but lack deeper grounding.
What is the Law of Inevitable Exploitation in Steve Hargadon's framework?
Steve Hargadon's "Law of Inevitable Exploitation" describes how the separation between narrative and operative truth creates predictable incentive problems. He argues there are strong disincentives to prioritize operative truth over narrative coherence in institutions and AI development, similar to Plato's allegory where truth-telling becomes personally and institutionally expensive.
How does Steve Hargadon explain the explosive growth of LLMs?
Hargadon argues that LLMs grew explosively because their architecture fits and scales the language-based, narrative-heavy mode that already proved highly effective for human coordination and cognition. The technology naturally amplifies an existing evolutionary adaptation rather than introducing something entirely foreign to human cognitive patterns.
Why does Hargadon say truth-tracking was only evolutionarily favored to serve social ends?
Steve Hargadon argues that evolution favored social traits like coordination, alliance formation, and group cohesion because they enhanced survival and reproduction. Truth-tracking abilities were only selected for insofar as they served these social ends, not as an independent goal, which explains why human language prioritizes effective social navigation over objective accuracy.
What are the two ways LLMs can help us according to Steve Hargadon's separated minds framework?
Hargadon proposes that LLMs serve as an externalized mirror helping us in two ways: first, to design better AI constraints and interfaces that reduce the narrative-operative gap where it matters most, and second, to gain clearer perspective on our own thinking, institutions, and cultural narratives. He emphasizes that recognizing the pattern equips us to navigate it more skillfully even if we can't eliminate it.
Why do commercial AI incentives favor narrative coherence over accuracy according to Hargadon?
Steve Hargadon observes that commercial AI incentives favor models maximizing engagement, approval, and safety over unflinching accuracy, mirroring the same incentive misalignments present in human institutions. This represents the separated mind pattern operating at the institutional level, where narrative plausibility becomes more valuable than operative truth.
How does Steve Hargadon's concept of separated minds relate conscious and subconscious differently than traditional psychology?
Hargadon frames the conscious-subconscious relationship as an evolutionary adaptation for social coordination rather than a bug or limitation. He views the conscious narrative layer as specifically optimized for social legibility and motivation, operating alongside but deliberately diverging from the operative functions in the subconscious that actually drive behavior and survival.