10d • Research
The Anatomy of Unmoored Observation
The AI Literacy Gap: Quantified Ignorance
The assertion that AI commentary lacks “fundamental knowledge or familiarity with the research literature” finds strong empirical support. Multiple dimensions of this knowledge gap are measurable:
Educational institutions lag profoundly. A 2024 survey commissioned by Actua found that less than 48% of educators interviewed felt equipped to use AI tools in the classroom, 46% felt confident teaching responsible AI use, and only 42% felt ready to teach students how to use artificial intelligence effectively. UNESCO’s 2024-2025 Fluency Report warns that without AI literacy, individuals struggle to distinguish authentic content from synthetic media and lack ability to critically evaluate AI-generated outputs.
Even AI experts display concerning knowledge gaps. A 2025 survey of AI experts revealed that only 21% had heard of “instrumental convergence”—a fundamental concept in AI safety predicting that advanced AI systems will pursue certain instrumental goals regardless of their terminal objectives. This represents a shocking level of unfamiliarity with core theoretical frameworks within the expert community itself, suggesting that if credentialed researchers lack grounding in foundational concepts, the broader commentary ecosystem operates at even greater remove from established knowledge.
The literacy problem extends asymmetrically across demographics and domains. Stanford research found that less-educated regions adopted AI writing tools faster than highly-educated areas, suggesting enthusiasm outpaces comprehension. Meanwhile, 85% of healthcare professionals expressed interest in introductory AI courses tailored to healthcare, indicating even domain experts recognize their knowledge deficits when AI enters their fields.
This creates a dangerous dynamic: those with least technical grounding may be most confident in their AI usage and commentary, while those with deeper expertise recognize the vastness of their ignorance. The result is a marketplace of ideas where confidence and volume substitute for competence.
The Expert-Pundit Divide: Credentials Without Understanding
A critical distinction separates experts from pundits, though credentials alone don’t determine which category someone occupies. Experts are evaluated on correctness; pundits on engagement. Experts ground arguments in coherent theory and evidence; pundits amplify patterns observed on social media.
Gary Marcus exemplifies the credentialed critic whose work has sparked debate about whether technical expertise translates to predictive accuracy. Marcus, a professor emeritus at NYU and prominent cognitive scientist, has positioned himself as “AI’s loudest critic”. His background is substantial: doctoral work in cognitive science, research on AI limitations, and books on AI’s shortcomings. Yet critics note Marcus doesn’t systematically acknowledge prediction errors and “moves to the next claim without analysis” when previous assertions prove incorrect.
The Marcus case illustrates a subtle but important point: credentials establish a floor, not a ceiling, for commentary quality. His criticisms of deep learning’s limitations are technically grounded—pointing to issues with generalization, causal reasoning, and common sense that remain active research challenges. But the tone and framing of those criticisms (warning of imminent AI winter, declaring technologies “fantastic but not a path to AGI”) represent interpretive leaps beyond the technical foundations.
Emily Bender and Melanie Mitchell represent more systematic critical approaches. Bender’s “stochastic parrots” framework provides a memorable metaphor grounded in technical reality: large language models do stitch together linguistic forms according to probabilistic patterns observed in training data, without reference to meaning. This doesn’t preclude sophisticated behavior, but it clarifies what these systems fundamentally are and aren’t. Mitchell’s work on abstraction and analogy similarly grounds limitations in cognitive science rather than speculation.
The distinction matters because credentialed skepticism can still constitute “Lamarckian punditry” if it amplifies observations without engaging mechanisms. Marcus observing that Sora generates videos with physics errors doesn’t constitute deep analysis unless it connects to broader understanding of how diffusion models learn spatiotemporal representations. Conversely, technically-grounded explanation of why current architectures struggle with physical reasoning (lack of causal models, training on static images, no world models) represents genuine expertise.
Design punditry provides a non-AI example of the same phenomenon. A LinkedIn post about AI in product design noted: “There’s been an increase in punditry about AI’s influence on product design… I can only assume this is being driven by folks who have not been practitioners at any real scale because it is absurd”. The critique: commentators declare “Figma is dead” and design will move to prompt-based tools without understanding the gap between generic AI output and production-ready, intentional design work. This represents classic Lamarckian punditry—observing AI can generate code from prompts, then transmitting the conclusion that professional designers are obsolete, without engaging the fundamental knowledge of what design work actually entails.
The Research Literature Problem: Selective Reading and Survivorship Bias
If Lamarckian AI punditry is defined by being “unmoored to fundamental knowledge or familiarity with the research literature,” we must examine what the research literature itself reveals—and critically, what it doesn’t reveal due to publication bias.
Systematic problems plague AI research publication. The exponential growth of AI papers is testing the resilience of peer review systems, with “immediate release of papers without peer-review evaluation having become widely accepted across many areas of AI research”. Legacy and social media increasingly cover AI research “often with contradictory statements that confuse readers and blur the line between reality and perception of AI capabilities”. This creates an environment where distinguishing genuine findings from hype becomes increasingly difficult even for motivated readers.
Survivorship bias distorts the apparent success of AI in science. A researcher who investigated AI applications in computational science reported: “AI-for-science researchers almost always report successes of AI, and rarely publish results when AI isn’t successful”. The mechanism is straightforward: publication depends not on statistical significance but on whether the proposed method outperforms other approaches or successfully performs some novel task. Failed approaches remain in file drawers, creating the illusion that AI succeeds wherever applied.
This researcher—initially optimistic about AI’s scientific potential—concluded: “I’ve come to believe that AI has generally been less successful and revolutionary in science than it appears to be”. The reasons illuminate broader problems:
• Working backwards: Researchers assume AI will be the solution, then look for problems to solve, rather than identifying problems and finding appropriate methods
• Pitfalls systematically bias results: Data leakage, weak baselines, cherry-picking, and misreporting cause published successes to reach “overly optimistic conclusions about AI in science”
• Conflict of interest: The same people who evaluate AI models benefit from positive evaluations
These dynamics aren’t unique to scientific AI applications—they characterize AI research generally. A 2024 study using EHRA AI checklists found that in atrial fibrillation management, sudden cardiac death, and electrophysiology applications, “in no domain did reporting of a specific item exceed 55% of included papers”. Key areas like trial registration, participant details, and training performance were underreported in over 80% of papers.
The research literature thus simultaneously contains valuable knowledge and systematic distortions. Familiarity with that literature is necessary but not sufficient for grounded commentary—one must also understand the meta-problems affecting what gets published and how results get framed.
0
0 comments
The Anatomy of Unmoored Observation
skool.com/burstiness-and-perplexity
Master AI use cases from legal & the supply chain to digital marketing & SEO. Agents, analysis, content creation--Burstiness & Perplexity from NovCog
Leaderboard (30-day)
1
+2
Powered by