the tech industry’s flawed model of your brain: a case study in how clever people can be so very stupid

Two news stories today hit a consistent theme of mine: the tech industry has a serious problem - profound, deep, worrisome.

BL, UF: they have precisely no idea how humans brains process information; they don’t care, and it keeps blowing up in their faces.

And then they chase the next shiney thing while not fixing the obvious things in their products.

Story one: ‘Microsoft wants you to talk to your PC and let AI control it’ (Öffnet in neuem Fenster)

This is so derangedly stupid that words almost fail me.

MQ (but read the whole thing):

Microsoft is launching a set of capabilities in Windows today that will start to weave AI features into regular Windows 11 PCs, instead of consumers having to buy a special Copilot Plus PC. The biggest change is that Microsoft thinks people will want to talk to their computers and have Copilot take actions on their behalf.

“You should be able to talk to your PC, have it understand you, and then be able to have magic happen from that,” says Mehdi. “With your permission, we want people to be able to share with their AI on Windows what they’re doing and what they’re seeing. The PC should be able to act on your behalf.”

Here you are in your shared workspace, saying out loud to an idiot powerpoint:

Move that line just a small bit to the left on the slide. No the other one. Now group those objects into a single object.

No, not those ones. The other ones. Stop. Try it again. It'd be faster if I did it myself…

They see billions of talk minutes on Teams and think "Aha! People love talking to computers!"

No. This is confirmation bias writ large. The mistake is simple:

People love talking to each other - for very good reasons: it’s our cognitive interface with others. (Öffnet in neuem Fenster)

And they will tolerate talking through computers to in order to talk to other people...

Story two: ‘AI-powered textbooks fail to make the grade in South Korea’ (Öffnet in neuem Fenster)

“All our classes were delayed because of technical problems with the textbooks,” Ko said. “I also didn’t know how to use them well. Working individually on my laptop, I found it hard to stay focused and keep on track. The textbooks didn’t provide lessons tailored to my level.”

Paperbooks are an ancient technology. I can go and take a five hundred year old printed book off the shelf and read it. A CD from the 1990s? Maybe. But really? Who remembers Encarta (Öffnet in neuem Fenster)? Good luck getting those CDs to work now.

your brain is a prediction engine

You are not a computer; you are a prediction engine. Until Silicon Valley accepts this very basic point, the promise of world-changing AI will keep colliding with human cognition; the results are predictable: this stuff will not work they way they think it will, because your brain does not work the way they think it does.

Summon a familiar annoyance: not the drama of a catastrophic crash, but the slow drip of small betrayals. Word quietly reformats a paragraph you already fixed; PowerPoint’s Slide Master produces a new, unintended layout that appears to have invented itself.

A thousand cuts; time is lost because the basics do not work.

Here’s today’s issue: I make two differing formats of a PDF of a PPT - slides and handouts. I try to load them to Blackboard, which can only see one of them. I assume the fault is mine (reader: it’s not), and redo the files again.

Crickets.

I try again. UGH. I go to copilot, which offers a long list of things to do. NONE of them work.

I drag the file out of the folder to the desktop, try it again. And, inexplicably, it works. I put it back in the folder. And now it works… wtf?

I tell Copilot, which replies: “That is wild — and oddly satisfying! It sounds like the file got “unstuck” somehow when you moved it to the Desktop and back. Here's what likely happened:” (longlist of gibberish about metadata follows).

co-pilot

These are not isolated defects:- the most basic functions are at issue here - the ToCs break, the Slide Master never works, and a hundred other problems. These are signals the systems in question do not serve the user they claim to serve.

And in recent months, we get grand announcements about operating systems “rewritten around AI” arriving with choreographed confidence; national edtech platforms promise personalisation, then a retreat under public pressure. The details differ each time; but the cognitive failure - a refusal to care about how humans think and act - is shared widely.

And the basics - the avoidable frictions - are NEVER fixed. I suspect they are simply not sexy enough.

the empty-vessel mistake

The tech industry proceeds from a tacit picture of the mind:- call it the Empty Vessel, the brain is a passive, lossy processor that needs more data; learning is about ingestion; and downtime is inefficiency.

And the implication of this utterly wrong model is simple: add a conversational agent to every surface; pipe in summaries; eliminate so-called friction.

This is a profound category error. The brain is not a rate-limited narrow gauge pipe; it is a predictive system drven largely by its own intrinsic activity.

you are a prediction engine

The predictive processing framework offers a better account than the techbros. The brain constructs many hierarchical generative models of the world; it predicts incoming signals at multiple levels of abstraction; it updates itself by minimizing prediction error.

Sensory input is not simply read; it is compared to a top-down forecast. The difference term—the prediction error—is passed upward for model revision; in some formulations, this is framed as free-energy minimization. Precision weights modulate which errors matter; attention is, in part, the selective increase of precision on specific channels. This is how perception, action, and learning hang together. And there’s much more - we have rapid implicit and explicit learning systems - ones that are very robust, and that take a lot of damage to knock them off course (you learned your own name at say two or three months of age - and you can still recall it a perhaps a century later. No machine has every down this.)

Two points follow that product teams consistently overlook:

First, learning is sample-efficient because the underlying model is rich in crosstabs and prediction. A child needs few exemplars to induce the concept of “cat”; prior structure does the heavy lifting. A veteran accountant inspects one spreadsheet and locates the anomaly; years of hierarchical priors make the pattern pop.

Drowning such expertise in auto-suggestions does not help; it injects noise into a well-tuned model.

Second, reward signals are not decorative; they are computational. Dopaminergic systems track reward prediction error; positive surprise updates value estimates and sustains exploration; chronic negative surprises degrade motivation and trust. And the latter is why we have such low expectations of IT systems - they work so badly, instead of so well - nobody cares about the basics - but the basics is what we want to work.

If your software produces frequent small mismatches between expectation and outcome, do not be surprised when users disengage; you have taught their brains to expect disappointment.

downtime is cognitive work

The default mode network is active during mind-wandering and quiet rest; hippocampal–cortical loops replay recent experience during sleep; consolidation strengthens useful traces and prunes noise.

This is not waste; this is maintenance and creative recombination.

The farmer who never lets a field lie fallow gets one good harvest; then the soil fails. The brain needs rest. It needs control (agency!). It needs aerobic exercise! It needs lots of things!

Tools that carve up attention into ever finer fragments ignore a simple truth: recovery and integration are part of thinking.

friction is often the thought

Some effort is a feature - cognitive friction can be a good thing. The act of manually shaping a document—choosing headings, adjusting structure, testing cadence, these all externalise cognition; manually shaping and typing allows rapid error detection by eye and hand; and it supports a fast visuomotor loop with minimal overhead.

Replacing this with voice instructions to an agent sounds futuristic; in practice it forces a slow, serial, auditory–vocal loop that increases cognitive load for many tasks.

You become a micromanager of an erratic intern; you trade fluent doing badly for constant supervision. It’ll worsen your performance.

Progress inevitably halts; and you mentally say ‘wtf - again with this 💩’.

when design collides with cognition

Consider the push for always-on conversational interfaces.

The social context is obvious: people do not want to narrate their work to a laptop in shared spaces; attention is a commons in a shared space, and speech is invasive. You know this. How difficult is to concentrate on your text when someone is speaking on the phone beside you on the bus or train.

The neurocognitive context is also vital: for common tasks, eyes-hands-screen is parallel and low-latency processing, absorbing little bandwidth, and giving you that vital time to pause, consider, think, move things around; ears-mouth-agent is serial and high-latency. It’s fine for dictation - but anything more complicated? Nah. The latter raises switching costs and disrupts flow; the promised productivity evaporates. And even your dictated text needs lots of work. I regularly use dictation to get the flow of a text going (I’ve written the first lousy drafts of my books that way (Öffnet in neuem Fenster)), but unless you speak in perfect considered prose, the writing actually happens in the rewriting and editing.

Or consider mass “AI textbooks” promising personalisation. True adaptation requires robust learner models; it requires accurate precision weighting of errors, sensitivity to prior knowledge, and stability in basic interactions. If the platform delivers interface glitches, mismatched lessons, and privacy uncertainty, it generates repeated negative prediction errors; students and teachers learn the correct lesson: do not trust this system.

And the books themselves must obey basic laws of human learning and memory, not fads or delusions:

• learning styles: a seductive but utterly useless idea - Fuggetaboutit, they don't exist (Öffnet in neuem Fenster)

• using your memory to strengthen your memory: retrieval practice - perhaps the best, general, scientifically-validated, most-easily implemented method for strengthening your memory (Öffnet in neuem Fenster)

• interleaving: the mix-it-up memory method that works: fluency is a bad teacher - what feels good misleads learners about what they've learned (Öffnet in neuem Fenster)

the deterministic-algorithm delusion

There is a deeper conceit: with enough data, human cognition can be treated as a deterministic algorithm; and surprise, an error term to be eliminated. This is neat and wrong.

A system with zero prediction error is a system with zero learning; creativity collapses; boredom expands.

a brief for human-centred tools

We are not inefficient machines waiting for a patch; we are predictive organisms that thrive on well-tuned models, stable contexts, and occasional, well-timed novelty. Better tools will do the following:

• respect cognitive load: fewer prompts; clearer affordances; assistance on request rather than constant interruption; sensible defaults that do not fight the user.

• protect attention and recovery: modes for deep work; quiet surfaces; schedules that leave room for rest and consolidation.

• build trust in small increments: reliability in routine interactions; graceful failure when things go wrong; transparent state that matches user expectations.

• leave space for positive surprise: features that surface serendipity without flooding the channel; recommendations that can be ignored without penalty.

There is nothing radical here; only psychology that has been available for decades. Yet we continue to watch clever people reinvent bad ideas with perfect confidence; and the results are as predictable as they are avoidable.

the warning in the papercuts

The next time your word processor “helps” by breaking a paragraph you just fixed, treat it as a signal. These are not trivial bugs; they are reminders the tools are under-designed, and will keep underperforming, unless the techbros do the necessary work of making the basics work every time.

We deserve systems that work with the brains we actually have; not with a fantasy of a user who never minds the interruption, never needs the pause, and never notices when trust is run down, one small error at a time.