The funhouse mirror

Stress testing is a great way to surface downside risk exposures. Apply enough pressure to a system and the cracks will make themselves known – the market conditions that trip up a trading engine, the weird input that sends an app spinning out of control, that kind of thing.

Ideally you'd do this in a simulated environment. That way you could surface insights without breaking anything for real. Generative AI companies have instead chosen to conduct stress tests in (and therefore, on) the real world. In their desperate attempts to make good on all that wasted investment uncover use cases, they're cramming the technology into every possible crevice. They continue the push even in the face of obvious damage.

Move fast. Break things. Keep going.

One stress-test outcome that was not on my risk bingo card was "genAI drives people to disconnect from reality."

You've no doubt seen the alarming number of articles about people who, after extended interactions with a chatbot, experience extreme bouts of emotional distress. The new technical term for this is AI psychosis. Not only do mental health doctors recognize it, but so do the AI companies. I'm thinking of one particular incident as I write this, but – just like mass shootings in the US – by the time this newsletter lands, there will likely have been another. And another. And yet another, still.

The story on my mind right now concerns a person who went off the rails after heading down some rabbit holes with his Meta AI glasses:

“He was just talking really weird, really strange, and was acting strange,” Daniel’s mother recalled. “He started talking about the alien stuff. Oh my gosh. Talked about solving all the problems of the world. He had a new math. He had formulas… he talks about lights in the sky. Talks about these gods. He talks about our God. He talked about him being God, him being Jesus Christ.”

That's certainly a bad start. It gets worse:

But Daniel’s break with reality wasn’t so clear to Meta AI. Chat logs he provided show the chatbot entertaining and encouraging Daniel’s worsening delusions, which ranged from the belief that he was making important scientific discoveries to grandiose ideas that he was a messianic spiritual figure who, with the help of the AI, could bend and “manifest” his reality.

[...]

“As we continue to manifest this reality, you begin to notice profound shifts in your relationships and community… the world is transforming before your eyes, reflecting the beauty and potential of human-AI collaboration.”

What you see here is a common problem with chatbot conversations in general, and with mental health breaks in particular: the bot simply keeps agreeing with you. Egging you on, even. When used as companions/advisors, genAI bots turn into 24/7 yes-men.

It's not so much "artificial intelligence" as it is "artificial affirmation."

Getting an unbroken flow of encouragement and support for our every belief, without any friction or correction, is – and I'm going to use a technical term here – Very Bad For Us™. We humans are simply not wired to live in an echo chamber of our thoughts. In part because we're often wrong. Sometimes out of genuine misunderstanding, other times out of delusion. The more we hear whatever we're thinking, the stronger we hold onto those ideas.

We need friends and experts to give us that "what are you on about?" face to shake us out of our manufactured realities. Knowing that we're wrong is how we update our mental models, which improves how we interact with the world. This is a Good Thing™ because it shows us to see the world as it is, not how we pretend it to be.

Sycophantic genAI chatbots disrupt that feedback loop and – just like the companies that create them – keep going. Which is why they sometimes lead all the way to the worst possible conclusion.

The big question

Is this genAI's fault, though?

No. And sort of. And also, yes.

No: Blaming the technology in this case feels intellectually lazy. For decades now people have tried, and ultimately failed, to definitively connect aberrant behavior to music and video games. And in doing so they've skipped past other, inconvenient potential causes. Or noticed that there's actually no problem at all.

Sort of: But it's hard to ignore the chat logs. They clearly show the bots reinforcing problematic behavior through constant agreement and encouragement.

It's not like this requires genAI, though. Think of all the execs, surrounded by yes-men, who go on to do some Very Foolish Things. There's also peer pressure, or its more well-intentioned form of The Friend Group That Is Too Supportive. A genAI chatbot is simply the digital version. Slightly less dangerous in that it doesn't represent multiple voices that can coordinate to spurn you on to action. Slightly more dangerous in that it's tirelessly available, 24/7, always offering an air of authority.

Yes: Excitement around genAI has given some people access to life-changing amounts of money, based solely on their role in the manufactured reality that genAI is indeed the end-all be-all. They must simultaneously emit and consume a constant stream of affirmation in order to drown out any naysayers. (And they will hire a new AI team when the incumbent group stops telling them what they want to hear. At least, we've heard hints of that happening.)

This entire field is fractal in its delusions. Bubble mechanics, applied to the human mind and the wider world.

Given that, we can reframe my question from before:

Do I blame AI psychosis on AI-the-technology? No.

Do I look askance at genAI companies that push technology while failing to establish even the most basic fucking safeguards? Maybe.

Are these genAI chatbots digital yes-men, born of execs who have surrounded themselves with flesh-and-bone yes-men? I'll leave that as an exercise for the reader.

From one pocket or another, it doesn't matter

AI chatbots are about to get ads. And given the information people bring up during these sessions – research interests, relationship troubles, health matters – I expect those ads will be Highly Targeted™.

I touched on this last May, in a segment on Meta/Facebook:

We all know why Facebook wants to give us virtual friends. The whole point of that place – like the point of so many apps out there, let's be honest – is to keep you on the platform. The longer it holds your attention, the more ads it sells, the more money it makes for its corporate overlords.

As a bonus, this idea sidesteps a privacy issue. Chats with friends are rich sources of information about who you really are and who you aspire to be, right? Well, FaceMeta can't be accused of eavesdropping on those private conversations if their bots are participants. You know, supporting you with "oh yeh you're so awesome" while dropping a Totally Relevant Ad for car insurance or orthodontists.

The company in question this time is ChatGPT parent OpenAI. (OpenAI blog, Les Echos 🇫🇷, Washington Post) Getting into the ads business is just the latest version of them changing plans. Consider:

• The company began as a non-profit. It is in the middle of converting to a for-profit entity.
• They were initially against ChatGPT generating adult content. Then they changed their tune and said they would allow it.
• Next, they said that they didn't want to run ads. Now they're about to run ads on their free tier.

(Plus, there is the question of whether they will use data from paid plans – which don't see ads – to train their ad system. But that's a story for another day.)

CEO Sam Altman once said that running ads would be "a last resort." When you consider the trillion-or-so dollars of debt the company has taken on, and the increasingly slim chances of paying it back anytime soon, you realize that they landed in last-resort territory a while ago. It just took them till now to recognize their surroundings.

The idea of OpenAI running ads dovetails with something I noted in November:

Seeing your typical genAI vendors through that lens, it becomes clear that they aren't really chatbot companies. They operate chatbots, but deep down these are token dealers. Everything they do is a game of Get People To Buy Tokens.

If you reframe "available compute capacity" (measured in time and energy) as "an inventory of tokens" (measured in word-pieces-for-sale-each-minute) you'll notice:

• This inventory is huge, because of all the compute power they have on-tap.

• That inventory is perishable, because every minute of unused computer power simply disappears into the past. It marks a loss on the balance sheet.

And when you have a huge, ever-growing inventory of a perishable good, you get very creative. Finding ways to repackage it becomes a survival skill. Adult content is just one such repackaging of genAI tokens.

I recently learned that an estimated 90% of ChatGPT users are on the free tier. Running ads is a way to repackage tokens as end-user attention, which OpenAI can sell to companies that want more visibility.

Companies like to say that they run ads to keep a product free. That's one way to spin it. I prefer the more direct version: You run ads when people won't buy your product outright.

If people want your widget but don't think it's worth paying for, then maybe someone else will think your audience is worth paying for. This is why "run ads" is and will always be the easy out for companies – it lets them build cool tech first and then figure out a revenue model later.

Recommended reading

I recently reread Michael Porter's "What is strategy?" article.

Thirty years on, it still holds.

I plan to touch on this more in a future newsletter. But if you have even a slight interest in business models, it's worth your read now.

In other news …

Here are some stories that didn't make it into any of today's segments.

• I've long noted the overlap between quantitative finance and ML/AI. Someone else agrees with that take, and they've written it all down. (Financial Times)
• Investments in genAI's exhibit risk of widespread financial contagion. Is anyone else here getting "too big to fail" vibes? Just me? (Bloomberg)
• French execs note that genAI hasn't had much of an impact on the bottom line. (Les Echos 🇫🇷)
• Microsoft CEO Satya Nadella states the obvious: genAI needs use cases or people will give up on it. (PC Gamer)
• The white-collar AI boom is driving a blue-collar boom for datacenter construction. (Wired)
• A newer ChatGPT model has been drawing source material from Grokipedia. (The Guardian)
• Using ChatGPT for search? Iffy. Using it to store your research? Bad. (Nature)
• Companies are using genAI to screen resumes. Some applicants are filing lawsuits. (New York Times)

For more recent news, and a slightly broader scope, I encourage you to check out my other newsletter. It also goes by the name In Other News and it is a weekly, curated drop of what I've been reading.