#038 - Exploiting our weaknesses
When genAI chatbots and tools get into our human wiring.
You're reading Complex Machinery, a newsletter about risk, AI, and related topics. (You can also subscribe to get this newsletter in your inbox.)
AI has given the world a lot to talk about over the past couple of weeks. Here's a summary of what I've explored thus far, laid out in four loosely-connected segments.
(This would have been six segments, but I ran out of room. I'll cover the other two topics – one of which is the Builder.AI kerfluffle – next time around.)
Just one more spin of the wheel
There's a long-running joke in software dev circles, in which someone proudly spends two days automating a two-minute task. It's really half-joke, half-serious – funny but also true.
Sometimes that's a fair price for a business to pay – like when the task is to be performed ad infinitum. Automating it will save tons of effort over the long run and make it more consistent. That's a win-win.
Other times, it's not about reducing the marginal cost of that task to zero. It's about developers becoming obsessed with trying to automate something and losing perspective on the larger goal. A real forest-for-the-trees situation.
I thought about that when reading this blog post, which draws a parallel between genAI code assistants and cocaine. The author overlays their argument with the model from Nir Eyal's Hooked, a book on embedding habits into a product:
First, the trigger is what gets you in. e.g., you see a chatbot prompt and it suggests you type in a question.
Second is the action — e.g., you do ask the bot a question.
Third is the reward — and it’s got to be a variable reward. Sometimes the chatbot comes up with a mediocre answer — but sometimes you love the answer! Eyal says: “Feedback loops are all around us, but predictable ones don’t create desire.” Intermittent rewards are the key tool to create an addiction.
Fourth is the investment — the user puts time, effort, or money into the process to get a better result next time. Skin in the game gives the user a sunk cost they’ve put in.
Then the user loops back to the beginning. The user will be more likely to follow an external trigger — or they’ll come to your site themselves looking for the dopamine rush from that variable reward.
And, a little later in the piece, the author ties this back to code assistants:
With ChatGPT, Sam Altman hit upon a way to use the Hook Model with a text generator. The unreliability and hallucinations themselves are the hook — the intermittent reward, to keep the user running prompts and hoping they’ll get a win this time.
This may seem far-fetched – and to be fair, the post has a humorous bent to it, so it's meant to sound over-the-top – but the premise contains a kernel of truth. The dogged desire to solve problems is very real in software development circles. Plenty of people who write code for a living have stories about latching onto a problem and staying up till the wee hours working on it, unable to let it go.
You can imagine what happens when this mindset meets a system that generates code based on a written statement. The developer keeps fine-tuning their prompts – clarifying the request here, correcting the machine there – in order to get the desired code out of the machine. We're right back to the joke about spending an inordinate amount of time trying to automate something. Extreme cases plunge into Gambler's Ruin territory, in which the developer is repeatedly convinced that this next tweak will make up for all the time and effort they've burned up till now. Well, not that tweak, but the next one. And the next one.
To be clear, the code assistant/narcotics connection doesn't hold for everyone. This is about a certain subset of the population, one that is hard-wired to keep poking at the machine until it delivers the desired result. GenAI code assistants have struck those developers' resonant frequency, convincing them to spend far more time than is necessary looking for That Perfect Prompt.
(For a deeper look into this I recommend Natasha Dow Schüll's Addiction by Design, a study of the relationship between gamblers and the gambling industry. In short: gamblers want to gamble; casinos give them ways to gamble. They both go home happy. But only one walks off with money.)
Every business wants that sweet, sweet recurring subscription revenue. A product that taps into a developer's drive to Keep Working On It is bound to generate cash! Because what is an addiction, if not a subscription that the user will never terminate?
To wrap up, there's another part to that old half-joke: the developer bangs their head against the wall long enough that they're forced to step away for a moment. And it's only when they let go of the problem that they figure out how to solve it.
I expect we'll see a similar story play out with the genAI code assistants.
Your periodic reminder to check the outputs
If you've read more than one issue of Complex Machinery (or had the misfortune of speaking with me one-on-one) you'll know that:
GenAI bots occasionally spew nonsense. And as a result …
… "check the bots' outputs" is a key risk mitigation strategy.
There are other mitigation strategies, like "make sure you even need genAI for this task." But "check the outputs" runs a close second or third.
There are two problems with this approach, though:
1/ People don't usually do it, in part because the genAI companies – ever eager to tap into the lucrative search market – keep touting the technology as a suitable replacement for search engines.
That leads to multiple cases of people – including legal professionals, who should know better – citing nonexistent cases or papers in their work. Or people asking it about political issues (and getting upset when the bot declines to answer). Or smugly proclaiming that a genAI bot will cut through tax and immigration paperwork.
2/ On the rare occasions when people actually look at the machine's outputs, they take it as gospel. People just love to trust what genAI tells them. If a human being were to tell them the exact same thing, they'd demand support for the claim. And then ignore it, because they don't like how it sounds.
Again, the genAI providers shoulder some of the blame here. They'll claim their bots aren't suited for certain purposes – like, say, therapy – but their policing of those bots has been rather meh to say the least. This is especially troubling when the end-users are in emotionally vulnerable states and the bots convince them that the real world is anything but.
(GenAI will soon learn, like Big Tobacco and dope dealers before them, that an addictive product is no good if it kills the customer.)
This isn't just about genAI. It's a common problem with screens in general. People listen to computers too much and too easily. And they forget that those screens sometimes show mistakes. Like, say, marking highways as closed when they are open. Or guiding a food delivery driver through a secured airport entrance and right to the tarmac. Yes.
But given how it's sold to the masses, and how the bots can be so convincing at times, genAI represents a special concern here.
When it comes to AI, skepticism is an underrated risk control.
Soon. Very soon. Just don’t hold me to that.
Longtime readers will recognize this phrase: I started writing about Topic X, but then something else came up, and I had to switch gears. This segment is no exception.
This week Anthropic CEO Dario Amodei has been telling news outlets that AI is going to wipe out a bunch of white-collar jobs. Here's a fun excerpt from a CNN article:
“AI is starting to get better than humans at almost all intellectual tasks, and we’re going to collectively, as a society, grapple with it,” Amodei told Cooper. “AI is going to get better at what everyone does, including what I do, including what other CEOs do.”
Right as I asked myself how he substantiates that claim, the next paragraph had the answer: he doesn't.
To be clear, Amodei didn’t cite any research or evidence for that 50% estimate. And that was just one of many of the wild claims he made that are increasingly part of a Silicon Valley script: AI will fix everything, but first it has to ruin everything. Why? Just trust us.
This story sounded so familiar, I could've sworn I'd already read it. Turns out I was confusing it for an April Axios interview with Anthropic's CISO Jason Clinton, in which he describes how virtual AI-based "employees" will operate on company networks. (I suppose this AI Will Replace Everyone, And We Mean Everyone spiel is Anthropic's new marketing campaign.)
Axios's bread and butter is short, summary-format articles, so I don't have details on precisely what Clinton said. But it sounds like this was his take on what might happen in a year. A year? That's practically an eon in the emerging-tech space. And that flavor of hazy future-speak dovetails with something I noted back in October:
AI vendors keep getting tripped up on their Fake It Till You Make It routines, so they've changed tactics. They're moving away from concrete promises for today, and – taking a page from the cult playbook – declaring future dates for when things will pay off. "It's gonna be amazing in just a couple more years. You will be rewarded for your belief. Trust me. And also, pay me." This gives them extra time to turn their bold-yet-empty proclamations into something real. Or to wriggle out of the promises if things don't pan out.
(You know how cults always find some excuse when the big date passes without fanfare? With AI, they'll just rename the field and start over. Again. You heard it here first.)
The overall idea of what Anthropic is saying makes sense: technology is all about automation; automation eats work; and technology tends to improve over time. So it logically follows that AI should eventually become more capable at certain tasks.
The attorneys among you will recognize all of the couching language in that last sentence.
Calling that level of automation "virtual employees" is a bit of a stretch. Most of what the article describes – technology that keeps track of events and takes action – already exists today. We don't use the term "agents." That technology is built on plain old (deterministic) code, with a splash of ML thrown in for good measure. But this is already A Thing™. While genAI would represent a next step in workplace automation, that's hardly a quantum leap.
Long story short:
When someone tries to sell you on what AI might do a year from now ...
... ask yourself what that crowd said AI would be doing by today.
They could step out of the fray
Reactions to Apple's annual showcase, the Worldwide Developer Conference (WWDC), have been uncharacteristically lukewarm this year. Apparently people are frustrated that there's not enough AI in the upcoming iDevices. Or maybe they're just sad that Apple's big AI talk has delivered lackluster results. Remember the "iOS summaries aren't accurate summaries" incident from a few months back?
If you ask me – and before you say "actually, I didn't ask you," remember that you're reading my newsletter so you kinda sorta did ask – Apple is going about this all wrong. They're rushing into genAI along with everyone else. Trying to keep up with the pack, even when that pack is headed over a cliff.
Apple is missing a chance to formally, publicly pull out of this foolish AI race. I'm not suggesting they give up on AI. I'm saying that they could stop shouting about AI and focus on what matters: building amazing experiences with their apps and hardware. They'll probably find some interesting places to use AI along the way. But that needn't be front and center in their marketing efforts.
I've noted elsewhere that most consumers don't give a damn about the underlying technology. They just want something that works well, and that they like. If the AI under the hood is doing its job, no one will notice it's even there.
Recommended reading
Barry Ritholtz, author of How Not to Invest, recently published a blog post on how to size up investment advice. It doubles as solid guidance on buying or implementing AI-based solutions in your company.
Investing is all about decision-making when there's money on the line. And if you see AI as an investment, Ritholtz's words will prove useful.
A little fun
Complex Machinery has covered a lot of GenAI Gone Wrong and GenAI Doesn't Really Do That. It was never my intent to spend so much time on those topics but, hey, that's what the world keeps giving me. So that's what I write about.
To shift the balance, I’ve built a little something I call Fortune Ex Machina. It uses genAI to create short text suitable for fortune cookies. Feel free to click that "random fortune" link till you get one you like …
In other news …
It's final exams season in China. In the same week the government temporarily blocked students' access to genAI systems to reduce cheating … (Le Monde 🇫🇷)
… OpenAI is trying to get more college kids using ChatGPT. Make of that what you will. (Gizmodo)
Do you hate the way that Google is cramming genAI into services? Well, be prepared to hate them even more when they create AI-driven summaries of your e-mails ... (Ars Technica)
… and short, podcast-style audio summaries of search results. (The Verge)
ChatGPT meets its match: the Atari 2600. Yes, that Atari 2600. (Der Spiegel 🇩🇪)
Canada stakes its claim as an AI research powerhouse. (Les Echos 🇫🇷)
A friend recently shared this leadership-level guide to handling crises. This has no direct connection to recent AI news, but … methinks a lot of companies will need this soon. (The Economist)
We've already seen AI models lie and cheat to meet their objectives. Here's the latest example. (The Register)
Facebook – I mean, "Meta" – is building a dream team in search of AI "superintelligence." Sure. Because that whole "Metaverse" thing went over so well. (New York Times)
As though charlatan influencer-types weren't bad enough, we now have them in genAI form. (PressGazette)
The UK has had enough of attorneys citing nonexistent, AI-generated cases in their work. (TechCrunch)
The wrap-up
This was an issue of Complex Machinery.
Reading online? You can subscribe to get this newsletter in your inbox every time it is published.
Who’s behind Complex Machinery? I'm Q McCallum. I think a lot about AI and risk, which I write about here.
Disclaimer: This newsletter does not constitute professional advice.