#025 - My risk, or yours?
A subtle risk/reward tradeoff sheds light on AI product companies.
You're reading Complex Machinery, a newsletter about risk, AI, and related topics. (You can also subscribe to get this newsletter in your inbox.)
People and machines
A Character AI incident has put genAI companions back in the news. One of their bots allegedly told a teen to commit acts of violence against their parents. This is not to be confused with a different Character AI bot telling a journalist, who was posing as a teen, how to murder someone and hide the evidence. Both of these add to the list of AI Chatbot Companions Gone Wrong, which includes people developing very strong, often worrisome relationships with bots and at least one allegation of leading a teen to commit suicide.
The trouble isn’t much of a surprise. With even a little knowledge of AI, it's clear that genAI companion bots are a 99.999% Bad Idea™. Perhaps the technology will improve in the future. Perhaps. But today's bots lack socioemotional context, they don't genuinely "understand" the words we type, and the responses they offer don't stem from genuine emotion. They exhibit plenty of randomness out of the box, and can cause even more chaos when their parent companies update the supporting models. Especially when those updates are hasty reactions to mishaps.
GenAI companion chatbots are ahead of facial recognition as far as terrible AI use cases. That's saying something.
While those two fight over the top spot, companies issue plenty of other AI products that fail to live up to expectations. It's like they didn't see this image I ran in newsletter #018, about the sweet spot for building AI products:
AI companies are riding hard on the "Here Be Dragons" quadrant while giving "Perfection" the short shrift. What gives?
The answer lies in the friction between those ideas. Which in turn sheds light on an underappreciated risk exposure.
Robots welcome … or not
Let's start with the places where AI robots are expected to help in the near-term. Warehouse work and food prep top the list as they fall under the usual criteria for automation – my "dull, repetitive, predictable" or the military's "dull, dirty, dangerous." People will eagerly hand these tasks to machines. Efforts in these arenas, while hardly perfect, are already bearing fruit.
Then you have the cases of AI competing with people. Like the radio station that replaced its on-air hosts with genAI personalities. Individuals in the advertising industry have mixed feelings about AI but their employers – ad agencies – love it. This includes the shadier side of online advertising, which has used genAI to create product reviews and even fill entire news sites with generated articles to sell ad space.
But despite claims of outright replacing human labor, AI's invasion of the corporate workplace is more of a human-machine partnership – a game of "keep an eye on the bot." Groups like French newspaper Le Monde and Japanese manga outfit Orange have turned to AI to perform the first step in translation. And I emphasize "first step" there. A similar story plays out in software development, where genAI assistants create a rough draft of an idea that experienced developers refine and correct. The general consensus is that genAI bots can actually be helpful so long as they play the role of an assistant or intern.
This approach follows my standard guidance of "never leave the machines unattended." And it works.
AI, unfiltered
Given that, you'd think companies would focus on the "serve the experts" market. It's a way to get AI and people working in concert rather than in competition. The experts' cleanup follow-up work compensates for the bots' foibles. And those experts seem impressed by the way the bots save them time and energy.
(A note to entry-level hires: You certainly aren't impressed, as these expert-helper genAI tools pose a threat to your role. But vendors only care about pleasing people who have budget authority. Which means they don't care about you. Sorry to be the bearer of bad news.)
In spite of that, AI companies are just as eager to seat the genAI tool, unfiltered, right next to a non-expert end-user. That arrangement implies that the bot is a peer. Or worse, an authority figure whose words should be taken seriously enough to support one's position in a debate.
We first saw this with AI summarization tools attached to search engines. Google's "glue on pizza" fiasco was the most newsworthy of the bunch, but there have been others. And there will be even more, now that AI companies are eager to get into the search-summarization market and people are turning to genAI bots in lieu of a proper web search.
More recently Apple Intelligence, the genAI built into iOS 18, experienced its own summarization gaffe when it botched some BBC News headlines. The problem? iOS presented the (incorrect) summary next to the BBC News app icon and name, making it look as though The Beeb – a far more credible source than "some AI bot" – had made the erroneous statement. The generated summary skipped past the expert journalists, who could have fixed the goof, and went straight to the end-user.
Still, robot companion apps take the crown in the Unfiltered AI™ championships. Individualized human attention was already tough to come by, then onset of the Covid-19 pandemic pushed that even lower. I get why AI companies would try to fill that void with an automated service. And given the bots' popularity, it's clear that people want them. But the bots aren't up for the job.
It's not just that they're bots pretending to express empathy and emotion. It's that their target market perceives them as being more human and trustworthy than they are capable, putting the bots in a role better filled by a human confidant. The resulting problems are painful to witness, yes. But they were also very clear in the distance.
My risk, or yours?
All of which brings us back to risk.
Remember that risk is a two-sided affair – upside gain and downside pain go hand-in-hand. But this isn't a matter of chance. You’d do well to choose risk/reward tradeoffs that work in your favor, where the potential upside gain outweighs the potential downside loss.
Given that, have we made a wise decision about AI? Are we being properly compensated for the risk we're taking on?
The answer depends on where you sit:
As a consumer of genAI – and this holds for any user of AI-based products, be they individuals or corporations – you’ve drawn the short straw. GenAI doesn't provide you much benefit, and you have little recourse when it falters.
This goes double when you are subject to AI-based systems, like facial recognition or custom scoring techniques. You rarely get the choice of whether to use AI in those cases, and yet it's tough to hold providers to account.
If you're building AI products, you're winning. You operate in a legal gray area in which your products only kinda-sorta have to work. Mix that with a tight terms of service agreement (TOS) and you shift the downside risk to the end-user while keeping the upside gain for yourself. You can probably wave off the few lawsuits that come your way. All you have to do is stomach the public outcry.
(I've been told, money does a great job of blocking out negative news. You can cover your eyes, your ears, and sometimes even your conscience. Assuming you still have one.)
As with so many risk exposures, it's a race against time. New laws or a groundswell of consumer ire can eventually shift the tide against you. Case in point: the FTC is already pursuing companies for overstating their AI products' capabilities. But what are the chances they'll get to you, with so many other companies cranking out AI products?
If you can't beat 'em …
You might think that's a gloomy place to end a newsletter. It's a matter of perspective.
Remember: genAI is hot right now. Not only is it hot, but players in the space have a vested interest in keeping it hot over the long haul. It's no accident that "agentic AI" – which I'll touch on in the next newsletter – landed right as the disappointment in AI chatbots really set in.
(Expect another pivot once agentic AI fades. And so on after that one. It's just the latest twist in the "rename the data field to keep the dream alive" game.)
I hate to encourage bad behavior – and remember, this newsletter does not constitute professional advice – but if you want to ride that wave, all you have to do is spray-paint "AI-powered" on the side of your product idea. Any idea at all! You can build half-working, sometimes-dangerous bots. In lieu of addressing problems in advance, you can wait until obvious failures occur before adding guardrails. Investors and corporate customers will still fight each other to bring dump-trucks of cash to your door.
As for the regulators? Well, the flood of AI startups will keep them busy for a while. You can try your hand in the meantime.
As a startup founder, the AI lottery costs you next to nothing to play. You stand a good chance of winning, and a slim chance of any hardship if you lose. Risk 101 says that this is a Good Deal™. How could you not try it?
All you have to do is cash out before reality sets in. Or, as so eloquently framed by one fictional startup CEO: "Sell your house of cards while it still looks like a palace."
In other news …
- Faceb– sorry, "Meta" – wants more bots on social media. Sounds like a bad idea for end-users but, hey, end-users don't (directly) pay Meta's bills. (Financial Times)
- In February, philosopher Eric Sadin will hold a kind of anti-AI summit in Paris. This may read like a publicity stunt on the surface, but it makes sense to create a space for people to raise their concerns. Remember that speculative bubbles grow, in part, because naysayers' voices get drowned out. (Le Monde 🇫🇷)
- Volkswagen has experienced a sizable data leak. And by "leak" I mean "it left vehicle location data, plus some car owners' details, in an unprotected AWS cloud storage bucket." Since this happened in Germany it'll be interesting to see how EU data privacy law plays into regulators' reaction. (Der Spiegel 🇩🇪)
- Therapists weigh in on people using AI chatbots for therapy. (CNN)
- You've heard of "SEO poisoning?" According to researchers, ChatGPT's search tool is vulnerable to similar attacks. (The Guardian)
- We can blame a lot of things on bad AI. But sometimes, AI is just along for the ride on a bad idea. Like, say, weird videos. (The Verge)
- Snack maker Mondelez is using genAI to prototype and test recipe ideas. I genuinely like this approach, but I can't resist linking to everyone's favorite "Oreo CEO" video all the same. (WSJ)
- Police departments are considering genAI bots to draft reports. The ACLU is not amused. (The Register)
The wrap-up
This was an issue of Complex Machinery.
Reading online? You can subscribe to get this newsletter in your inbox every time it is published.
Who’s behind Complex Machinery? I'm Q McCallum. I think a lot about AI and risk, which I write about here.
Disclaimer: This newsletter does not constitute professional advice.