#026 - A brief chatbot detour
Bots say the darndest things. Why do we keep learning this the hard way?
You're reading Complex Machinery, a newsletter about risk, AI, and related topics. (You can also subscribe to get this newsletter in your inbox.)
I had plans for today's newsletter. Not exactly "big" plans, but "different" plans. I had a lot of things I was going to write about. And then Faceb– sorry, Meta did A Thing™.
So now I have to write about that Thing™.
As much as I want to thank FaceMeta for giving me something to write about, I … also wish they'd given me a better topic? Is that too much to ask?
These are not the chatbots you're looking for
In the previous newsletter's In Other News section, I mentioned that Meta had announced plans to put human-looking genAI bots on its social networks. Because they seem to think that people love chatting with bots.
This week several people stumbled across Meta-supplied genAI bots on Facebook and Instagram. "Wow, that was fast!" Or so we all thought. Apparently these were not the new bots, but some old bots that were still wandering around without adult supervision.
We know this because people interacted with the bots. The most noteworthy of which was "Liv" (username "himamaliv") – a self-described "proud Black queer mama of 2 & truth-teller ❤️🌈 Your realest source for life's ups and downs."
Some of you are already conjuring up ideas of what a conversation with "Liv" would entail. You imagine the bot making some truly cringe-worthy statements, the kind that drive home the fact that AI chatbots are fancy blobs of linear algebra incapable of self-awareness or social skills. Rest assured, the reality was much worse. Washington Post reporter Karen Attiah shared screencaps in a Bluesky thread as proof.
I was about to write "it was really something else." It was, and yet, it … was not? To be clear, Attiah did solid work interacting with the bot. I was surprised to see just how ridiculous things got. But I was not at all surprised to see that yet another genAI chatbot had gone off the rails. These days I'd be more surprised by a bot that stayed on the rails. Because that would imply that the bot had a concept of rails.
(Meta has since removed the bot profiles. Given the fanfare, though, "oh those were the old bots kthxbye" is precisely what I'd expect a company to say if the new bots turned out to be trash. But I digress.)
I almost feel bad for Meta, because anything they do will come under scrutiny and people will suspect the worst. They can no longer make innocent mistakes without it becoming conspiracy bait.
But then I remember that this is Meta. So. There's that.
You already know where I'm headed next: laugh if you want, but make sure that your own house is in order. Because, dear reader, every public-facing goof committed by a well-known company is a learning moment for us all. Here's what Meta can teach us:
Big players still goof. My main disappointment of genAI is that major players – big names, with big budgets, and big big AI talent pools – keep stumbling over problems. Entirely foreseeable and preventable problems. Meta should have known that these bots were not ready for prime time. Yet they went ahead and released them anyway.
Big players' goofs lay the groundwork for smaller players' goofs. Smaller companies lack FAANG-sized money and access to talent, sure. But the bigger issue is that they love to copycat the FAANGs' every move. Remember when everyone was rushing to set up Hadoop clusters? Even when their data was barely a blip in their plain old relational database? We're going to see that same sad tale play out with genAI.
(Independent thinkers will avoid that pain. But expect them to be few in number.)
Risk management still matters. It's not enough to have brainiacs build the bots. You also need brains behind your risk management. And for genAI bots, that means round after round of scenario planning, testing, and red-teaming – all of which should lead to filters and other protective measures. If you're doing this right, you'll have the occasional no-go decision because the bot has proven too unstable for public interaction.
(When a company skips over risk management, they're saying that they don't care about the outcome. Which, if you are buying services from such a company, should tell you something.)
That leads us to the next two points:
Never let the machines run unattended. I often point out that an AI model is a piece of factory equipment – one that emits text or decisions instead of widgets. Just like you have minders and monitoring systems for machines on the factory floor, you need extra eyes on whatever the models produce.
Red-team your bots. Because if you don't, someone else will. Sometimes without meaning to. By releasing a chatbot to the public, you're saying that you want people to interact with it. That you're ready for them to interact with it. So when everyday interactions surface major bugs, I have news for you: those aren't edge cases. They're problems you chose to gloss over during product planning, and they've come back to haunt you.
We're going to see more bots. More terrible bots. I noted in the previous newsletter that AI companies shift most of their chatbots' risk to the end-user. On top of that, buyers and investors still think bots are cool, so they are throwing money at anyone building one. This creates a minimal -downside, all-upside scenario for companies: building a genAI chatbot is still a good idea, even if it is also a bad idea. It's no surprise that so many of them give it a go.
To sum it up, I'll remind everyone of a point I raised back almost a year ago: every genAI chatbot is a wild animal. Pretending it's tame is how you wind up on the wrong end of the pointy bits. But since there's almost-guaranteed money involved, the claws will continue to come out.
This sounds familiar
Companies creating genAI chatbots are excited. The people meant to use the bots, less so. It's not that people aren't interested at all. But Meta's social media bots are yet another reminder that there are limits. And when a company is more excited about a technology than the people intended to use it, that's a red flag.
Remember voice assistant devices? Alexa and Google Home were going to be all the rage. And they were! Until the novelty wore off. Because people realized that they preferred tapping into their phones rather than issuing voice commands to a glorified desktop speaker. Oh, and it turned out that the companies behind the devices hadn't really sorted out how to monetize the things. So they trimmed their aspirations. As well as their voice assistant teams.
Part of why this keeps happening is that the companies behind these ideas have deep pockets. It's relatively easy for them to get something off the ground and then market the hell out of it. If it takes off, great. If it doesn't, no worries; they can eat the loss and try the next hot thing to come along.
All of this to say: genAI bots won't be the last of the foolishness. Next time around I'll (finally) get around to covering "agentic AI" and you'll see what I mean.
A little bit of everything
Have you noticed that companies these days have become every kind of company? They're all doing a little bit of everything, in order to keep a finger in every possible pie.
I first noticed this early in my career, when I built custom software. Whatever the company's main mission, they were now also a software company. Which means they had to learn about software dev best practices in order to survive. (Some chose to learn this the hard way. But that's another story.)
The same cycle has played out many times over, as every company has since become a data analysis company, and then a data broker, and now a genAI promoter.
Which, I guess, is fine? We can utter a smug "jack of all trades, master of none" but if a company is large enough, it has a chance of getting away with it. Maybe.
But to do so, they still need to learn best practices for the job. Said practices may seem like needless friction. But they represent hard-learned lessons about what works and what doesn't. Which is a key element of managing risk.
I'm not sure why this comes to mind right now. Not at all.
Oh hey, remember Volkswagen? The car company? They now have lines of connected cars. Which means the "car" company is now also a tech company, plus a data company. And they've experienced a massive data leak. Not a breach. That's when ne'er-do-wells run off with the jewels. I mean a leak, because someone missed the memo on best practices for data storage.
(Credit where it's due – I first read about this in Der Spiegel 🇩🇪, which had partnered with a research group to conduct a thorough analysis of the incident and what was in the data. Ars Technica dropped a short note about it a couple days later.)
What data did VW leak? Oh, nothing special. Just about 800,000 vehicles' location history. And for some of those vehicles, the owner's contact details. Nothing to be worried about, right?
It'll be interesting to see how this plays out. On the one hand, data leaks have become so common that they barely raise an eyebrow. On the other hand, this happened in Germany, so EU data privacy laws may shape how regulators handle this.
(Not) getting the word out
A couple months ago I mentioned that Bytedance, parent company of TikTok, had allegedly fired an intern for having poisoned some AI models (as reported by The Register, PCMag, and others).
Model poisoning is an underappreciated risk in AI: it goes beyond the usual concern of "the model will be wrong now and then" and straight to "the model will sometimes be wrong on purpose." That's an even scarier prospect when you consider how models feed into downstream business processes, sometimes having ingested records from other models, and so on. One poisoned model can have widespread impact yet be almost invisible in a sea of automated decision-making.
I haven't heard much about this beyond the initial news reports. Bytedance has been fairly tight-lipped about the affair. That's unfortunate. While no one likes to air their dirty laundry, I'd expect plenty of companies are facing the same problem.
AI is such a new concept that the entire field is still sorting out best practices, and that includes areas around risk mitigation and internal threat management. Sharing information would serve to accelerate the creation of those practices, making AI smoother and safer for all involved.
In other news …
- It's not just Volkswagen. Plenty of newer cars are roving data collectors. Including Teslas. (Washington Post)
- Famous producer Timbaland talks about AI's impact on the music industry. (Bloomberg)
- AI has made some big promises. Investors say the bill's soon to come due. (Les Echos 🇫🇷)
- Israel's use of AI in warfare raises questions. (Washington Post)
- Samsung wants more AI in appliances. Have they asked consumers what they want? (Bloomberg)
- Here's a quick look at how some early adopters are using AI agents. (WSJ)
The wrap-up
This was an issue of Complex Machinery.
Reading online? You can subscribe to get this newsletter in your inbox every time it is published.
Who’s behind Complex Machinery? I'm Q McCallum. I think a lot about AI and risk, which I write about here.
Disclaimer: This newsletter does not constitute professional advice.