#040 - All hammer, no nail
GenAI companies keep cooking up wild use cases. Here's how to spot the winners.
You're reading Complex Machinery, a newsletter about risk, AI, and related topics. (You can also subscribe to get this newsletter in your inbox.)
Given all the summer holidays, and the summer travel that goes with, here's a shorter-than-usual newsletter. Just three quick segments.
Quite a rerun
Remember Watson? IBM pumped serious cash into that venture – I think an initial outlay of a billion dollars to build it, plus another few billion on some health care projects – then pushed very hard to make it A Thing™. Kind of a risky move as far as product development goes. But if you have deep pockets, you might just pull it off.
Did they pull it off, though? That's up for debate. And it also depends on the context. If you worked at IBM and played your cards right, Watson probably opened the door to really interesting research projects. Maybe it extended your career by a few years. Bonus points if you were already heading into retirement.
But from the outside, the marketing blitz always felt oversized relative to the press reports of client projects. Some of the ads were a stretch, too. It's like Watson was a solution in search of meaningful use cases. And for client dollars.
The Watson heyday was, what, ten or fifteen years ago? For today's version, replace "Watson" with "genAI" and "IBM" with "[insert any LLM company here]."
The big players built the perfect genAI hammer, but they forgot to first check whether there were enough nails around. And once they realized that wasn't the case, they started grasping for use cases like "put a hole in your wall" or "open that jar of pickles" or "repeatedly smash your thumb." Things for which a hammer is a terrible fit, or that we don't want at all.
Longtime readers have heard this from me before, but it bears repeating:
AI is pretty cool. In some cases, it's amazing. But it's nowhere near as good as the hype crews would want us to believe.
If it were indeed that useful, buyers – consumer and commercial alike – would be beating down vendors' doors to get it. We'd have uncovered so many use cases. Meaningful ones. Truly transformational ones.
But instead we have vendors who are so desperate that they push really, really hard on the marketing. In the hopes we mere mortals fall for the age-old mirroring tactic and we start to sing along in agreement. In a recent New York Times piece, Ismail Muhammad framed it as:
Why Does Every Commercial for A.I. Think You’re a Moron?
Ads for consumer A.I. are struggling to imagine how the product could improve your day — unless you’re a barely functioning idiot.
This is a sentiment I have never expressed outright (at least not in discoverable form) but I can now almost-say it by quoting someone else's work. Sort of like how I quote The Verge's Elizabeth Lopatto when she refers to a certain tech exec as "constitutionally bitch-made." But I digress.
I've been thinking about this more as I see a rise in the "AI hype is bullshit but AI is definitely gonna prove world-changing and useful" argument on social media.
I don't completely disagree with the premise. Strip away the media barrage and AI holds some potential, sure. But asking us to wave off today's widespread hype and failures because Someday, Eventually, This Will Be Huge is asking too much. It's a sentiment best shared in private, among fellow AI practitioners who understand that this has a ways to go before it's truly bragworthy.
Robot racing
More than a decade ago, in the early days of self-driving cars, I had a thought: If you want to see real advances in a technology, make it a sport. The competitive aspect will lead to investment, performance tuning, efficiencies, and all that – all because you can't hand-wave yourself into a victory. Games are won based on what the thing does today; you get no points for what you hope will work in a few years.
What I internally dubbed "algo-racing" has yet to materialize, because the autonomous vehicle hype sent the cars straight to the streets. But a couple of months ago I read about robots participating in a different kind of race – a foot race, in Beijing. (A city which I sometimes mistakenly refer to as Peking because French news outlets still use the term Pékin. I'm sure there's some rhyme or reason behind them using the old name, but hell if I know.) And now humanoid robots have squared off in both kickboxing and soccer.
The physical domain offers another testing ground for new technology: factory and warehouse labor. I first mentioned this in newsletter #024, which links to a must-read by Peter Eavis. More recently, Amazon has launched a new R&D group around AI-driven warehouse bots and various companies in China are exploring humanoid bots for manufacturing work.
My point: if you want to gauge how the latest AI tools are doing, look to fields where success is concrete and independently verifiable. "Did the robot move the box from Point A to Point B?" "Did this machine cross the finish line first?"
As a bonus exercise, you can then compare the concrete-success fields with the more hand-wavey set. Observe the difference between them. That will tell you a lot about how much hot air fills the AI hype bubble. And how far things stand to fall should it deflate.
Making everyone happy?
Content delivery network Cloudflare has entered the genAI training data debate. Sort of. They're establishing a way for genAI crawler-bots to pay as they hoover up online content.
On the one hand, this has the potential to shift the trawling-for-data discussion. While plenty of people have complained about the bots chewing up bandwidth as they steal lovingly read sites, resistance has been small and fragmented.
On the other hand, this makes Cloudflare the broker in a marketplace. It has to keep both parties happy if this is to work. That's a delicate dance, in part because one side has tons of incentive to simply not participate.
There's still a chance this could work out. Cloudflare has all the makings of a good middleman here. And maybe the AI companies, as they forge more licensing deals with big-name news outlets, will reach smaller players through a collective Cloudflare agreement.
Time will tell.
(For more on what it means to be a good middleman, here's a two-parter I wrote a while back on O'Reilly Radar.)
In other news …
- Fortune Ex Machina, my genAI fortune-bot, now has a home on Bluesky! Feel free to follow along if you're into that social media thing.
- A study by Carnegie Mellon says that AI agents are wrong about 70% of the time. (The Register)
- Arguing with the robot ref doesn't quite have the same feel, but that's the new deal at Wimbledon. (The Guardian)
- Remember how well genAI works as a search replacement? Neither do I. But everyone's Favorite Site Formally Named Twitter will use genAI to assist in "fact-checking." Given what the word "fact" means these days, I guess that fits? (Bloomberg)
- Looking for a job? Expect to spend time interacting with AI-based recruiter-bots. Can we call that "interacting?" I guess so. Maybe. (Washington Post)
- British Airways shows off some of its AI. (The Times of London)
- Turns out those AI bots are just being nice when they agree with you. (WSJ)
- Think twice before spraying "AI-powered" across your product. It may be a turn-off. (WSJ)
- The folks at The Verge are keeping track of robotaxi failures. (The Verge)
- In yet another example of my "algorithmic trading can tell us a lot about where AI is going" mantra, here's an example of bots haggling over price. It's not exactly HFT but it's pointing in that direction. (Technology Review)
- A book on machine learning apparently cites resources that were made-up by genAI. To be fair, the table of contents makes no mention of ML ethics, so … (RetractionWatch)
- Thanks to genAI bots both harvesting online information and producing summaries of said information, the web is eating itself. (Le Monde 🇫🇷)
- So, remember how execs keep bragging about how AI will eliminate roles? Turns out, um, we're not quite there yet. Some companies are actually hiring people to clean up after bots' goofs. (WSJ and BBC, respectively)
The wrap-up
This was an issue of Complex Machinery.
Reading online? You can subscribe to get this newsletter in your inbox every time it is published.
Who’s behind Complex Machinery? I'm Q McCallum. I think a lot about AI and risk, which I write about here.
Disclaimer: This newsletter does not constitute professional advice.