Bard, Bing, and the 90% problem
news analysis
Feb 15, 20238 mins
Artificial IntelligenceAugmented RealityChatbots
People heralding these next-gen AI chatbots as the future of the search seem to be overlooking a pretty hefty elephant in the room.
I don’t know if you’ve heard, but Search As We Know It™ is, like, totally on the brink of being changed forever.
Forever, gersh dern it! Did you process that?! THINGS MAY NEVER BE THE SAME!!
In all seriousness, if you’ve read much tech news over the past few weeks, you’ve probably been inundated with endless certain-sounding statements about how Microsoft’s move toward an AI-powered search setup and Google’s slightly more awkward stumbling in the same direction are about to usher in an entire new era of online searching — one where the tried and true pattern of hunting for information is completely replaced by a simple chat interface. Here, you just ask a bot a single question and then instantly get the answer you need.
Mind. Blown.
To be sure, this is some truly futuristic stuff. And it’s all based on wildly impressive AI technology that’s broken cover and crept into the public conscience over this early part of 2023.
But if you’ll allow me to take a slightly unpopular contrarian perspective, I’m unconvinced. Not that all this new AI chatbot hullabaloo will be transformative on some level, mind you — but that it’s ready to usher in a whole new era of search and make our good old-fashioned Googling instincts look antiquated.
Allow me to explain.
The ChatGPT search story
First, a quick bit o’ context to set the stage for all of this: This latest round of future-philosophizin’ got started with the sudden popularity of a tool called ChatGPT.
ChatGPT, in case you’ve been living under a 77-pound boulder lately, is an AI-driven chatbot that’s taking the web by storm. In a nutshell, ChatGPT lets you ask questions and get singular-seeming responses, as if you were chatting directly with another human — only this imitation organism has instant access to mountains of online info and is able to piece together original-seeming answers based on all of that data.
Oh, and it can also write stuff for you and perform a variety of data-based tasks, to varying degrees of effectiveness. But we’ll set that part aside for a moment — because it’s the search element of ChatGPT and its contemporaries that’s our focus right now.
After Microsoft co-opted the underlying technology from ChatGPT and brought it into Bing (which, yes, is actually still a thing!), Google rushed to announce a soon-to-be-public version of its own “conversational AI service” called Bard that’ll similarly let you search the web by typing to a bot.
And the promises around both ventures are bold, to say the very least. Microsoft says its AI-powered Bing-bot “reinvents” search and acts as a new “copilot for the web.” Google says its challenger “combine[s] the breadth of the world’s knowledge with the power, intelligence, and creativity of [the company’s] large language models.” Both are being positioned as pivotal twists in how we’ll think about search and what sorts of experiences we’ll encounter.
But here’s what those lofty assessments are overlooking: In terms of their search prowess, specifically, the experiences these new systems offer aren’t really a wholly new sort of info-seeking search alternative. Instead, they’re more advanced versions of the same core concepts Google’s been giving us for years now, with added horsepower under their hoods and an even more specific interface.
And as anyone who’s used Google Assistant can tell you, that sort of setup is spectacular for info-seeking — when it works. The problem is that oftentimes, it doesn’t. Not reliably, anyway.
With Assistant, the issue usually comes down to the service not entirely understanding what you’re asking or not being able to provide the right sort of answer. With these next-gen AI chatbot services — both the ChatGPT-based Bing search system and the Bard-based Google equivalent — the problem is even more disconcerting: The technology frequently serves up flat-out incorrect information. And it does so with an alarming level of confidence.
Surely you’ve seen the stories about this, right? Bing’s new AI chatbot gets all sorts of stuff wrong, ranging from the mundane — like flight times or basic facts — to the far more troubling) and sometimes just plain bizarre. Google’s Bard chatbot has had far less firsthand visibility outside of Google so far, but even the company’s own initial marketing material included an embarrassing factual flub-up — not exactly a reassuring start.
That gets us to the central issue with all of this technology, whether we’re talkin’ Google Assistant or any of these more advanced equivalents. And boy, is it a bigg’un.
[Get level-headed insight in your inbox with my Android Intelligence newsletter . Three things to know and three things to try every Friday!]
The AI chatbot percentage problem
I won’t sugarcoat this: When it comes to online interactions, getting something right even 90% of the time just isn’t good enough.
With search in particular, accuracy and thoroughness matter. One simple answer is fine — when it’s right. And when you can trust that it’s right.
But it certainly seems like right now, that’s anything but the case with any of this technology. Hell, Microsoft’s Bing-bot includes prominent disclaimers that it’s likely to provide inaccurate or incomplete information! And all novelty and cool factor aside, I just don’t see how that’ll make for an especially useful utility from a search context, for as long as that remains the case.
You need only look as far back as Google Assistant and its virtual voice assistant contemporaries for the proof. It’s really quite simple: If even one out of every 10 attempts at using something produces a flawed or for any reason unsatisfactory result, folks tend to lose faith in said thing pretty fast. And they then end up turning to another tool for the same purpose more often than not.
That’s why lots of us rely on Assistant for functional commands, which work fairly consistently — but when it comes to more complex searches, whether we’ve got Assistant at our beck and call on a phone or built into the core system interface on a Chromebook, we’re still more likely to go to Google to get an answer. Plain and simple, we’ve learned over time that Assistant just isn’t gonna give us the info we need a fair amount of the time.
It’s a well-known rule of technology: Something has to be pretty damn near perfect for it to be truly useful and worth using. More than a couple rare misses here and there, and it’s just not worth the effort for most people.
So far, at least, both the AI-powered ChatGPT Bing-bot and the Google Bard system appear to fall well within that “lots of misses” terrain — and with far worse than a one in 10 odds of failure, for that matter. That makes it tough not to remain a teensy bit skeptical about how broadly these types of tools will actually be used beyond the initial novelty period, as folks start to grow frustrated with the lack of reliability when it comes to actual information seeking.
(The other, perhaps more troubling possibility is that people might not always even realize the information is inaccurate or flawed in some way. But that’s a whole other can of worms for a whole other kind of column.)
And all of that’s to say nothing of the many search instances when a single conversational answer just doesn’t suffice — when you’re looking for images, reviews, specific info from different sources, or other such frequent needs.
Ultimately, it comes down to an almost shockingly simple pair of questions.
The pressing chatbot search questions
All oohing and ahhing aside, the questions we have to ask ourselves with all of this shiny new stuff is (a) if it’s really faster and more practical than traditional searching — not just once in a while and in specific limited scenarios but consistently— and (b) if regular people will actually want to use it over those tried-and-true tools once the initial newness wears off. Those are questions that are much more difficult to answer with a resounding “yes!” than all the excitement around this subject would lead you to believe, and the answers may be far more nuanced than our current giddiness suggests.
For context, remember: The tech world is always eager to declare things as “dead.” Companies and pundits alike love to speak in certainties about how every new development is a major game-changer and the start of a whole new era. We saw that happen with virtual reality and then augmented reality. We saw it with 3D screens and folding phones. We saw it with “the metaverse,” whatever the hell that even is (or was, more likely).
And you know what? With all of these alleged game-changers, the reality rarely ends up being anywhere near as dramatic or black and white as the initial exhilaration suggests.
As it stands now, one thing seems difficult to deny: For all the hype and excitement around these AI search systems, on the whole, you frequently still get better, more accurate, and more thorough results from a regular ol’ Google-style search.
And as long as that remains true at least 10% of the time, it’s tough to imagine Bing, Bard, or any other such system replacing search as we know it and, like, totally changing everything forever anytime soon.
Want even more Googley knowledge? Sign up for my weekly newsletter to get next-level tips and insight delivered directly to your inbox.