How AI can Loosen Google's Grip on Search
Implications of Ubiquitous Access to AI-powered Search
Google is dead.
Long live Google + “site:reddit.com”.
https://dkb.io/post/google-search-is-dying
Earlier this year we had a moment where the deteriorating qualities of Google search broke into the mainstream. Blogger dkb articulated a feeling that many of us seem to have—Google search results are become increasingly inauthentic. Trustworthy information has become a casualty of Google's advertising business model as the incentive to put commercial content has increased considerably over the years. Despite all these complaints, why do people still keep going back to Google?
One theory is that it's the default and it's good enough. Despite all the complaints about Google, it's a hassle for most people to dig into settings and switch the default search engine. User experience research has shown that 95% of people stick to default settings. Google gained dominance in a world where people first typed out the full domain of "www.google.com". In that world, the effort to type "www.yahoo.com" vs "www.google.com" was comparable so there wasn't much of a switching cost for people to type "google" instead of "yahoo". Today we start our search directly on a browser. Google pays upwards of $15 billion to Apple for the privilege of being the default search engine on Safari browsers. Given that Google hovers around 91% market share according to SimilarWeb, the rule of thumb that suggests 95% stick to the defaults seems to check out.
How about the second part of this theory, is Google truly good enough? In many ways Google is still leagues ahead of other search engines when it comes to performance and user experience for factual searches. Google with its economies of scale can serve results quickly and crawl through the newest webpages with breakneck speed. As a large company, Google can also invest heavily in crafting experiences to address some of the high frequency searches like the news and weather as well as high value searches like flights and hotels. The realm of common factual queries is where Google shines.
Cracks begin to show when results are less factual and more subjective. This is especially true for serving quality content within specific verticals. Here’s a collection of quotes from the dkb article making this argument:
Ask HN: Has Google search become quantitatively worse?
The results keep getting "refined" so as to suit the popular 80% of queries, while getting much worse for any technical or obscure queries. Forced synonyms and "people also searched for" are typically useless and almost infuriating. Once you get off the first or second page, the results get even worse-- with pages entirely unrelated to the query (e.g. not even containing the searched phrases). They are probably testing/already implemented some sort of multi armed bandit type optimization like on Youtube's search results where they just show any popular pages (ignoring relevancy) to see if they yield a click.
A recent small medical issue has highlighted how much someone needs to disrupt Google Search. Google is no longer producing high quality search results in a significant number of important categories.Health, product reviews, recipes are three categories I searched today where top results featured clickbait sites riddled with crappy ads. I’m sure there are many more.
This may not just be a problem with Google but possibly also the recipe for beating Google. A startup usually has to start with a niche market. Why not try writing a search engine specifically for some category dominated by SEO spam?
I'd like to linger on that last quote from Paul Graham. I believe with ubiquitous access to AI models, online businesses will have just the right ingredients to take on Google. On December 15th, 2022, OpenAI released their latest embedding model and within a day we saw AI search in action. Instead of getting health tips from bland listicles, you can get recommendations from Dr. Huberman himself (or rather the AI rendition of Huberman).
We can now organize the world's information without the help of Google. Given all this, let's wrap up with three predictions.
1. Companies with differentiated content will build their own search
Those with differentiated content and clear brands have already taken some of Google's market share when it comes to vertical specific searches. This happens mostly with other aggregators like Amazon.com where a majority of US consumers begin their product searches. The thing is, this usually works best when the items being searched for are well defined like a product SKU that comes with plenty of metadata for search. This challenge of indexing relevant content becomes much harder with pages of natural language text which is most of the web. A classic example of a company that struggles with this is Reddit.
A big reason why people use Google to search Reddit is because Reddit's search sucks. This means adding "reddit" to the end of Google searches remains the dominant way people search Reddit, until now. Soon, websites that previously had difficulty indexing all their content can tap into AI and quickly surface all the relevant threads as well as individual comments for a given search. AI is now clever enough to understand the semantic relationship of words and doesn't need extensive tagging and categories to index content. Overtime, the gravity of search traffic will shift and so will revenues from search ads.
2. Curating information will give you superpowers
Every couple years or so I go through a phase where I obsess over note-taking. I read a bunch of articles on note-taking from places like Tiago Forte's Building a Second Brain and spend hours tweaking the hip new note-taking tool of the day. The reason why I obsess over note-taking can be best summed up by Tiago Forte's famous Building a Second Brain course.
A Proven Method to Organize Your Digital Life and Unlock Your Creative Potential.
https://www.buildingasecondbrain.com/
For me and many others, note-taking is notoriously hard to keep up with and the costs outweigh the benefits. Even bookmarking information rather than jotting down notes yield little reward. How often do we hoard tabs, buy books, and download podcasts of interesting material that we never read or listen to? The feedback loop of useful insights retrieved from notes and bookmarks happens too infrequently for most people to justify the time spent. Most people don't want to take a pricey multi-week course or assemble a whole suite of thinking tools like Linus. People want the benefits of a second brain without having to build one.
Soon, information you consume or bookmark can be quickly catalogued and be called upon whenever you need it. I've actually never heard of the Huberman Lab Podcast until I saw Dan Shipper and Riley Tomasek build their Dr. Huberman chatbots. Usually when I discover a new podcast I'd start listening to the back catalogue. This time I was able to jump directly onto the Huberman AI chatbot and get the information that I wanted right away. Instead of reading books, listening to podcasts, and watching videos, we can now upload text and transcripts directly to an AI powered tool and instantly access all the relevant knowledge in a body of work. The bottleneck in personal knowledge management isn't capturing the right information but rather organizing it and retrieving the relevant parts when needed and this wave of AI solves exactly that.
3. Google will adapt or die
There's also a good chance that Google will begin to look quite different as well. Google already has recognized that the best user experience cannot be served by simply returning blue links. Instead, we're seeing more use case specific search experiences.
Perhaps Google might realize the best way to surface research is through AI chatbots that summarizes findings, remembers context, and cites sources. Perhaps Google has had this ability all along but chose to withhold it since showing a list of links along with ads maximized profits. One thing that I am confident of is that search as we know will change.
Pre-Publish Updates
Developments surrounding generative AI are so rapid that I had to squeeze in a couple updates before I hit publish.
Re: Prediction #3 Apparently there’s a “Code Red” at Google’s search business because of the advent of ChatGPT and novel generative AI products.
Re: Prediction #2 Another great AI semantic search project by Travis Fischer gives us the ability to search any YouTube playlist or channel for information nuggets.
Great read! It feels like these advancements are going to help level the playing field and counter the incredible advantage that Google has built up through years and years of being the #1 search engine.
Great breakdown. Love the highlights on the need for vertical search - this feels like a wide open playing field