85
votes
After two decades the dominance of Google Search comes into question
Link information
This data is scraped automatically and may be incorrect.
- Title
- The end of the Googleverse
- Authors
- Ryan Broderick
- Published
- Aug 28 2023
- Word count
- 3709 words
The issue that I deal with google search is that most of the indexed web or at least the results that reach the top are all a combination of SEO or AI derived drivel that is useless and clogs things up. I don’t think google is exclusively to blame here (lots of content is being tucked behind walls nowadays) but the usefulness of their search has deteriorated significantly.
Mainly for privacy concerns, I’ve grown used to using DuckDuckGo as my first try. If that doesn’t work, then I’ll try Google, which fares a little better at times. I also add the term “reddit” when I’m trying to find a useful recommendation. Although I’ve largely stopped using Reddit post API changes, it is still a large repository of people-created media.
I use the same methodology when searching, but have also noticed a significant reduction in the effectiveness of search modifiers like booleans. Hell, sometimes the moment I add one, I literally lose all results. No hits on the combinations I'm searching. When it happens, I can understand if I'm combining two words or phrases that are unlikely to be common, but sometimes the combination is barely a refinement on the search parameters, just the next level of paring down.
What fascinates me is that I grew during the dawn of search engines and learned how to use them effectively, but my nieces who are both teenagers have the same level of access to the internet and lack almost any understanding of how the internet works or how to effectively search for answers on the internet. It feels like a weird regression in human learning, because my nieces aren't the only kids their age that I notice this with. It seems like the compartmentalization of the internet through apps has had a huge effect.
Google will sometimes ignore verbatim search, and I get unreasonably frustrated every time I rememeber
me: Searches search term, where the spelling really matters
Google: Shows me results for something phonetically similar and completely unrelated, without the little "Did you mean"
me: Searches for "search term"
Google: Shows me the exact same results
me: Searches for """search term"""
Google: Shows me the exact same results
me: Closes tab in frustration
The Great Eshitification is upon us. Seriously though so many tech companies are following the similar trend of increasing costs while reducing functionality and user options. For my part I'm moving away from services where I can and trying to only use services that meet my (admittedly undefined) balance of services and cost. Monetary or otherwise.
I remember, when I was younger, that Google was amazing, great features and tools coming out all the time.
Then they had their IPO and little by little they went downhill.
I don’t know if this results in better/worse results than using quotations but, when you search something, you can click on the tools button and select between ‘all results’ and ‘verbatim’.
I don't think that apps have much to do with it, honestly. Was I the only one who would try random command line parameters in PKZIP for DOS to see what difference it made?
Early adaptors thrive on exploration and will spend hours and hours on attempts that may never turn into anything other than a learning experience that it doesn't work. In the early days of ARPANET/internet the digital explorers often had to read the manuals just to get around and then spent hours poking around just exploring/looking for things. They would then share their findings with friends or like-minded explorers. In time, it became such common knowlege to a wider population that it became, well, common. Once it became common there wasn't a deep desire in people to dig into the manuals for obscure switches, parameters and arguments so they lived with "good enough" results.
I don't think this is relegated to just digital explorers either. The first pioneers of motorized transportation were also mechanics by necessity because so much was made up as they went. After that initial generation and their rush of innovation was over, less people came to know the intracacies of their motorized vehicle as they became more and more common.
There's a moment I realized that whatever I knew about good practices for searching should be replaced by all the natural practices from the general population. Search engines were optimized for their use. Probably when I stopped bothering going beyond the second page.
Yes, I know what you mean. I stumbled on some of the craziest sites through Google as a kid. Ghost stories, a guy who claimed to control weather after being abducted by aliens, and all sorts of random rabbitholes where people could submit their life experiences...
But now I only ever see listicles, those dumb articles that are just lists. Like, ONLY those. It just sucks, there's no real diversity in what you see anymore. No randomly stumbling on cool rabbitholes or neat discussions, gotta sort through all the lists first.
I feel the same way.. Have you tried Marginalia search? It's an amazing search engine for finding old, obscure and weirdly specific websites with human content. I love spending time on there and going down random rabbitholes.
Sorry for the delay in replying. Thank you for sharing this! I'm going to find some fun rabbit holes to dive down~
I don't think it's strictly apps, so much as the internet becoming more centralized and streamlined. In the early days there were all kinds of websites and sources, and computers also tended to be a bit more hands-on. People were basically learning how to use the internet together.
But they've gradually whittled down and narrowed the scope. A lot of people only ever use a handful of sites these days, and a lot of smaller sites have shut down as a result of inactivity. So there's less need and incentive to go searching to see what it holds beyond those sites you're on. And with how Google is now, it's also harder to explore and find those more niche sites anyway.
An example of this decline was posted on Mastodon a couple weeks ago - Google vs DuckDuckGo result
Using the search term "country in Africa that starts with the letter k". Rather than returning an accurate academic response the top result generated is pushed to the top spot via a joke picked up by ChatGPT
You can quickly move a DuckDuckGo search to Google by adding a “!g” (without quotes) before/after your search query. I typically use “!s” to search Startpage if DDG results aren’t great.
More “bangs” can be found here: https://duckduckgo.com/bangs
Ads as well. I was googling FDA sharps locations to find a spot locally for my medical needles to be disposed of. The first time the result was FDA. But I got distracted and had to come back later. This time, unknowingly, I clicked on the first page which looked pretty legit if you weren't tech savvy but also super sketchy if you did.
Old people and less knowledgeable people probably have an even worse time than someone like us. I cant imagine the misinformation and scams they may be led to with googles bullshit ads
I used to use Kagi, and I really liked it. Nevertheless, I ended up stopping my subscription and going back to DuckDuckGo.
Kagi genuinely gave me better search results. I’ve considered resubscribing multiple times. My searches with DDG are noticeably worse, to the point that I’ll now just use Tildes for things I might have searched in the past.
You know that mythical garbage patch of plastic in the Pacific Ocean? Sometimes I feel like the internet is a digital version of that now. There is so much accumulated cruft, so much of it artificial, all pooled together. Searching feels like going fishing in that. You pull up your net and have to sift through all the junk looking for something that might not even be in the net in the first place.
Kagi didn’t feel like that. It felt like I was getting cleaner hauls. A lot of the noise was gone — what I’m not sure about was if there was enough signal. Are there still “fish” out there on the internet to catch, or has the “plastic” just killed the whole ecosystem? Even if a site isn’t AI generated or spamming SEO stuff, it still likely exists only for advertising or affiliate link purposes. What I want (quality information) is now usually at odds with what the web pages I land on want (my eyeballs and clicks). It’s gotten to a point where I’m genuinely surprised when that isn’t the case.
I think the reason I haven’t resubscribed to Kagi is less about Kagi itself and more about the wider internet: its decay is now the dominant experience. I’m less likely to search things because I’m less confident I’ll be able to find them — or that they even exist in the first place. When I do find something, I’m suspicious of it and its motives. The whole experience is a drag — a chore. Searching isn’t any fun anymore, and I don’t know if it ever will be again.
Yeah, like another commenter here, I’ve noticed the disappearance of many useful sites in my own collection of bookmarks. I recently went through the bookmarks on a specific topic that I’d saved over the years, and easily half of them no longer led anywhere.
Also a Kagi subscriber - I agree with this assessment. I keep subscribing because I guess I'm desperately trying to keep some level of utility, rather than the alternatives. (which are now so bad that I might as well just scream my search terms out loud and then pray for an answer)
The quality of search is just Not Good though, and I don't think it will be again. I prefer wiby.me for just browsing cool stuff like I used to back in the day, but it's pretty useless for work or anything serious.
I was just thinking earlier that I need to sit down and think about how I'm going to use the Internet from now on. The enshittification of Google became really obvious to me in the last 2 or 3 years, and the state of social media is just dire. What you wrote here is not surprising in the least, but it is discouraging. As someone who used to be chronically online, and who grew up with the Internet, I keep wishing for something different and better, but I don't think it's coming.
I gotta say, though, that Tildes feels a tiny bit like a step in the right direction. :)
How many times have we did a search query and opened a search result. And after reading its two page intro the page basically says we don't know the answer or something in that vain. Is that really Google's fault or web is generally like that now?
After two decades the dominance of Google Search comes into question
Has Google dominance come into question finally after two decades?
Google is a search engine.
Google is a dominant search engine for last two decades. But has it dominance come into question now?
And it just goes on and on. Bing and DuckDuckGo also give similar results.
This is my "shrinking web" theory. The number of sites on the web has grown enormously, but overwhelmingly they are all spam. Meanwhile, all of the actual, real, useful content has been consolidated into smaller and smaller handfuls of sites that are increasingly walled and moderated. It's a vast difference from the web of 20 years ago, where most of what was out there was generated by humans for the purpose of writing it, not for clicks or SEO or anything else. And until recently, a human still had to write all of that click/SEO drivel - AI will be the final nail in the coffin.
Consequently, the old adage of "nothing is ever deleted from the internet" is also not really true anymore, as this content consolidation has accelerated. Occasionally I remember something I saw on the web 20 years ago, finding it now is often impossible.
This is literally why I have to ignore the first full page of results on Google for many of my searches anymore. I can usually spot the "buzzword bingo" language on half of the AI pages (that will somehow talk in circles on the subject yet never actually answer my query). It always seems that the rest of the first page are either sponsored links or slightly better AI written articles that also don't answer my question.
The "dumbing down" of results is also a huge PITA. I'll look for something on a specific motherboard like "Biostar BFD-1234G-V" and it will return results like "ASUS DDG-5678" and I'm just like "Why??"
It is a little disappointing, isn't it? I've been able to successfully migrate my bookmarks for many, many years, and I dread to think how many of them are dead links. I lost a knitting pattern that had three slight variations on three different websites, all of which are now defunct. I'm pretty sure I can find it again with the archive or a new search, but it was a little disconcerting that my redundancy (which seemed like overkill at the time) wasn't enough.
I've actually started printing out recipes I really love. It feels like a step backward, but does save me from scrolling the gauntlet of the SEO life story before the actual instructions. I suppose paper backups will help protect me from the Cylon hordes, too. That doesn't help with miscellanea I don't know I'll want to reference again in the future, though.
Ugh, I just had the same thing happen the other day. Someone mentioned Hazbin Hotel and I looked to see if there was any news of more of it coming out, but it was page after page of titles that implied they knew, 5-10 paragraphs of background about the show, and then "nobody knows".
And you'll find the same thing if you do a similar search for anything, even if the show has been definitively cancelled.
I think we have arrived at the point that we need human-curated search engines. If there is an algorithm, it will be gamed, and this is something we have known since before computers were a thing.
Some of the blame lays on Google's feet. They set the standard by which SEO is defined. Due to this, sites desperately chase it to be blessed by the almighty algorithm. If Google wanted better results, they should improve their SEO metrics. Preferably to something less gameable.
That makes it sound so simple, but this is a hard problem. Google and it’s competitors devote a massive amount of developer time to this, but there are vastly more SEO professionals working the problem from the other end and they have way more incentives to succeed. Excellent SEO results can mean tens of millions of dollars in revenue.
I'm aware it's difficult, but Google has the resources for hard problems. This could be a testing ground for their AI to conduct surveys on content to see how useful they are. Combine that with a human based article survey to improve rankings and detect abuse patterns. They should also look to adding a "usefulness" rating on the search so users can rank results they see. I know Google already tries to get that automatically by watching user behaviors, but it fails to capture frustration or failure well.
I really can’t overstate how much effort Google is putting into this problem and how difficult this problem actually is. They are already doing human based article surveys and have been doing so for a very long time on a massive scale. You may find the Google quality rater’s guide interesting:
https://services.google.com/fh/files/misc/hsw-sqrg.pdf
I guarantee you that Google Bard and similar LLM technologies are being used to conduct the same kind of analysis on an automated basis.
Google releases regular engine updates. A large percentage of them are about tackling spam and low quality results. You can see a history of those updates here:
https://www.searchenginejournal.com/google-algorithm-history/
Every spam update is effective for a month or so before spammers find new ways to beat the system. It is a constant arms race that is only being made worse by the widespread availability of publicly accessible high quality models like Llama.
I should mention that much of the content that is being generated by the AI models is never intended for human consumption, instead being posted on PBNs (private blog networks) which are sometimes truly private and used only for internal promotion, and sometimes rented out for hundreds to thousands of dollars in order to boost ranks for whoever can afford it. A lot of these PBNs are very deep cover, very hard to distinguish from legitimate sites.
In addition to technological SEO attacks, Google must deal with human corruption. There are several marketplaces where I can buy links from major sites like Forbes or the New York Times (for prices that can run into the thousands of dollars per link) to boost the credibility of my site. These links are carefully inserted into otherwise legitimate looking content that is written solely for the purpose of boosting SEO results.
Capturing behaviour and preference data is likewise a difficult task. Any signal that Google utilizes is another angle for spammers to game. For a while, one common task on Amazon Mechanical Turk was to perform searches, go to a specific result, and then click through and spend a certain amount of time in the site (and prove it by competing a little task, like finding the tenth word in the second paragraph of the page for example) to game the bounce metrics.
Any usefulness rating would likewise become a target for spammers falsifying data.
As someone who has spent almost two decades in the trenches doing a full range of white hat to black hat SEO, I assure you this problem is incredibly complex and dynamic. If Google could solve it, they absolutely would have. Poor quality results are one of their primary engineering challenges — they take it very seriously.
If Google is devoting so many resources to this endeavor, why aren’t they able to remove pages from their search results that are obvious keyword stuffing with no actual content, pages that are just copies of Wikipedia pages, and pages that are just copies of Stack Overflow pages?
It’s because it is actually a very hard problem to solve. There are a lot of signals involved, and it can be very difficult to remove spam without inadvertently removing legit, valuable content at the same time.
I understand how seeing obviously duplicated content seems easy to remove. It’s not always the case that duplicate content is bad though — there are lots of valid cases where there is added value to what might superficially look like mere duplication to an algorithm.
There is a lot of content being published though. We are talking about hundreds of trillions of web pages, and Google’s users want fresh information to be surfaced quickly — that limits how much deep analysis can be conducted on each new page.
Spam filtering tasks happen on a different schedule (kind of, there are many layers, some of which happen at the initial ingest time, like the Penguin algorithm). Low quality pages are frequently removed within a few days, but it can take time, especially when a page’s “reputation” is buttressed with a high quality spam network.
Google is constantly searching for signs that sites are engaging in link manipulation. Actions against these sites is difficult as well though because of the existence of negative SEO campaigns — it is not unheard of for SEOs to go and buy known bad links off of previously burned PBNs in hopes of tricking google into handing down a manual penalty. This is one reason why it’s much less common to get a manual penalty such as being forcefully deindexed.
Even while these things are going on, this massive volume leads to a perceptual problem — it looks like Google is allowing all these individual spam links to stay up, but really there are just a lot of new spam links that get shown briefly and then get replaced with new spam.
Yeah, the model isn't that bad. It lacks many features like blocking some results (e.g. mentioned Pinterest) or bringing results from certain pages to the front (e.g. how many people would treat Reddit), but ultimately this feels unmanageable. There's just too much garbage, too many garbage actors, swathes of the net hiding away in unindexable Facebook groups and Discord channels. There really isn't a solution to this, too. Is Google or any other search provider supposed to create bots to join Discord and drone-bomb the repeat malicious actors?
Fixing Google Search would require fixing the human condition.
Fixing the part of the problem where google ignores the words in quotes that I chose to search and signal that they were specifically important by using quotations would go a long way.
No Google I don't want results for this other word or phrase that you think might be kind of similar. I want to find what I searched for.
I don’t think the problem is ultimately unmanageable, and I am pretty confident that I know what the long term solution will be. It’s going to be “intelligent” agents that go through search results and other databases on our behalf and actually read the content for us and have access to public (and possibly private) trust databases for analyzing sources to automatically filter down to the real content.
Anecdotally, my experience has been that Bing and DuckDuckGo still produce worse search results than Google, so I'm not sure if the dominance of Google Search can really be questioned yet, even if it has diminished.
With the type and quality of content Google surfaces now, especially the social content it pushes, though, along with the ads, I've switched to a paid search provider - who also leverages AI (which I suppose everyone says now, it's the new hot thing!) - and am much happier with the experience.
What paid search provider are you using?
probably Kagi, it lets you customize a lot of settings in your search (example : prioritize programming related results, or deprioritize foxnews).
It doesn't exactly fit the context of this thread, though my experience using paid search has been really positive, starting with Neeva before it disappeared. There are some very reasonably-priced alternatives, such as Perplexity, that are worth trying using their free plan. So far, I have been pleased with those I have used.
FWIW, I switched to Waldo, but it is ridiculously expensive if only using it for search. Its value is as a research tool to deep dive into specific subjects, data, and information, but I also use it for 100% of the simple searches I otherwise would have used Google for, which is a nice benefit.
I have no doubt that seo in general, blogspam and other low quality content are a problem for getting quality results.
But closer to a decade right now than not Google results are getting ever more irrelevant. I have no idea how they interpret or "optimize" the query on the backend but it is making searching for anything even slightly complex or technical very frustrating. It has absolutely no problem ignoring important keywords which should be by default assumed are all of them. Mostly it seems that it searches for generalizations, ignoring the context given entirely.
And if the query even slightly resembles something popular you are just out of luck. And search operators such as + or - does not seem to do anything much.
As a sidenote I really wonder what do engineers working on the engine use for search, they are not using the stock one unless they have no choice or can nudge it to actually show them results for the query they entered instead of some drivel it was interpreted as.
I would guess that Google has massively deprioritized the parts of their system that separates the wheat from the chaff in favor of increasing the amount of information they display on the actual search pages themselves.
The problem was that Google was never good at determining quality; their "quality" rating was instead data science - something determined by the measuring the environment around it rather than the content itself. Computers traditionally had no way to determine quality in terms that humans would agree with. AI and LLMs might get there soon, though. Hopefully.
Ugh, this is easily when I’ll get annoyed the most with searching. And it leads to getting the exact opposite of what’s being asked, eg so many times I’ve google something along the lines of “turn off X” and every single result is about how to turn it on (because that’s the normal use case) and it’s just like ???
This has also been my experience. I know they are designing for what they want. Delivering accurate useful information is not it.
I'd say about twice a week, there's a search that turns up better results from Google than DDG. Considering that I'll easily do 20 searches a day or more, that's not exactly high praise.
I'll take this time to shout out kagi.com. After years of using duckduckgo.com for my main search engine, I got tired of falling back to Google due to bad results (pretty much daily).
Kagi has removed that completely, it works great, the features are amazing, and its totally worth $5/mo to me in saved time.
Frankly paying a subscription for a search engine irks me, everything on the internet nowadays is a subscription of some kind. Music? Subscription. Videos? Subscription. News? Subscription. Software? Subscription. Even games are becoming subscription based. I'm not made of money
you greedy sillicon valley fucks!And like, $5 for only 300 searches is far too little, I easily do 20 searches on a single day. I feel like I would be deliberately avoiding using the search engine to not reach the limit, which is obviously counterproductive.
I look at it as I'd rather pay a subscription than get monetized in other ways. I've no particular investment in it or anything, but have been subbed since the early days and will happily keep forking over my 10bux for as long as the search results are good.
Unless you run your own self-written software, there will always be some level of trust when using services or software made by others. All the alternatives definitely mine your data in some way and will continue to do so without hiding. Yes, there is the possibility that Kagi is flat out lying, though that will be a very weird business strategy. They are a very niche search engine with their users already very privacy focused and techminded. I doubt that data is more valuable than having their entire business model dissapear in the blink of an eye if there was ever the slightest doubt about their privacy.
Same here. I did the trial, liked the results so much that I now use a family plan that I split between two accounts (work and home). While I slightly bristle at paying yet another subscription, it's been worth it.
For both work and home purposes I get desired/quality results much faster, and the ability to completely filter out domains from searches (or weight and pin domains to affect how much they show up and where) has made many of my searches give me the desired result in the top 3-5 results.
Also, no ads. As far as privacy goes- it's just a matter of trust. I was already using Google which ties every search to me, so I'm at least sidegrading, if not upgrading privacy. No matter how convincing any company's promises are around privacy- ultimately it just comes down to giving a certain level of trust to that entity- I've decided to tentatively trust Kagi, especially given that there are no ads, and the option even for having your own history turned on for your own use is not even available yet. That said- it's still trusting without truly knowing- I've just decided that's a tradeoff I can make.
A whole lot of words in that article, but I don't see anything in the article that backs up the headline inference that Google is losing any market share.
So ... is this a misleading headline?
I already revised the headline to be less click bait. In my mind, the question is the complaints about quality as documented in the article. To lose market share we need a good alternative
theverge.com wrote the terrible headline ... and yet you have to fix it :D
holy shit you are right. wow. TIL.
Amusingly they must have thought the same thing,
Clicking on growing literally takes you to a hacker news comment.
And I was kinda sad, I tried using ddg for a good few months and it was just really bad at easy thing. I typed in the title of a song that I thought was fairly known, and it didn't have a youtube link to it on at least the first 3 pages (that's all I checked before just googling it to make sure I wasn't crazy.) I don't remember what the song was but I know that was the last straw and I just switched back to google as my browser default. :/
Some people have been suggesting Kagi, but I haven't tried it myself.
For a couple of years now I've been using 'reddit' as a suffix to get results that actually mean something instead of AI generated blogpost garbage.
Unsurprisingly, ChatGPT is turning out like a more reliable Google search to me as well for various questions, although it needs some sanity checking. GPT-4 is definitely more reliable than 3, though.