I think the youtube downloader example really showcases why search personalization is important. A leading quote And it's also the north star for one of the "tests" Seriously, take a step back,...
Exemplary
I think the youtube downloader example really showcases why search personalization is important.
A leading quote
Here's a fun experiment to try. Take an open source project such as yt-dlp and try to find it from a very generic term like "youtube downloader". You won't be able to find it because of all of the content farms that try to rank at the top for that term. Even though yt-dlp is probably actually what you want for a tool to download video from YouTube.
And it's also the north star for one of the "tests"
Download youtube videos
Ideally, the top hit would be yt-dlp or a thin, graphical, wrapper around yt-dlp. Links to youtube-dl or other less frequently updated projects would also be ok
Seriously, take a step back, and think about how absolutely insane what Xu and Luu is saying - the FIRST result should be youtube-dlp? A python command-line tool - not just a command line tool, a FORK of a command line tool, because the original got DMCA'd, to the incredibly generic search term "youtube downloader". I'd honestly say that if a search engine, a priori to any user information, gave back the youtube-dlp repository to a search of "youtube downloader", it did a horrible job.
But, it can be the right result, if the search engine knows you're someone who uses command line tools. And in that respect, Google out of all the search engines performs the best for me. If I search "Ruby", the first result I get is the homepage for the Ruby programming language - again, that would be insane behavior a priori, for 99.9% of the population they would want the precious mineral, but because I'm logged in, it makes sense in the context of what I google for.
All that data hoovering and profile building does come into use.
As an aside, the fact that they seriously used that as one of their metrics is shocking - what a bubble that is!
That's an incredible bubble. I would never want a fork of a repository of something. I just don't got the time to figure it out. I want a site that lets me paste in a link and spit out a file to...
That's an incredible bubble. I would never want a fork of a repository of something. I just don't got the time to figure it out. I want a site that lets me paste in a link and spit out a file to download, preferably with no ads.
But you know, surely that can be done with zero user data hoovered. Here's a slider for "technical difficulty": for some subjects I'll slide it up, for some subjects, gimme the simple wiki explanation. It doesn't need to know anything else about me other than what I'm telling it at that very moment.
I hate it when devices/software make poor guesses at what I want instead of just having options I can see and interact with. Or worse yet: insist on offering me things I find horrible with no way to turn it off.
We all grew up with keywords and tags now: let users use them.
Ruby programing.
YouTube downloader python.
A search algorithm doesn't need to know my race and ethnicity and where I stand on politics and my blood type and star sign.
I think using scoping keywords like "code" or "programming" is how it used to be and maybe still is with regards to differentiating words with different contextual meanings. And I like that...
I think using scoping keywords like "code" or "programming" is how it used to be and maybe still is with regards to differentiating words with different contextual meanings. And I like that approach. The slider approach I don't think would be very practical. Think about all the dimensions of expertise that exist about so many topics. The number sliders you would need to even make a dent would be overwhelming.
What I think we really need is a universally honored do not track signal as a browser option. That way people can allow sites they trust our whose features make the trade worth it, and others can opt out.
Case in point, I just searched for "youtube downloader" on Kagi and yt-dlp is the third result. Which is not bad, considering that someone who is actually interested in yt-dlp would not phrase the...
I think the youtube downloader example really showcases why search personalization is important.
But, it can be the right result, if the search engine knows you're someone who uses command line tools.
Case in point, I just searched for "youtube downloader" on Kagi and yt-dlp is the third result. Which is not bad, considering that someone who is actually interested in yt-dlp would not phrase the query so naively. "youtube download cli" puts it at the top. (And this is not a "workaround for bad software", this is just not being stupid and type in what you really want.)
Everybody gets the programming language when they search for Ruby on google though. That’s not a result of personalization. In fact, personalization effects on Google are much weaker than people...
If I search "Ruby", the first result I get is the homepage for the Ruby programming language - again, that would be insane behavior a priori, for 99.9% of the population they would want the precious mineral, but because I'm logged in, it makes sense in the context of what I google for.
Everybody gets the programming language when they search for Ruby on google though. That’s not a result of personalization. In fact, personalization effects on Google are much weaker than people often expect — the primary effects (by a huge margin, an order of magnitude at least) are on what advertising you see, not improving organic search results. Google likes to talk a lot about how personalization improves your results but they basically mean that in the sense of it “helping” you spend money faster.
This also exemplifies my primary complaint with this article — drawing any kind of conclusions from ad hoc sampling. To get a real idea of how search engine performance is and what it is trying (and sometimes failing) to do, you need a lot more quantity and a lot more deliberate planning of the data. There are entire industries of data analysis tools for monitoring the search engines and what kind of shifts they are making.
Well, there I go generalizing without sufficient evidence. After checking more, I get the programming language on most of my devices in private browsing (iPads, desktop, phone) as well as through...
Well, there I go generalizing without sufficient evidence. After checking more, I get the programming language on most of my devices in private browsing (iPads, desktop, phone) as well as through the Google paid search API, but I did get the gemstone using Firefox on my phone while logged in. Google does randomize their results to check click through rates though, so it’s hard to draw much of a conclusion from that either. Now I’m mildly curious to try some sampling using a residential proxy network.
I get this result, too, when searching via duckduckgos "!g" bang, in a browser with cleared cookies, not logged in. Same on bing, which I never use. I'd guess way more people search for the...
If I search "Ruby", the first result I get is the homepage for the Ruby programming language
I get this result, too, when searching via duckduckgos "!g" bang, in a browser with cleared cookies, not logged in. Same on bing, which I never use. I'd guess way more people search for the programming language than for the mineral. That's just the standard answer, I think.
If I were to google for the mineral, as a layperson, I'd probably only do so maybe a handful of times in a decade. How many times a day does a Ruby programmer search for something with "Ruby" in...
If I were to google for the mineral, as a layperson, I'd probably only do so maybe a handful of times in a decade.
How many times a day does a Ruby programmer search for something with "Ruby" in the query? I'd bet no less than 5. That almost certainly skew the results very heavily.
I got the mineral, followed by programming language, followed by Ruby Tuesday. I think my favorite example of personalization is when I typed “Gold Ore”, immediately facepalmed as of course that...
I got the mineral, followed by programming language, followed by Ruby Tuesday.
I think my favorite example of personalization is when I typed “Gold Ore”, immediately facepalmed as of course that wouldn’t give what I was looking for without “WoW” attached, and then watched in awe as google populated wowhead as the top result anyways. These days it’s back to results on the real world metal, so the bubble adapts pretty quickly too.
I suspect this has to do with broader internet trends more than actual search ability. I've noticed in the last couple of years that the quality of everything on the internet has gone downhill:...
I suspect this has to do with broader internet trends more than actual search ability. I've noticed in the last couple of years that the quality of everything on the internet has gone downhill:
Public social media like Reddit and Twitter has been overrun by astroturfing.
Private social media like Instagram and Facebook have successfully exterminated all first-party content from friends and family. All that remains is ads and "shared" memes and inflammatory news posts that inevitably trend towards flame wars.
Video content is full of sponsored content and increasingly trends toward low quality "shorts" content full of hot takes and clickbait-style claims.
Online shopping searches have become near impossible, because there's no good way to find and compare a single product across multiple retailers. To boot, most sites use the same crappy Shop purchasing portal that ends up locking me out behind an inescapable Cloudflare redirect loop (and lacks the option to mark a product as a gift purchase). And because of anticompetitive practices Amazon ends up with both the cheapest price and the cheapest shipping 90% of the time anyway... but you'll probable get a knockoff product at least 40% of the time because they intermingle inventory from legitimate sellers and scammers.
Messaging has fractured more and more between iMessage, Whatsapp, Telegram, Facebook Messenger, Instagram DMs, Snapchat, Discord, Slack, and a half dozen other non-interoperable services I can't remember. I have no idea where to reach anyone any more.
All of my favorite podcasts have been bought up by conglomerates and stuffed with ads until they burst. Many of the podcasters have either left or silently stepped away.
Basically the only thing that's worth a damn on the internet anymore are independent blog sites who have a revenue source other than linkspam -- either because they're run by one crazy person who loves writing, or because they have some kind of membership. I guess Marginalia does well because they really emphasize those independent sites. But Marginalia search has its weaknesses too, because sometimes I actually do want to search for [rainjacket XYZ] and find it at a bunch of retailers that Marginalia doesn't index.
This situation actually reminds me quite a bit of the early days of the internet, before Google's search superpower made it easy to find things. So maybe there's a golden opportunity for Marginalia or Kagi to seize the crown of Best Search Engine by filtering out the exact SEO and ad-laden crap that pays Google's bills.
I think you’ve nailed the root of the problem. There’s money to be made by gaming search results, so they will be gamed. Personally, I’ve been using Kagi for the last six months (and used...
I think you’ve nailed the root of the problem. There’s money to be made by gaming search results, so they will be gamed.
Personally, I’ve been using Kagi for the last six months (and used DuckDuckGo for years before that) and the results are noticeably better than Google when it comes to searches that are typically targets for spam. It also easily lets you down rank or block sites that you don’t want (such as Pinterest or Quora).
Also a recent convert to Kagi, from DuckDuckGo; I think it’s really, really good. DDG was mostly good, but about ~1-in-5-ish searches I would need to fall back to google for. No such issue with...
Also a recent convert to Kagi, from DuckDuckGo; I think it’s really, really good. DDG was mostly good, but about ~1-in-5-ish searches I would need to fall back to google for.
No such issue with Kagi- comprehensive results, quality equal or above DDG and google, the “specialisation” toggle is super handy, and the custom weighting functionality is absolutely worth the subscription price alone.
My experience has been similar to @FridgeSeal. DuckDuckGo was good for basic stuff, but I would find myself falling back to Google a lot more than I liked. I don’t think I’ve done that once since...
My experience has been similar to @FridgeSeal. DuckDuckGo was good for basic stuff, but I would find myself falling back to Google a lot more than I liked. I don’t think I’ve done that once since switching to Kagi a few months ago.
The real litmus test is that I asked my SO if she noticed Google search results were getting worse lately and she said yes. I asked her if she wanted to try Kagi and since she’s remarked several times that her search results have noticeably improved.
To expand on the part about shopping: it's hard to search for things that are purchasable items, on an informational basis, without the results being full of unwanted shopping results. If I want...
To expand on the part about shopping: it's hard to search for things that are purchasable items, on an informational basis, without the results being full of unwanted shopping results. If I want to find places to buy something, I'll specifically search for that.
The Web was conceived as a means of disseminating information, not as a replacement for Skymall.
Well said. I honestly wonder what the folks who work on the Google Shopping tab think they're doing, when normal Search is so inundated with shopping results. It gets worse with really obscure...
Well said. I honestly wonder what the folks who work on the Google Shopping tab think they're doing, when normal Search is so inundated with shopping results.
It gets worse with really obscure stuff. Sometimes, I want to find in stock shopping results for niche products. But search is so fuzzy these days that you can't search the product name and get results for only that product -- if it's niche, you'll start getting results for similar (but not quite the same) products on the very first Google search results page. Even if you include a model year or some other specific identifier. I wonder if there's still a search engine out there that respects explicit quoted searches?
I use Kagi and find it to be pretty accurate, especially since I've added custom filters. This article just shits all over Kagi and I do not see the same problems that the author describes. So I...
I use Kagi and find it to be pretty accurate, especially since I've added custom filters. This article just shits all over Kagi and I do not see the same problems that the author describes.
Agreed. This comparison just feels off to me. Marginalia ranking high made me curious to go try it, and after a few queries I don't even find it functional compared to Kagi. For example, searching...
Agreed. This comparison just feels off to me. Marginalia ranking high made me curious to go try it, and after a few queries I don't even find it functional compared to Kagi.
For example, searching "open source wideband o2 sensor" without quotes on Kagi returns highly relevant blog posts, board designs, products for sale, and github repositories of exactly what I'm looking for. On Marginalia it returns nothing. Even paring the query down to the much more general "wideband o2 sensor" only returns 5 results on Marginalia, none of which are relevant. The first two results are at least vaguely automotive, but results 3-5 are iPhone and Apple Watch reviews.
I understand Marginalia isn't operating at the same scale as Kagi, and that one search query isn't illustrative of overall performance, but I also feel like that's a pretty mainstream, softball query for me. After reading the article I wasn't expecting it to be a total blowout like that.
Marginalia isn't even trying to compete with Kagi/Google. They are targeting the "small web" or whatever it is that you call lighter websites. This kind of philosophy : https://potato.cheap/
Marginalia isn't even trying to compete with Kagi/Google. They are targeting the "small web" or whatever it is that you call lighter websites. This kind of philosophy : https://potato.cheap/
As other commenter noted, Marginalia is deliberately playing a different game, which I think is awesome. Also, in the hacker-news comments, the Marginalia dev piped up to say they felt it got...
As other commenter noted, Marginalia is deliberately playing a different game, which I think is awesome.
Also, in the hacker-news comments, the Marginalia dev piped up to say they felt it got lucky with the search terms, and wasn’t amazingly representative. On the whole I think this test isn’t very designed or executed TBH, especially given the comments here + HN.
In summary, the author does the following 6(!) queries: Download youtube videos Ad blocker Download Firefox Why do wider tires have better grip Why do they keep making cpu transistors smaller?...
In summary, the author does the following 6(!) queries:
Download youtube videos
Ad blocker
Download Firefox
Why do wider tires have better grip
Why do they keep making cpu transistors smaller?
Vancouver snow forecast winter 2023
And then there are 17,000(!) words extrapolating from that.
The following is pretty much all of the conclusions:
SEO spammers are pretty good at their jobs
there's a lot of intentionally and unintentionally fake information out there
LLMs, and IR technology in general, does not have excellent comprehension or discrimination of factual information
carefully constructed search queries are better than naive search queries
If you scroll down to the appendix you can get to the actual queries used in the research and a summary of results. The results are all lackluster and generally disappointing across the board.
If you scroll down to the appendix you can get to the actual queries used in the research and a summary of results. The results are all lackluster and generally disappointing across the board.
I'm not entirely sold on the choice of the queries worded as questions here -- It feels a little like the author is getting sidetracked by a wider point they've wanted to make about both...
I'm not entirely sold on the choice of the queries worded as questions here --
Why do wider tires have better grip?
Why do they keep making cpu transistors smaller?
It feels a little like the author is getting sidetracked by a wider point they've wanted to make about both questions. For the latter:
I had this question when I was in high school and my AP physics teacher explained to me that it was because making the transistors smaller allowed the CPU to be smaller, which let you make the whole computer smaller. Even at age 14, I could see that this was an absurd answer
Similarly you could ask 'why are tables made of wood' and be dissatisfied with the answer 'wood is sturdy' - simple questions typically return simple answers, but can be explored endlessly. This doesn't seem like common use-case.
For the tire question, it seems a wrong answer is generally perpetuated, both across forums and SEO blogs, that's fair enough. At the same time, I think there's an interesting question in how search engines are expected to know and retrieve the right answer, especially for unusual queries that aren't documented in something like Wikipedia or NYT.
Anyway, maybe a few queries of different commonalities and depth would have been better to gauge how well each performs, instead of two fairly niche choices that feel a little intentional. Some might do better with quickly answering simple questions, other might be better at entertaining more complex ones.
Yeah, this post just seems off to me. I've agreed with Luu in the past when it comes to websites being too big and so forth but I always felt he took it too much to the extreme- to his own...
Yeah, this post just seems off to me. I've agreed with Luu in the past when it comes to websites being too big and so forth but I always felt he took it too much to the extreme- to his own detriment, for instance, the refusal to even use a modicum of website styling on the site means without user intervention it looks miserable on desktop browsers- user intervention that a lot of users aren't understanding of how to perform- only power users- even his site design reeks of being in a bubble that doesn't consider every day users- and this search experiment only further confirms that.
It seems his ideal overcorrects the problems, and that extreme desire for web minimalism leaks into this post too- where the supposed top search engine in his tests is a "small web" search engine and one that often does NOT return quality results, or enough results. The fact of the matter is most of the web as it exists right now does not meet Marginalia's criteria- and Marginalia is not even intended to really be a comprehensive search engine. It even describes itself as:
search engine that focuses on non-commercial content, and attempts to show you sites you perhaps weren't aware of in favor of the sort of sites you probably already knew existed.
Kagi has been so good for me, especially with its customization options, that I'm gladly continuing to pay for it because going back to Google and other options produces far worse quality for most searches. It has fundamentally changed web search for me in terms of clean, accurate, extremely relevant results.
Maybe it's just that my specific experience is so different from these "tests"- but I think that's a relevant point- what Luu is seeing as quality in results is very specific to his wants/needs and small-web-ideology in a way that doesn't extrapolate to everyone else. A bubble, as others have noted. I'm just quite surprised this whole post isn't more self-aware.
So many engines tailor their results based on data they've farmed from the user. It would be a lot more interesting to repeat the experiment on a wider range of systems.
So many engines tailor their results based on data they've farmed from the user. It would be a lot more interesting to repeat the experiment on a wider range of systems.
After reading this on HN, I started using kagi again, loaded Orion onto my phone, and it's like a whole new internet experience! Tildes loads ahem poorly on mobile Safari but is lightning quick in...
After reading this on HN, I started using kagi again, loaded Orion onto my phone, and it's like a whole new internet experience! Tildes loads ahem poorly on mobile Safari but is lightning quick in Orion!
Thanks for the article. Interesting. Tried DDG a few times and it struggled to come up with results that google had no issues with I have a Kagi account - yet to really try it extensively. Isnt...
Thanks for the article. Interesting.
Tried DDG a few times and it struggled to come up with results that google had no issues with
I have a Kagi account - yet to really try it extensively. Isnt the free account limited to 100 searches a day or something? (having to sub for more than the limit is a downside i imagine but not if the search results are infinitely better imho).
Anyone tried Marginalia?
I have been trying to main DDG but man the results are so poor I find myself habitually adding !g when I start my searches at this point. Kagi doesn't have a free tier, they just have a 100 search...
I have been trying to main DDG but man the results are so poor I find myself habitually adding !g when I start my searches at this point.
Kagi doesn't have a free tier, they just have a 100 search trial (they even label it as a trial in the pricing)
I think the youtube downloader example really showcases why search personalization is important.
A leading quote
And it's also the north star for one of the "tests"
Seriously, take a step back, and think about how absolutely insane what Xu and Luu is saying - the FIRST result should be youtube-dlp? A python command-line tool - not just a command line tool, a FORK of a command line tool, because the original got DMCA'd, to the incredibly generic search term "youtube downloader". I'd honestly say that if a search engine, a priori to any user information, gave back the youtube-dlp repository to a search of "youtube downloader", it did a horrible job.
But, it can be the right result, if the search engine knows you're someone who uses command line tools. And in that respect, Google out of all the search engines performs the best for me. If I search "Ruby", the first result I get is the homepage for the Ruby programming language - again, that would be insane behavior a priori, for 99.9% of the population they would want the precious mineral, but because I'm logged in, it makes sense in the context of what I google for.
All that data hoovering and profile building does come into use.
As an aside, the fact that they seriously used that as one of their metrics is shocking - what a bubble that is!
That's an incredible bubble. I would never want a fork of a repository of something. I just don't got the time to figure it out. I want a site that lets me paste in a link and spit out a file to download, preferably with no ads.
But you know, surely that can be done with zero user data hoovered. Here's a slider for "technical difficulty": for some subjects I'll slide it up, for some subjects, gimme the simple wiki explanation. It doesn't need to know anything else about me other than what I'm telling it at that very moment.
I hate it when devices/software make poor guesses at what I want instead of just having options I can see and interact with. Or worse yet: insist on offering me things I find horrible with no way to turn it off.
We all grew up with keywords and tags now: let users use them.
Ruby programing.
YouTube downloader python.
A search algorithm doesn't need to know my race and ethnicity and where I stand on politics and my blood type and star sign.
I think using scoping keywords like "code" or "programming" is how it used to be and maybe still is with regards to differentiating words with different contextual meanings. And I like that approach. The slider approach I don't think would be very practical. Think about all the dimensions of expertise that exist about so many topics. The number sliders you would need to even make a dent would be overwhelming.
What I think we really need is a universally honored do not track signal as a browser option. That way people can allow sites they trust our whose features make the trade worth it, and others can opt out.
Case in point, I just searched for "youtube downloader" on Kagi and yt-dlp is the third result. Which is not bad, considering that someone who is actually interested in yt-dlp would not phrase the query so naively. "youtube download cli" puts it at the top. (And this is not a "workaround for bad software", this is just not being stupid and type in what you really want.)
Everybody gets the programming language when they search for Ruby on google though. That’s not a result of personalization. In fact, personalization effects on Google are much weaker than people often expect — the primary effects (by a huge margin, an order of magnitude at least) are on what advertising you see, not improving organic search results. Google likes to talk a lot about how personalization improves your results but they basically mean that in the sense of it “helping” you spend money faster.
This also exemplifies my primary complaint with this article — drawing any kind of conclusions from ad hoc sampling. To get a real idea of how search engine performance is and what it is trying (and sometimes failing) to do, you need a lot more quantity and a lot more deliberate planning of the data. There are entire industries of data analysis tools for monitoring the search engines and what kind of shifts they are making.
I searched in an incognito tab and the first result was the Wikipedia article on the gemstone (the programming language was #2 though).
Well, there I go generalizing without sufficient evidence. After checking more, I get the programming language on most of my devices in private browsing (iPads, desktop, phone) as well as through the Google paid search API, but I did get the gemstone using Firefox on my phone while logged in. Google does randomize their results to check click through rates though, so it’s hard to draw much of a conclusion from that either. Now I’m mildly curious to try some sampling using a residential proxy network.
Very interesting post! Its also nice to see a benefit of the data harvesting.
I get this result, too, when searching via duckduckgos "!g" bang, in a browser with cleared cookies, not logged in. Same on bing, which I never use. I'd guess way more people search for the programming language than for the mineral. That's just the standard answer, I think.
If I were to google for the mineral, as a layperson, I'd probably only do so maybe a handful of times in a decade.
How many times a day does a Ruby programmer search for something with "Ruby" in the query? I'd bet no less than 5. That almost certainly skew the results very heavily.
Plus non-english users who'd use their languages word for the mineral, but the programming language is still ruby.
I got the mineral, followed by programming language, followed by Ruby Tuesday.
I think my favorite example of personalization is when I typed “Gold Ore”, immediately facepalmed as of course that wouldn’t give what I was looking for without “WoW” attached, and then watched in awe as google populated wowhead as the top result anyways. These days it’s back to results on the real world metal, so the bubble adapts pretty quickly too.
I suspect this has to do with broader internet trends more than actual search ability. I've noticed in the last couple of years that the quality of everything on the internet has gone downhill:
Basically the only thing that's worth a damn on the internet anymore are independent blog sites who have a revenue source other than linkspam -- either because they're run by one crazy person who loves writing, or because they have some kind of membership. I guess Marginalia does well because they really emphasize those independent sites. But Marginalia search has its weaknesses too, because sometimes I actually do want to search for [rainjacket XYZ] and find it at a bunch of retailers that Marginalia doesn't index.
This situation actually reminds me quite a bit of the early days of the internet, before Google's search superpower made it easy to find things. So maybe there's a golden opportunity for Marginalia or Kagi to seize the crown of Best Search Engine by filtering out the exact SEO and ad-laden crap that pays Google's bills.
I think you’ve nailed the root of the problem. There’s money to be made by gaming search results, so they will be gamed.
Personally, I’ve been using Kagi for the last six months (and used DuckDuckGo for years before that) and the results are noticeably better than Google when it comes to searches that are typically targets for spam. It also easily lets you down rank or block sites that you don’t want (such as Pinterest or Quora).
How do you like Kagi compared to Duckduckgo?
Also a recent convert to Kagi, from DuckDuckGo; I think it’s really, really good. DDG was mostly good, but about ~1-in-5-ish searches I would need to fall back to google for.
No such issue with Kagi- comprehensive results, quality equal or above DDG and google, the “specialisation” toggle is super handy, and the custom weighting functionality is absolutely worth the subscription price alone.
My experience has been similar to @FridgeSeal. DuckDuckGo was good for basic stuff, but I would find myself falling back to Google a lot more than I liked. I don’t think I’ve done that once since switching to Kagi a few months ago.
The real litmus test is that I asked my SO if she noticed Google search results were getting worse lately and she said yes. I asked her if she wanted to try Kagi and since she’s remarked several times that her search results have noticeably improved.
Yes!! Simple recipes, easy home improvement, crafts, hobbies, pets, gardening....
No blog spam garbage.
To expand on the part about shopping: it's hard to search for things that are purchasable items, on an informational basis, without the results being full of unwanted shopping results. If I want to find places to buy something, I'll specifically search for that.
The Web was conceived as a means of disseminating information, not as a replacement for Skymall.
Well said. I honestly wonder what the folks who work on the Google Shopping tab think they're doing, when normal Search is so inundated with shopping results.
It gets worse with really obscure stuff. Sometimes, I want to find in stock shopping results for niche products. But search is so fuzzy these days that you can't search the product name and get results for only that product -- if it's niche, you'll start getting results for similar (but not quite the same) products on the very first Google search results page. Even if you include a model year or some other specific identifier. I wonder if there's still a search engine out there that respects explicit quoted searches?
I use Kagi and find it to be pretty accurate, especially since I've added custom filters. This article just shits all over Kagi and I do not see the same problems that the author describes.
So I disagree and think the author is wrong.
Agreed. This comparison just feels off to me. Marginalia ranking high made me curious to go try it, and after a few queries I don't even find it functional compared to Kagi.
For example, searching "open source wideband o2 sensor" without quotes on Kagi returns highly relevant blog posts, board designs, products for sale, and github repositories of exactly what I'm looking for. On Marginalia it returns nothing. Even paring the query down to the much more general "wideband o2 sensor" only returns 5 results on Marginalia, none of which are relevant. The first two results are at least vaguely automotive, but results 3-5 are iPhone and Apple Watch reviews.
I understand Marginalia isn't operating at the same scale as Kagi, and that one search query isn't illustrative of overall performance, but I also feel like that's a pretty mainstream, softball query for me. After reading the article I wasn't expecting it to be a total blowout like that.
Marginalia isn't even trying to compete with Kagi/Google. They are targeting the "small web" or whatever it is that you call lighter websites. This kind of philosophy : https://potato.cheap/
As other commenter noted, Marginalia is deliberately playing a different game, which I think is awesome.
Also, in the hacker-news comments, the Marginalia dev piped up to say they felt it got lucky with the search terms, and wasn’t amazingly representative. On the whole I think this test isn’t very designed or executed TBH, especially given the comments here + HN.
In summary, the author does the following 6(!) queries:
And then there are 17,000(!) words extrapolating from that.
The following is pretty much all of the conclusions:
If you scroll down to the appendix you can get to the actual queries used in the research and a summary of results. The results are all lackluster and generally disappointing across the board.
I'm not entirely sold on the choice of the queries worded as questions here --
It feels a little like the author is getting sidetracked by a wider point they've wanted to make about both questions. For the latter:
Similarly you could ask 'why are tables made of wood' and be dissatisfied with the answer 'wood is sturdy' - simple questions typically return simple answers, but can be explored endlessly. This doesn't seem like common use-case.
For the tire question, it seems a wrong answer is generally perpetuated, both across forums and SEO blogs, that's fair enough. At the same time, I think there's an interesting question in how search engines are expected to know and retrieve the right answer, especially for unusual queries that aren't documented in something like Wikipedia or NYT.
Anyway, maybe a few queries of different commonalities and depth would have been better to gauge how well each performs, instead of two fairly niche choices that feel a little intentional. Some might do better with quickly answering simple questions, other might be better at entertaining more complex ones.
Yeah, this post just seems off to me. I've agreed with Luu in the past when it comes to websites being too big and so forth but I always felt he took it too much to the extreme- to his own detriment, for instance, the refusal to even use a modicum of website styling on the site means without user intervention it looks miserable on desktop browsers- user intervention that a lot of users aren't understanding of how to perform- only power users- even his site design reeks of being in a bubble that doesn't consider every day users- and this search experiment only further confirms that.
It seems his ideal overcorrects the problems, and that extreme desire for web minimalism leaks into this post too- where the supposed top search engine in his tests is a "small web" search engine and one that often does NOT return quality results, or enough results. The fact of the matter is most of the web as it exists right now does not meet Marginalia's criteria- and Marginalia is not even intended to really be a comprehensive search engine. It even describes itself as:
Kagi has been so good for me, especially with its customization options, that I'm gladly continuing to pay for it because going back to Google and other options produces far worse quality for most searches. It has fundamentally changed web search for me in terms of clean, accurate, extremely relevant results.
Maybe it's just that my specific experience is so different from these "tests"- but I think that's a relevant point- what Luu is seeing as quality in results is very specific to his wants/needs and small-web-ideology in a way that doesn't extrapolate to everyone else. A bubble, as others have noted. I'm just quite surprised this whole post isn't more self-aware.
So many engines tailor their results based on data they've farmed from the user. It would be a lot more interesting to repeat the experiment on a wider range of systems.
Yes, people report varied results on Hacker News too. I think this sort of study needs be done with multiple people.
After reading this on HN, I started using kagi again, loaded Orion onto my phone, and it's like a whole new internet experience! Tildes loads ahem poorly on mobile Safari but is lightning quick in Orion!
Thanks for the article. Interesting.
Tried DDG a few times and it struggled to come up with results that google had no issues with
I have a Kagi account - yet to really try it extensively. Isnt the free account limited to 100 searches a day or something? (having to sub for more than the limit is a downside i imagine but not if the search results are infinitely better imho).
Anyone tried Marginalia?
I have been trying to main DDG but man the results are so poor I find myself habitually adding
!gwhen I start my searches at this point.Kagi doesn't have a free tier, they just have a 100 search trial (they even label it as a trial in the pricing)
Yep. DDG hasn't been fantastic tbh.. Few times I've used Kagi its been ok.can't see myself running more than 100searches a day tbh
I think it's capped at 100 free searches per account, not per day.