On Google's Spam Woes and the Need for More Players in Search

Wednesday, January 26, 2011

Fresh on the heels of search newcomer Blekko’s launch in November, and the ensuing debate over whether it’s particularly wise for anyone to challenge Google, the last few weeks have seen a surprising, though gratifying, spate of criticisms of the declining quality of Google’s results for some searches. And, surprise surprise, by and large the critiques run parallel to issues raised by Blekko Founder, Rich Skrenta, in the months leading up to Blekko’s launch, as well as to arguments we’ve been making since Helioid’s first launch in 2008. Finally, it seems as though people are catching on to the fact that Google isn’t infallible, that it doesn’t handle every search perfectly, that the kinds of searches it struggles with are actually increasing in number, and therefore that there is indeed room in the field for fresh players.

On his personal blog, Stack Overflow Co-Founder, Jeff Atwood, laments the increase in content-copying search spam sites regularly beating out Stack Overflow pages in Google searches, simply by scraping Stack Overflow content and displaying it with more ads. On TechCrunch, Vivek Wadhwa regales us with tales of his struggles against the mass manufacturers of quasi-content, like Demand Media, which are increasingly dominating Google’s search results, without actually offering users any worthwhile information. The response from Google has been to brush these complaints off as being in response to a “recent uptick” in spammers infiltrating the top results, but Atwood points to complaints dating back to 2009, from Richard McManus and Paul Kedrosky touching on the very same issues. Kedrosky in particular shares his personal horror after struggling to find trustworthy comparisons of dishwasher brands online and being bombarded with page after page of search spam, and concludes that the “appliance search” genre is too spam-laden for even Google to handle. Clearly this problem isn’t due to an uptick in spammer activities confined to the last few months. And all of these complaints resonate strongly with an issue we discussed in our very first blog entry in 2008, which had first been raised shortly prior by Nova Spivack: that the exponential explosion of content on the web would sooner or later start to strain the ability of keyword search engines to consistently place the most relevant pages amongst the top results. It seems as though the success of content mass-manufacturers like Demand Media and Associated Content is helping Spivack’s prediction come about sooner rather than later.

[caption id="" align="aligncenter" width="368" caption="Web Spam Example"][/caption]

The good news is, the creeping dissatisfaction with at least some Google searches may help abate some of the knee-jerk skepticism of new search engines like Helioid or Blekko, especially when they propose innovative ways of filtering out the noise plaguing certain Google searches. Blekko CEO Rich Skrenta actually touches on a similar issue to the ones raised by Kedrosky and Mcmanus in the CrunchTV interview we discussed in our last blog post, concerning song lyrics. Skrenta rightly points out that if you search for lyrics online, because it’s so easy for search spammers to copy and paste lyrics supplied by trusted sites, you end up getting bombarded with gratuitous ads and may even pick up some malware. Blekko has dealt with this problem by only drawing results for lyrics searches from a limited list of trusted sites added by users under the “/lyrics” slashtag. Blekko’s ability to avoid Demand Media filler content by drawing upon user input has been one of Skrenta’s favorite reasons to give for switching to Blekko. Even a slight decline in the quality of Google’s results may push more users into experimenting a bit, and making that switch, and a little extra adventurousness on the part of the users will make for a more fertile environment for innovative search start-ups in general, including future releases of Helioid’s web exploration tools.