I can never think of any "fair" searches to try. So often I just use google to get to a specific webpage from the Firefox search bar faster than I could by typing in the full address. If I search for something with a specific result in mind, all 3 engines will nearly always give me that as the first result (for example: woot, eztv, twit).
I never use search engines to look for a generic term the way search engines are always marketed, for example searching for "buy used textbooks" or "cheap prescription drugs" or "book a flight" since those results are always so commercialized that they're generally useless anyway.
Yesterday, I had two searches for which Google was failing me, "prius chipmunks" and "ie6 image reload", so on a whim I tried bing, and bing had the answer in the top three for both. I was really surprised. I'll keep using Google, but I'll probably try bing when I'm stuck in the future.
Search results are important, but they aren't the _only_ thing that matters here.
Which one is faster?
Which has a better UX?
Which should trust with my data?
Which integrates with other products I like?
Which one works best on my phone?
...
I'd argue that search results trump many of these. The UIs (including mobile UIs) of most search engines these days are virtually identical, although maybe slightly flashier in some cases. No Cuil here. If a search engine offers better results but is a little slower, then that's an advantage over having to flip through pages in a faster search engine.
While this is true to a certain extent, it only becomes true once search results are good enough to no longer be the deciding factor.
Even if cuil had a mobile version which magically doubled the screensize and gave me a magical flip out keyboard, it still would be a bad way to locate information.
The methodology for this is flawed because people will tend to search for things they have searched for before on Google and then pick Google as the best because they find that ranking more familiar.
What would you propose instead? It seems that asking people which they prefer is the only way to do this, even if it does stilt towards the market leader.
Here's a thought.. build a list of questions. Give them a search box to find the answer, using their own queries, and let them choose which result set was most helpful.
"What is" "Who is" "How was" type of questions.
Would be much more work, but could be worth it if you're trying to answer this question accurately.
Tho personally, I've begun using Bing in addition to Google. I think this is what most people will do. The idea that we only have room in our lives for one search engine seems silly. Doesn't anyone remember what life was like before Google? I'd often use multiple engines for a given query -- and each engine had links to 'Try this search on ____' at the bottom of each result page.
Google has been so far better than the competition that such behavior has died off. But I think it's going to make a comeback now. I guess we'll see.
The best way I can think of would be to have a long term test where a number of users did all of their searching through a site that would randomly show the results of one search engine, and then have the users rate their satisfaction with the results. It wouldn't be quite as good at cross comparing, but it would have a much more realistic selection of searches to draw from, and it would reduce the sort of biases that the gp post was talking about.
Maybe kick it off with the first search pre-populated from, say, a dictionary word. (Though maybe that's not fair either, if it isn't the type of search you're likely to perform in real life)
It seems a bit like the author is rationalizing why he's still using Google. Although I personally despise Google for the way they mangle my search queries, I don't think there's a need to explain why you choose to use the search engine you use.
Creating your "own" opensearch provider for any search engine on the internet is ridiculously easy.
In fact, going to any search engine which allows you to add a search provider should give you a simple enough template to work with. Just customize the URLs used there and point it at this thing. Then make a html file with the needed html-header field and you are good to go.
Google: 45%, Bing: 33%, Yahoo: 22% | 9,496 votes
While they're all the same order of magnitude, I wouldn't call that "virtually identical"...