Rusty just posted the initial results from RustySearch – an attempt to measure search engine relevancy by allowing users to rate listings.
Interestingly, so far it looks like all the search engines are returning equally relevant results.
Sidenote: Orion (Dr. Garcia), author of the Fractals paper, commented on Rusty’s site about problems regarding the project.
The problem with this type of test is that you are measuring user’s perception of relevancy, not what a search engine scores as relevancy. This mismatch has nothing to do with sample size. See Nacho’s post at this site.
Each SE has its own scoring functions, so it is like comparing apples with oranges. I wish you luck with the experiment.
Orion’s comment is totally backwards to me – If a search engine’s idea of relevancy is not the same as user’s idea of relevancy then what good is the search engine? As a tool for information retreival, a search engine needs to return data that users determine to be useful; ultimately a search engine’s goal should be returning the results that users perceive to be relevant. Maybe I’m not getting the point of Orion’s comment.