Search Bistro’s Henk van Ess and GoogleGuy are having a tiff over Henk’s posts covering Google’s evaluation methods. In a 12+ page thread over at WebmasterWorld, Henk and GoogleGuy are engaged in a heated exchange that makes for a great read. Just keep in mind, it is no surprise that Google is evaluating the quality of its search results. I would be shocked if they didn’t.
Henk’s reporting, although crossing the lines a bit, has uncovered interesting details regarding Google’s feedback process. But don’t read more into the documents than they really contain. This program collects feedback & quality ratings about Google’s algorithms and does not directly influence Google’s search rankings.
Henk recently released:
- Screenshots from http://eval.google.com/
- A flash movie from part of the eval site
- Some of Google’s instructions for the testers
- Google’s whitelist/spam process
- CommQuest’s interface for Google’s evaluation lab
- Google’s guidelines for evaluating random queries
- Google’s guide for spam recognition
Today, GoogleGuy confirmed Henk’s exposé was not a hoax.
The system that was up at eval.google.com was a console to evaluate quality passively, not to tweak our results actively. But when Henk van Ess submitted his own blog to Slashdot, he asserted “Real people, from all over the world, are paid to finetune the index of Google,” and that made it sound like people were reaching in via this console to tweak results directly, which just isn’t true at all.
GoogleGuy also expressed a concern that Henk’s posts may be misleading to readers.
I do believe that the assertions you’ve made are misleading. In my original post, I was replying to walkman, who asked “ok, so how do you know you’ve been manually hit by this?” which implies that walkman thought that eval.google.com was responsible for sites being hit. Likewise, I have a ton of respect for Tara Calashain at ResearchBuzz. But her post about your site says “Basically what Henk seems to have found is a part of Google that allows humans to tweak search results to ostensibly get rid of spam and let the most contextually-relevant search results rise to the top.” Again, Tara wonders whether your posts said that results were being directly tweaked. Then there are assertions from your site like “The Google testers are paid $10 – $20 for each hour they filter the results of Google.” “Filter” again makes it sound like an active process. And your self-submission to Slashdot (“Real people, from all over the world, are paid to finetune the index of Google”), which also gives the impression that people used eval.google.com to change our search results.
So yes, I looked at the wording from when you submitted your own site to Slashdot, plus the use of active verbs such as “filter” on your own site, plus the comments of smart people such as Tara and walkman and how they interpreted what you wrote, and in my opinion your posts have been misleading. Again, this was not a console in which people could directly fine-tune, tweak, filter, or otherwise modify our search results. eval.google.com was for “eval,” i.e. passive evaluation.
Your follow-up question was “Why pay them for something if it has no effect om the index? Must be charity then.” Why are you surprised that we would pay people to rate search results? The job posting has been public, after all. We do provide ways for people to volunteer to help Google (e.g. see our translation console at https://services.google.com/tc/Welcome.html ), but to rate search results consistently and well takes time and training. I think it’s perfectly normal to pay people for their time.
When you quoted me on your site, you said “Google Guy: I’ve serious reservations about Henk van Ess” and in your post you said “Google’s spokesmen Google Guy, who I love to read, has serious reservations about me.” Just to be clear, that’s not accurate: I don’t have reservations about you personally, Henk. I think I stated clearly that I have serious reservations about two of your actions. I mentioned those two specific things in my first post, and I’ll reiterate them: you took information from one of your students, and you posted information that (in my opinion) was clearly proprietary/confidential. Regarding the first, I believe you wrote in a comment on your own site that this information came from a student of yours? Regarding the second, I’m quite surprised that you assert “I’m not aware of restrictions.” Besides the copyright symbol that you mentioned earlier, the very first picture you posted has a link “An NDA Reminder…” on the left in the Important Announcements section, where NDA stands for non-disclosure agreement. Are you honestly saying that if you had realized there were restrictions, you wouldn’t have done five blog posts (so far), posted screenshots, posted employee’s real names on the web without consulting them, and posted two training documents? In that case, I’ll ask politely. Henk, this information was for ratings training. It’s copyrighted, and I’m sure that the evaluation group considers it proprietary/confidential. I’d appreciate it if you would stop posting these documents.
By the way, I apologize in advance if this post comes across as strident. I hate he-said-she-said stuff, and normally I try not to post when I’m at ruffled at all. But I do think that things like posting an innocent employee’s name from internal training documents is rude and unnecessary. Henk, feel free to include this entry on your blog, but if you do, I’d appreciate if you’d quote the entire post.
In response, Henk has pulled the names of any Google’s employees from his blog and said that he would not post on this subject anymore.
He did make one last snarky remark:
Don’t worry, (I) told Search Engine Watch that this was the last post anyway. To avoid the thought that I do anything you say, I will publish one more secret document – the one that reveals your true identity. Sorry Larry, it had to come out. Just kidding.