Improving Google's Search Quality

Business Week has an interview with Scott Huffman, who runs Google's search evaluation team. Matt Cutts calls it "the deepest public discussion of how Google evals search quality that I've ever seen".

The interview has a lot of details about human evaluators, search experiments and the way Google measures search quality.

"People have a tendency to pick what we would call really popular queries [to compare search engines]. Look, I typed flowers in both and this one showed me pictures and this one showed me flowers to buy, and I like pictures better. But we get a lot of queries that are a lot more rare than that, and we feel that our gap in terms of the competition really opens up there."

But that doesn't mean popular queries aren't important.

"On a continuous basis in every one of our data centers, a large set of queries are being run in the background, and we're looking at the results, looking up our evaluations of them and making sure that all of our quality metrics are within tolerance. (...) Both broad query sets and navigational query sets, like "San Francisco bike shop" to the more mundane, like: Here’s every U.S. state and they have a home page and we better get that home page in the top results, and if we don't … then literally somebody's pager goes off."

Scott Huffman says that Google constantly improves ranking and search quality, even if that's not always obvious. People usually notice user interface changes and Google prepares many visible changes.

"We're doing a lot more on the UI side. We're trying to do a lot more experimentation — how can we push the envelope? You don't want to be just the 10 blue links. I see us definitely trying to be more aggressive with a lot of features that will start to show up on the search page. Even if you look at the search page today compared with a couple years ago, it's actually quite a bit different. On the UI side, my impression is that the bar for what people expect from search engines is higher today. That's partly because of the features Google and others have added."