Yahoo vs. WolframAlpha vs. Bing?

Andrew Tomkins, Chief Scientist of Yahoo Search on how semantic technology has improved search

Bambi Francisco was recently at the 2009 Semantic Technology Conference where she interviewed Andrew Tomkins, Chief Scientist of Yahoo Search. Among the questions asked was how is Yahoo's search positioned against the newcomers WolframAlpha and Bing?

Here's the interview, partially edited.

BF: At the end of the day, the purpose of semantic technology within Yahoo Search is to really improve search results and the search experience. So how has it really improved Yahoo Search?

AT: It's been pretty dramatic so far. Yahoo has been pretty aggressive in doing semantic technology through a couple of programs. The primary one in terms of getting data in is something that we call "Search Monkey." What we have been doing is developing a set of vocabulary that publishers can use to tell the world about the structure of their data. So if they are publishing the name of the restaurant, they would put in the name of the restaurant, here is the address and the phone number. If they are a profile page publishing information about a person, then there are standards to say this is the name of the person and this is all the relevant information.

BF: Wow. It seems like a tall order to get publishers to start tagging.

AT: There has been a lot of question about what the right incentive structure is to have people do this. Everybody understands the value in the long term if everybody were to participate. So what we're doing in Yahoo Search is putting in place a set of programs where if a publisher annotates the information, a bunch of new things start to happen in the search experience. So in some cases we'll actually feature that information directly in our Web search results. So the result might have a picture, address, or phone number for a restaurant directly in line with results. It's a powerful way to present the data and it means that users are more likely to engage with it and it will drive more traffic to the publisher.

BF: So do the coders of the publishing platform actually having to insert this stuff into the code?

AT: There are two things that we do. We allow the coders of the publishing platform to put in the code in a way that it can be used by anybody. So we've been pushing this, but there is uptake from others, like Google who will consume this data. So there is a growing ecosystem for people who are ready for the data.

BF: How many publishers are using this?

AT: We don't have numbers on publishers. But we do have some numbers in what's happening in the actual search results. We have a set of partners where results will show a richer presentation all the time. So if you search restaurant on Yahoo Search, you might get information from Yelp, or recipe data from Epicurious, or some media companies where there will be more than just two lines of text describing the result.  We'll actually have embedded media and we've been expanding that to include certain formats that will actually give the ability to tell us about video on your site, a presentation, or even a game. You can actually play the game on the search.

BF: So, this is how you are improving the search experience?

AT: Yes and we are seeing up to 15% in click through rate through these rich results which is enormous for us.

BF: What was the click through rate before?

AT: These are algorithmic search results so they tend to be pretty high and with a 15% boost on top of that is a big deal.

BF: What are those clicks?

AT: We don't specifically talk about that.

BF: With new players in this space such as Wolfram Alpha and Bing. How do you  position yourself against them? How is Yahoo different from them?

AT: We see Bing released and going after decision support capabilities which are very important. We see Wolfram Alpha going after structured data. What we see at Yahoo is that these are very important constituents to the puzzle. The puzzle is really about task completion, understanding the users intent, and mapping that to the right tools to get things done. So at Yahoo, we're focused on the task, the user intent, and what can we do in the search experience to move you further down the path of where you're trying to get. So we're looking to put tools in place.

BF: Is the ultimate goal just to put standards around the meaning of this data? How far are we from that and will we ever get there?

AT: I don't know if we'll ever get there. We’ve started. We’ve spent a lot of years trying to understand the right open standards and that's been in happening in a very broad community. Yahoo has been playing a role of an adopter of those standards as well as expanding them into new domains. When we offer publishers to give us a product result or a recipe result, then in some cases we will need to work with the community to work out the exact vocabulary to describe a product. What is the name, title, and description? So we are still working out all the details. We’re at the forefront. We'll get there. You will see many changes.

BF: We haven't made any quantum leaps in semantic technology but someday we will.

AT: We will.