The deep Web and the future of finding things

The discovery landscape

Wize Deep Web

A recent article in the New York Times about the “Deep Web” got us thinking about the future of finding things.  At Wize, we hold a few things dear when it comes to search and discovery. They are different. Providing search results requires some notion of intent from the end user, but often no real understanding of the information served up. Providing discovery requires a better understanding of the end user's intent, and the actual meaning of the content. 

Like a concierge, Google, and other search engines, provide an algorithmic directory of what they perceive to be credible – the Google brain looks at your request and says, this site appears to be genuine and a lot of other people refer to it, so it must be pretty good, and I think I’ll show it. Although often useful, that’s also a bit like going to Dr. Phil for medical advice, he’s probably OK, but it’s not going to be exactly what you need.  Often the information from a general search is a bit superficial and generic.  Discovery happens when a person’s inquiry finds an intelligent agent or expert to guide their efforts. This is where the Deep Web comes in, the concept that The Times reported on.

Let's first take a look at intent.  We believe intent, or said more specifically, how you plan to use something, or what you plan to do, is critical for discovery.  In our case, it’s largely product use. We think of the world through the lens of how products are actually used by consumers.  So not just the best blender in an abstract sense, but the best blender specifically for smoothies, or the best air purifier specifically for smokers.  Just as Michael Jordan was the best athlete for basketball, but not the best athlete for baseball, so too are products highly contextual to their application.  Being able to dunk from the free throw line doesn’t mean you can hit a curveball, context (or “intent”) is a critical part of the process.

For the Deep Web, it’s not just about the algorithmic popularity of a piece of content, but the actual meaning of the content. Further, it’s about layers of understanding and multiple views of the information.  The New York Times piece has an emphasis on searches of vertical databases as a primary framework for understanding the Deep Web.  Databases are clearly part of the issue, but in our minds it’s not a question of new sources, it’s a question of new interpretations. 

Just as many of us own and have access to lots of books, the value isn’t in having them, it’s in READING them. So rather than think of the Deep Web as an uber-database, we prefer to approach it as a network of experts. Experts who have already read all the right books on a certain topic by the time you get to them.

These experts will come from all walks of life and will communicate in many, many ways.  Some use micro-blogs, like a Twitter search, others may be highly-structured indexes, like Digger, or in our case a set of recommendations based on what consumers had to say about a specific product (i.e. Canon SD1100).  But the experts will either have already read, or know exactly where to find, the right information for the users specific inquiry.

The advantage that a service like Wize has over a broad-based search engine like Google, is that we know when our user is on our site they're looking for products, so we can engineer a huge amount of intelligence around that use-case.  Google has very little idea what the user is looking for when they first enter a phrase into their search box. It could literally be anything.  Google has to be ready to set you up with a mechanic, nurse, computer tech or accountant (or name-your-service here) right out of the gate.  That’s a pretty tall order.

So in many cases it’s somewhat of an apples and oranges comparison, and even the mighty Google will probably have a difficult time keeping up with every expert-discovery system out there.  So Google will continue to be the launching point, but will likely have to focus on being a concierge to the right expert as often as they are the expert themselves.  We do like the Kosmix approach, as quoted in the New York Times, “Kosmix has developed software that matches searches with the databases most likely to yield relevant information, then returns an overview of the topic drawn from multiple sources.”  Delivering the user to the right discovery expert, rather than try to be the be-all-end-all expert themselves.

We’ve also developed this topography of interesting Deep Web companies.  Each of this is a great example of an expert at work for you, take a spin through and get to know you’re new experts!

Sites featured above: Kosmix, Twitter, Powerset, Digger, BlogPulse, Hakia, Twine, RightHealth, UpTake, RapLeaf, Spock, Outside.in, Zvents, SimplyHired, Trulia, Boo-Rah, Wize, ShopWiki, TheFind, SearchMe, Like, Baynote, Loomia, Aggregate Knowledge, Searchub