The deep Web and the future of finding things

Tom Patterson · March 16, 2009 · Short URL: https://vator.tv/n/765

The discovery landscape

Wize Deep Web

A recent article in the New York Times about the “Deep Web” got us thinking about the future of finding things.  At Wize, we hold a few things dear when it comes to search and discovery. They are different. Providing search results requires some notion of intent from the end user, but often no real understanding of the information served up. Providing discovery requires a better understanding of the end user's intent, and the actual meaning of the content. 

Like a concierge, Google, and other search engines, provide an algorithmic directory of what they perceive to be credible – the Google brain looks at your request and says, this site appears to be genuine and a lot of other people refer to it, so it must be pretty good, and I think I’ll show it. Although often useful, that’s also a bit like going to Dr. Phil for medical advice, he’s probably OK, but it’s not going to be exactly what you need.  Often the information from a general search is a bit superficial and generic.  Discovery happens when a person’s inquiry finds an intelligent agent or expert to guide their efforts. This is where the Deep Web comes in, the concept that The Times reported on.

Let's first take a look at intent.  We believe intent, or said more specifically, how you plan to use something, or what you plan to do, is critical for discovery.  In our case, it’s largely product use. We think of the world through the lens of how products are actually used by consumers.  So not just the best blender in an abstract sense, but the best blender specifically for smoothies, or the best air purifier specifically for smokers.  Just as Michael Jordan was the best athlete for basketball, but not the best athlete for baseball, so too are products highly contextual to their application.  Being able to dunk from the free throw line doesn’t mean you can hit a curveball, context (or “intent”) is a critical part of the process.

For the Deep Web, it’s not just about the algorithmic popularity of a piece of content, but the actual meaning of the content. Further, it’s about layers of understanding and multiple views of the information.  The New York Times piece has an emphasis on searches of vertical databases as a primary framework for understanding the Deep Web.  Databases are clearly part of the issue, but in our minds it’s not a question of new sources, it’s a question of new interpretations. 

Just as many of us own and have access to lots of books, the value isn’t in having them, it’s in READING them. So rather than think of the Deep Web as an uber-database, we prefer to approach it as a network of experts. Experts who have already read all the right books on a certain topic by the time you get to them.

These experts will come from all walks of life and will communicate in many, many ways.  Some use micro-blogs, like a Twitter search, others may be highly-structured indexes, like Digger, or in our case a set of recommendations based on what consumers had to say about a specific product (i.e. Canon SD1100).  But the experts will either have already read, or know exactly where to find, the right information for the users specific inquiry.

The advantage that a service like Wize has over a broad-based search engine like Google, is that we know when our user is on our site they're looking for products, so we can engineer a huge amount of intelligence around that use-case.  Google has very little idea what the user is looking for when they first enter a phrase into their search box. It could literally be anything.  Google has to be ready to set you up with a mechanic, nurse, computer tech or accountant (or name-your-service here) right out of the gate.  That’s a pretty tall order.

So in many cases it’s somewhat of an apples and oranges comparison, and even the mighty Google will probably have a difficult time keeping up with every expert-discovery system out there.  So Google will continue to be the launching point, but will likely have to focus on being a concierge to the right expert as often as they are the expert themselves.  We do like the Kosmix approach, as quoted in the New York Times, “Kosmix has developed software that matches searches with the databases most likely to yield relevant information, then returns an overview of the topic drawn from multiple sources.”  Delivering the user to the right discovery expert, rather than try to be the be-all-end-all expert themselves.

We’ve also developed this topography of interesting Deep Web companies.  Each of this is a great example of an expert at work for you, take a spin through and get to know you’re new experts!

Sites featured above: Kosmix, Twitter, Powerset, Digger, BlogPulse, Hakia, Twine, RightHealth, UpTake, RapLeaf, Spock, Outside.in, Zvents, SimplyHired, Trulia, Boo-Rah, Wize, ShopWiki, TheFind, SearchMe, Like, Baynote, Loomia, Aggregate Knowledge, Searchub

Image Description

Tom Patterson

Tom is Founder of Heighten. Pres. at Nextag, CEO Wize, EIR at Mayfield , SVP Markettools, Board Dale Carnegie, weekly wisdom @ http://wizebits.com

All author posts

Related Companies, Investors, and Entrepreneurs

@WalmartLabs (Kosmix)

Startup/Business

Joined Vator on

Kosmix was acquired by Walmart in May of 2011 to create @WalmartLabs.

Through the innovative fusion of retail, social and mobile, @WalmartLabs is redefining Commerce for the largest retailer worldwide. We are a group comprised of the brightest technologists and businesspeople in the industry, excited about the limitless opportunities that this next generation of Commerce will bring to billions of people around the globe, all in an effort to help them save money and live better.

Kosmix was funded by Time Warner Investments, Accel Partners, Lightspeed Venture Partners, Dag Ventures, private investor Ed Zander and Jeff Bezos' personal investment company, Bezos Expeditions.

***

WalmartLabs is hiring.

http://www.walmartlabs.com/open-positions/

 

TextDigger

Startup/Business

Joined Vator on

TextDigger is a San Jose, CA-based startup developing advanced semantic solutions for the Web, including hosted semantic search, automated content tagging and topic generation, and optimized keyword generation. These products make Web pages more findable, both to outside search engines such as Google and to other pages within the same site via cross-linking and related search. The result is increased revenue from higher inbound traffic and longer sessions. TextDigger was founded by a group of former CNET employees and executives who developed patented linguistic technologies that, today, are used to enhance the content on thousands of pages within CNET's award winning websites.

Bullpen Capital

Angel group/VC

Joined Vator on

Bullpen Capital is second round investor who invests in companies previously seeded by the Super Angel funds.

Searchme

Startup/Business

Joined Vator on

Searchme lets you see what you’re searching for. As you start typing, categories appear that relate to your query. Choose a category, and you’ll see pictures of web pages that answer your search. You can review these pages quickly to find just the information you’re looking for, before you click through.

We’re just getting started on our first step towards creating an entirely new way to search the Web. The quality of our results will vary as we make Searchme better and better. If you have comments or suggestions we’d love to hear from you. We love feedback.

We’re also looking for more great people to join our team, as well as ways to partner with companies to deliver more value to our users. And, of course, we’re always happy to chat with the press.

Thanks for using Searchme!

RapLeaf

Startup/Business

Joined Vator on

Rapleaf helps companies build personalized services for their customers.

Wize.com

Startup/Business

Joined Vator on

Finding the product that's right for you is hard. Reading thousands of reviews and picking from hundreds of products is difficult and frustrating. It takes a lot of time to make sense of all this information and find the product that fits your needs. We built Wize.com to solve this problem.

Wize gathers millions of product reviews from websites like Amazon.com, Walmart.com, and CNET. Using our proprietary technology we figure out which products are the best for how people want to use them.

Looking for a Television for "Watching Movies?" Simply tell us what you're looking for, and we'll read through all the product reviews from the Web to help you find the best product.

Want more information about products before you buy? Learn more about a product by seeing all the information, pictures, pros and cons, reviews, ways to use it, and stores to buy it.

Not sure which product to choose from? Compare the products you like using our product comparison, and find the best product that's right for you.