Twitter search sees one billion queries/day

Engineers quietly launched new search structure during same period as redesign rollout

Technology trends and news by Ronny Kerr
October 7, 2010
Short URL:

Twitter searchTwitter revealed Wednesday afternoon that during the last few weeks, while most users explored the ins and outs of the site’s redesign, a team of engineers were implementing a completely new search backend.

If you’re wondering why nobody ever complained about outages or even slowdowns related to search on Twitter, that just shows how great a job the search team did in making the switch. (Developers curious to read about the technical details involved should click through to the Twitter Engineering Blog.)

The accomplishment is even more astonishing when one considers how much data Twitter search must sift through. Users send over 1,000 tweets/second (that’s 86.4 million tweets/day). Seeking pearls in this sea of tweets, users search 12,000 queries/sec (that’s over one billion queries/day).

In fact, it’s precisely these ballooning statistics that forced Twitter to restructure its search in the first place. The search team estimates that now they’re “only using about 5% of the available backend resources,” meaning they’ve scaled search to handle at least a couple more years of growth.

All Twitter search needs now are some extra filters added to search, something that will probably land on the site sooner than later:

“[...] the new system is extremely versatile and extensible, which will allow us to build cool new features faster and better.”

It’s a bit confounding that a site sitting on such a large amount of data doesn’t have any sort of advanced search options already. After performing a search, one can filter by “Tweets,” “Tweets with links,” “Tweets near you,” and “People,” but that’s a fairly rudimentary foundation for what could be on the way.

Twitter should take notes on the recently launched LinkedIn Signal, which lets users filter imported tweets by network, industry, company, time published, region, school, or hashtag. Obviously, Twitter doesn’t have access to all the same information as LinkedIn, but the point remains that users want more options.

Why can’t we search by location or time or even (though this might sound silly) tweet length? After all, Twitter users do love their data.

Related companies, investors and entrepreneurs

Description: What is Twitter? Twitter is an online information network that allows anyone with an account to post 140 character messages, called tweet...

Related news