If you’re wondering why nobody ever complained about outages or even slowdowns related to search on Twitter, that just shows how great a job the search team did in making the switch. (Developers curious to read about the technical details involved should click through to the Twitter Engineering Blog.)
The accomplishment is even more astonishing when one considers how much data Twitter search must sift through. Users send over 1,000 tweets/second (that’s 86.4 million tweets/day). Seeking pearls in this sea of tweets, users search 12,000 queries/sec (that’s over one billion queries/day).
In fact, it’s precisely these ballooning statistics that forced Twitter to restructure its search in the first place. The search team estimates that now they’re “only using about 5% of the available backend resources,” meaning they’ve scaled search to handle at least a couple more years of growth.
All Twitter search needs now are some extra filters added to search, something that will probably land on the site sooner than later:
“[…] the new system is extremely versatile and extensible, which will allow us to build cool new features faster and better.”
It’s a bit confounding that a site sitting on such a large amount of data doesn’t have any sort of advanced search options already. After performing a search, one can filter by “Tweets,” “Tweets with links,” “Tweets near you,” and “People,” but that’s a fairly rudimentary foundation for what could be on the way.
Twitter should take notes on the recently launched LinkedIn Signal, which lets users filter imported tweets by network, industry, company, time published, region, school, or hashtag. Obviously, Twitter doesn’t have access to all the same information as LinkedIn, but the point remains that users want more options.
Why can’t we search by location or time or even (though this might sound silly) tweet length? After all, Twitter users do love their data.