DataSift unleashes the full Twitter Firehose

Developers looking to monitor and analyze tweet data from the entire microblogging site now can

MediaSift Ltd., a developer of social media tools and platforms for the real-time social Web, announced Monday that it and Twitter have partnered up to make the full Twitter Firehose commercially available. The tweets will be accessible through DataSift, a curation and data-mining platform created by MediaSift.

DataSift, which was the highlight of MediaSift’s announcement at the Data 2.0 Conference in San Francisco, gives developers the ability to “monitor and analyze Twitter data” using any number of customizable filters.

What’s convenient about the new product is its pricing scheme:

DataSift aggregates multiple social media feeds, augmenting additional data sets, and creating a common abstraction layer (through CSDL), to provide meaningful insight into unstructured data chaos! But we wanted to do more. We wanted to revolutionise the economics. So we built a Pay Per Use subscription model.

Basically, DataSift-using developers will have access to the entire Twitter Firehose, but will only pay for the parts they use.

The announcement from MediaSift and Twitter echoes one from last November, when social media data company Gnip unveiled its own partnership with Twitter to sell access to parts of the Twitter Firehose. Key word: “parts.” Because Gnip is more focused on capturing what everyone is saying, in general, the Halfhose (50% of the full Firehose) and the Decahose (10 percent of the full Firehose) are intended to suffice as statistically relevant pools of data.

DataSift, on the other hand, draws literally every tweet on Twitter, based on a filter. For example, every single tweet about Lady Gaga or tweets from California about Kenneth Cole.

The official launch of DataSift is slated for early Q3 2011 and, at the end of the year, MediaSift plans to release another product analogous to DataSift, except targeted at consumers instead of developers.

MediaSift actually has experience with a consumer-facing Twitter service in TweetMeme, which aggregates the most popularly shared links on Twitter in a Digg-like format. But DataSift takes the kind of data available on Twitter to a whole new level.