Twitter Strikes Again

Twitter have decided to drastically reduce the number of search API calls that Pluggio is allowed to make per hour. Before this reduction Pluggio had the capacity to make 10,000 calls per hour which was perfectly fine for Pluggio’s 3,700 users.

This breaks “Search” and “Friend Finder”.

When I asked Twitter why? Here was the response…

Hi Justin,

Thank you for your continued interest in our API. Instead of using the Search API to accomplish your application’s needs, we encourage you to look into the Streaming API instead: http://dev.twitter.com/pages/streaming_api . Specifically, you can establish one connection to the Streaming API that pipes in all the tweets containing at least one of your users’ keywords, instead of repeatedly sending search queries, using the statuses/filter method with the track parameter. Additionally, you can apply for elevated Streaming API access roles (such as track restricted, which allows you to track up to 10,000 keywords) by filling out the form at http://twitter.com/help/request_streaming . I hope that you are able to implement your app with the Streaming API, and I’m sorry for any inconvenience that this may cause you and your team.

In order to make Pluggio work with this new system it will take me 2-3 days hard core programming.

So hopefully this should be resolved my the end of the weekend.

I apologize for this service interruption and am doing everything I can to get things back on line.

Justin

Update Thu April 22nd 9:38PM PST

After thinking about this I have to say I’m quite disappointed that yet again Twitter have shut down large functionality on Pluggio without ANY kind of warning. Given that Pluggio now has almost 3700 registered users it would have been very easy for them to have sent a warning email offering me a few weeks grace to implement this new API. Which, incidentally is no walk in the park to integrate and has all sorts of ramifications about what type of server I can use for Pluggio.

Update Fri April 22nd 1:22AM PST

I have some news about this issue. It is not very good news. And makes me even more frustrated that Twitter shut down Pluggio’s search capability without warning.

Rather than use their search API they want Pluggio to use the stream API which in essence is a completely different concept to the search API. It’s very complex but I will try to explain.

With the existing “Friend Finder” pluggio will cycle through your keywords once every three hours and look for people who tweet about that topic.

So, let’s say you have #php, #apple, #ipad

Then pluggio will search for #php and then three hours later will search for #apple and three hours later will search for #ipad and then cycle back to the beginning again.

So pluggio just searches once for each term and twitter sends back the latest 20 tweets. Pluggio recommends the tweeters as friends. Simple.

As explained this no longer works because twitter turned it off.

With the stream API it works differently. Pluggio only gets access to one “real time” stream. The tweets that get sent to the stream are specified by “registering” keywords. So in the above example if you had the following friend finder words of #php, #apple, #ipad. Pluggio would need to register them and then they would come in, in real time in “one single stream” that pluggio has to store and deal with. The problem is that since Pluggio is only allowed “one stream” ALL users keywords on Pluggio need to be registered with the one stream.

At the moment Pluggio users have setup just under 1000 keywords. I ran a test with the stream with only 100 keywords plugged in, and the stream received about 5000 tweets in two minutes. By those calculations the stream will be pulling in 25,000 tweets per minute with 1000 keywords. That’s 36 million tweets per day.

I hope you can see that making Friend Finder work with the stream API is VERY different to the search API. With the search API as it is, all Pluggio needs to do is ONE search to get back 20 results every three hours. With the new stream API Pluggio has to deal with 36 million tweets a day (and that’s just with the current user base of 3700 users).

So, long story short. In this instance I don’t actually know what to do. There’s no way I will be able to build a system that deals with 36 million daily tweets by the end of the weekend (let alone the fact that twitter have not even given me access to that many search terms with the stream – Pluggio can only access about 100 searches right now).

One possibility is to make Friend Finder for premium accounts only and that way it would limit the stream, but even then it will be pulling in over 5000 tweets a minute with the current paid user-base.

Also, just to add insult to injury, the stream API does not allow keyword/location searches in a way that would be possible to tie back to any user who was using friend finder. So no matter what happens that is one nice function that will not work in the future.

Needless to say I’m upset about all of this and will be looking for some kind of solution over the next week or so.

All I can do is ask you to bear with me while I think about how to tackle this huge issue.