Twitter stream deprecation FAQ
On 2018-08-16 twitter will start shutting down the user stream API and the current direct message APIs (source)
This is often referred as "twitter killing third party apps" and, while it's not exactly the friendliest move, we can survive this.
Streams deliver real-time tweets and direct messages, along with other events which, luckily, bitlbee doesn't rely on.
August update: Over the period of a week the stream will start failing more often, starting with one hour of downtime every 6 hours and slowly increasing until it is shut down completely on 2018-08-23 (source and details). That page also says the legacy DM endpoints will stay for a month but that's only relevant for sending, receiving is part of the stream.
They also announced the text of the errors will be the following:
The Site Streams and User Streams endpoints have been turned off. Please migrate to alternate APIs. See https://t.co/usss
...which, unfortunately, doesn't display, and we get rate limited after a few reconnection attempts.
What do I need to do?
All current versions of bitlbee support an older, slower method of fetching tweets (home_timeline polling)
To use that method, disable the stream like this:
account twitter set stream off
This will be the only way to access tweets after 2018-08-23.
If you still get the following error, wait 15 minutes and try again:
<@root> twitter - Logging in: Error: Could not retrieve /1.1/friends/ids.json: 429 Too Many Requests (Rate limit exceeded)
Also, keep an eye on this page (and follow our announcements twitter account), as there will be relevant updates in the near future.
What are the downsides of disabling streams?
- New tweets will only be shown once every 60 seconds
- This includes your own after you send them ("You:")
- DMs won't be received
- Sending will work until they remove the API for it
- New follows won't get added to the nicklist without reconnecting
- Mute/unmute lists won't update without reconnecting
Are there any ways to improve this situation?
Maybe. But it needs code changes and those haven't happened yet because life keeps happening.
- We could implement the new DM endpoints in bitlbee
- The rate limit is just as awful, once every 60 seconds
- On the bright side they seem to be better designed and have newer features
- We could implement more frequent fetching of mentions, since that has a more generous rate limit
- We could do something weird with filters (see next question)
- We could add options to set your own api keys
- Useful in case you accidentally find consumer keys that don't have these rate limits
Are filter streams affected?
Oddly, no! This may change, who knows. But they aren't included in the current deprecation plans.
In theory, we could use them to get realtime tweets of some people by passing your entire following list to the follows parameter of the filter.
Known caveats:
- They have a limit of 5000 users
- May not actually fetch all the tweets (may work over a fraction of all posted tweets)
- Will definitely miss tweets from locked accounts
This requires more research.
What's this "account activity API"?
It's a replacement for some use cases of userstreams (such as bots) but not ours.
Main issues with it:
- It doesn't include the home timeline, no tweets from people you follow, only from your own account.
- It's webhook based. You need a central webserver listening in a publicly reachable address.
- It requires us to own central infrastructure for all bitlbee users.
- It only supports 15 users in the free plan. We have quite a bit more than that.
It can be ignored completely.
How do I get 280 character long tweets?
account twitter set message_length 280
Post-3.5.1 git versions of bitlbee already changed this default.
That's not directly relevant to this API change but someone asked that while I was writing this.
Does twitter hate us?
Not really, but they don't care too much about our use case either. They just tolerate us.
<iximeow> @andypiper is there an officialish reason for deprecating user streams? the replacement with the Account Activity API satisfies automated accounts somewhat (callbacks in response to being @'d etc) but ... 1/
<iximeow> for anyone with non-official clients, there's no migration path as far as i can tell. people who use and enjoy a client i've made probably can't get webhooks delivered to their computers, for a myriad of reasons. 2/
<iximeow> clearly they're going away, but i think a lot of folks would appreciate understanding why?
<iximeow> were they extra engineering burden? does twitter not want third-party clients? is it problematic w.r.t tracking user metrics? i assume all three, but it'd be nice to know.
<andypiper> It's the first, not the second or third. I believe we have been clear about that in our announcements over the past year, but it may have been buried. These are 9 year old APIs running on bare metal servers that could not reasonably be sustained.
<iximeow> it's definitely not been clear (at least to me) - appreciate an answer though! it'll be a bummer when user streams do go. they made twitter accessible in places official clients just don't work well
<dekisu> oh neat thanks for asking this, that's not how i understood it either.
<dekisu> i was still going by the older message of "avoid replicating the core twitter experience" which seems to mean giving desktop-client-like use cases the lowest priority.
<andypiper> to be doubly clear in case I'm now accidentally causing confusion: the reason for *this* change is API modernisation, infrastructure needs, & investment in key use cases. We've also stated that our API platform today is about data access, not client-like experiences.
<andypiper> ... the final part of that Tweet has not changed; the new API is not part of an effort to actively stop other use cases, but to focus on the ones that are core to us today, and to improve the infrastructure, scalability, and sustainability of the developer platform.
Can I use mastodon instead?
Yes! Wise move. See HowtoMastodon