Differences between revisions 1 and 5 (spanning 4 versions)
Revision 1 as of 2018-05-17 23:56:45
Size: 5578
Editor: 94-98-17-190
Comment: create page
Revision 5 as of 2018-08-17 17:59:57
Size: 6604
Editor: dx
Comment: mention rate limit
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
On 2018-08-16 twitter will shut down the user stream API and the current direct message APIs ([[https://twitter.com/TwitterDev/status/996782994181558273|source]]) On 2018-08-16 twitter will start shutting down the user stream API and the current direct message APIs ([[https://twitter.com/TwitterDev/status/996782994181558273|source]])
Line 8: Line 8:

'''August update:''' Over the period of a week the stream will start failing more often, starting with one hour of downtime every 6 hours and slowly increasing until it is shut down completely on 2018-08-23 ([[https://twittercommunity.com/t/details-and-what-to-expect-from-the-api-deprecations-this-week-on-august-16-2018/110746|source and details]]). That page also says the legacy DM endpoints will stay for a month but that's only relevant for sending, receiving is part of the stream.

They also announced the text of the errors will be the following:

    The Site Streams and User Streams endpoints have been turned off. Please migrate to alternate APIs. See https://t.co/usss

...which, unfortunately, doesn't display, and we get rate limited after a few reconnection attempts.
Line 19: Line 27:
'''Don't do this yet!''' Enjoy the stream while you have it. But this will be the only way to access tweets after that date. This will be the only way to access tweets after 2018-08-23.

If you still get the following error, wait 15 minutes and try again:

{{{
<@root> twitter - Logging in: Error: Could not retrieve /1.1/friends/ids.json: 429 Too Many Requests (Rate limit exceeded)
}}}
Line 34: Line 48:
Over the next couple of months we will work on some improvements and prepare a release: Maybe. But it needs code changes and those haven't happened yet because life keeps happening.
Line 36: Line 50:
 * We will implement the new DM endpoints in bitlbee
   * The rate limit is just as awful, once every 60 seconds.
 * We could implement the new DM endpoints in bitlbee
   * The rate limit is just as awful, once every 60 seconds
Line 39: Line 53:
 * We might implement more frequent fetching of mentions, since that has a more generous rate limit.
 * We might do something weird with filters (see next question)
 * We will add options to set your own api keys
   * Useful in case you accidentally find consumer keys that don't have these rate limits.
 * We could implement more frequent fetching of mentions, since that has a more generous rate limit
 * We could do something weird with filters (see next question)
 * We could add options to set your own api keys
   * Useful in case you accidentally find consumer keys that don't have these rate limits
Line 48: Line 62:
In theory, we could use them to get realtime tweets of some people, by passing your entire following list to the follows parameter of the filter. In theory, we could use them to get realtime tweets of some people by passing your entire following list to the follows parameter of the filter.
Line 104: Line 118:

== Can I use mastodon instead? ==

Yes! Wise move. See HowtoMastodon

Twitter stream deprecation FAQ

On 2018-08-16 twitter will start shutting down the user stream API and the current direct message APIs (source)

This is often referred as "twitter killing third party apps" and, while it's not exactly the friendliest move, we can survive this.

Streams deliver real-time tweets and direct messages, along with other events which, luckily, bitlbee doesn't rely on.

August update: Over the period of a week the stream will start failing more often, starting with one hour of downtime every 6 hours and slowly increasing until it is shut down completely on 2018-08-23 (source and details). That page also says the legacy DM endpoints will stay for a month but that's only relevant for sending, receiving is part of the stream.

They also announced the text of the errors will be the following:

  • The Site Streams and User Streams endpoints have been turned off. Please migrate to alternate APIs. See https://t.co/usss

...which, unfortunately, doesn't display, and we get rate limited after a few reconnection attempts.

What do I need to do?

All current versions of bitlbee support an older, slower method of fetching tweets (home_timeline polling)

To use that method, disable the stream like this:

account twitter set stream off

This will be the only way to access tweets after 2018-08-23.

If you still get the following error, wait 15 minutes and try again:

<@root> twitter - Logging in: Error: Could not retrieve /1.1/friends/ids.json: 429 Too Many Requests (Rate limit exceeded)

Also, keep an eye on this page (and follow our announcements twitter account), as there will be relevant updates in the near future.

What are the downsides of disabling streams?

  • New tweets will only be shown once every 60 seconds
    • This includes your own after you send them ("You:")
  • DMs won't be received
    • Sending will work until they remove the API for it
  • New follows won't get added to the nicklist without reconnecting
  • Mute/unmute lists won't update without reconnecting

Are there any ways to improve this situation?

Maybe. But it needs code changes and those haven't happened yet because life keeps happening.

  • We could implement the new DM endpoints in bitlbee
    • The rate limit is just as awful, once every 60 seconds
    • On the bright side they seem to be better designed and have newer features
  • We could implement more frequent fetching of mentions, since that has a more generous rate limit
  • We could do something weird with filters (see next question)
  • We could add options to set your own api keys
    • Useful in case you accidentally find consumer keys that don't have these rate limits

Are filter streams affected?

Oddly, no! This may change, who knows. But they aren't included in the current deprecation plans.

In theory, we could use them to get realtime tweets of some people by passing your entire following list to the follows parameter of the filter.

Known caveats:

  • They have a limit of 5000 users
  • May not actually fetch all the tweets (may work over a fraction of all posted tweets)
  • Will definitely miss tweets from locked accounts

This requires more research.

What's this "account activity API"?

It's a replacement for some use cases of userstreams (such as bots) but not ours.

Main issues with it:

  • It doesn't include the home timeline, no tweets from people you follow, only from your own account.
  • It's webhook based. You need a central webserver listening in a publicly reachable address.
  • It requires us to own central infrastructure for all bitlbee users.
  • It only supports 15 users in the free plan. We have quite a bit more than that.

It can be ignored completely.

How do I get 280 character long tweets?

account twitter set message_length 280

Post-3.5.1 git versions of bitlbee already changed this default.

That's not directly relevant to this API change but someone asked that while I was writing this.

Does twitter hate us?

Not really, but they don't care too much about our use case either. They just tolerate us.

  • <iximeow> @andypiper is there an officialish reason for deprecating user streams? the replacement with the Account Activity API satisfies automated accounts somewhat (callbacks in response to being @'d etc) but ... 1/

    <iximeow> for anyone with non-official clients, there's no migration path as far as i can tell. people who use and enjoy a client i've made probably can't get webhooks delivered to their computers, for a myriad of reasons. 2/

    <iximeow> clearly they're going away, but i think a lot of folks would appreciate understanding why?

    <iximeow> were they extra engineering burden? does twitter not want third-party clients? is it problematic w.r.t tracking user metrics? i assume all three, but it'd be nice to know.

    <andypiper> It's the first, not the second or third. I believe we have been clear about that in our announcements over the past year, but it may have been buried. These are 9 year old APIs running on bare metal servers that could not reasonably be sustained.

    <iximeow> it's definitely not been clear (at least to me) - appreciate an answer though! it'll be a bummer when user streams do go. they made twitter accessible in places official clients just don't work well

    <dekisu> oh neat thanks for asking this, that's not how i understood it either.

    <dekisu> i was still going by the older message of "avoid replicating the core twitter experience" which seems to mean giving desktop-client-like use cases the lowest priority.

    <andypiper> to be doubly clear in case I'm now accidentally causing confusion: the reason for *this* change is API modernisation, infrastructure needs, & investment in key use cases. We've also stated that our API platform today is about data access, not client-like experiences.

    <andypiper> ... the final part of that Tweet has not changed; the new API is not part of an effort to actively stop other use cases, but to focus on the ones that are core to us today, and to improve the infrastructure, scalability, and sustainability of the developer platform.

Can I use mastodon instead?

Yes! Wise move. See HowtoMastodon

BitlBee Wiki: HowtoTwitter/StreamDeprecation (last edited 2018-08-17 17:59:57 by dx)