Hi Developers,
I'm sure you've read @jasoncosta's blog post announcing the new version of the API. This thread is to collect a few loose ends, upcoming improvements, and known errata.
- Look for the blue pills that will indicate what version of content you're looking at. v1.1
- Check out the Overview: Version 1.1 of the Twitter API for a high-level summary on what's changed.
- The entire 1.1 API requires a user context when authenticating at this time with OAuth 1.0A. That means an oauth_token representing an access token must be present in each request. In the near future we'll have an additional form of authentication for a "userless" or application-only context for many methods.
- Write operations are considered the same way as they've always been in API version 1.0. Accounts have specific allowances for tweeting and sending direct messages and so on and those are still intrinsic to the account and handled similarily. In the near future we'll have the same per-method rate limiter also handle write operations, allowing for more dynamic & generous limiting on content creation and modification.
- While the majority of 1.1's documentation is present, you'll find us continuing to update much of the documentation throughout the next few weeks. Report any documentation errors you find for API v1.1 here or in this thread.
- If you've been experiencing rate limiting on widgets at your corporate office or other shared network, you'll definitely want to check out our new embedded timelines.
- API v1.1 has a concept that hasn't been well-documented yet: "user entities" -- these are primarily to identify and resolve t.co links in user objects. For most developers experienced with entities, the meaning of these fields should be clear. Expect explicit documentation on this soon.
- Some methods may still support a "per_page" parameter when "count" is the parameter we intend. These are few and far between and will be rectified in the coming weeks.
- Occasionally you may see a documentation page that claims it may represent deprecated information. Unless it's a 1.0 resource method, consider it a warning and use your own judgement on whether the documentation represents a conflict with the version 1.1 world.
- Let us know if you see any issues with error response conditions -- whether they occur unexpectedly, if the response format is not in JSON, or if the status line and payload is unclear. We'll be improving the Error Codes & Responses documentation to include more information on the better-structured errors you'll be encountering in 1.1.
- You may find it difficult to properly tweet with /1.1/statuses/update_with_media at the moment (UPDATE: this has been resolved, use https://api.twitter.com/1.1/statuses/update_with_media.json for this operation)
- The Search API in 1.1 makes a few references to its deprecated page parameter. It also may occasionally be missing user objects. Fixes forthcoming. (UPDATE: 1.1 Search now has proper retweeted_status objects, user objects, and an improved search_metadata -- see When using "GET account/verify_credentials" - What parameters need to be passed in?
- We're still updating our API v1.1 FAQ. Please keep the questions coming!
The entire platform relations team is excited to work with you on API v1.1, the first major upgrade the API has seen in some time.
Thanks!
Taylor Singletary
@episod
Replies
The rate limits for followers/ids and friends/ids really are too low for processing larger Twitter accounts. With this rate limit, one can retrieve a maximum of 300,000 ids per hour, or 75,000 every 15 minutes.
Any chance of this being bumped up to the same rate limit as users/lookup?
It also begs the question, for what duration is a cursor valid? If 15 batches of 5,000 ids are retrieved in Minute-#1 of 15-Minute-Period-#1, will the next-cursor still be valid in Minute-#1 of 15-Minute-Period-#2?
This also jumped out at me - it seems like pulling down social graph information has been severely limited for large accounts.
We've gone from 1,750,000 followers+following per hour to 75,000 followers + 75,000 following per 15 mins.
Is this correct?
Hey twitter folks, any updates on these questions?
The limits in place are meant to serve the majority of user use cases. There are some outliers that will take longer to process within these limits. Cursoring is built in a way that will be resilient across longer periods of time.
Just wanted to add a +1 on followers/ids rate limit of 15 per 15 minute window is too low, that's a poor 60 calls I can make in an hour! I have a few users with accounts way above 1M+ followers and essentially can't provide them with quality service (all data is between 6 and 10 hours old at least, and that would assume my server is only handling this user).
So yes, not sure if there is a thread where people can give feedback on which rate limits we desperately want to have upped again, that would be my wish.
followers/ids at 50 / 15 minute window would already mean the world.
+1 While we might be minority. I think we're too severely affected :(
+15 (per 15 minute black of time)...
Just want to echo what's been said above. I do a good deal of network analysis on Twitter follower and friends networks -- the v1.0 general limit of 350 has never given me problems, but if I'm looking at a hard limit of 60 calls per hour for this endpoint, I foresee problems.
Is it 60 calls per endpoint per hour per user per app?
The reason I'm asking is because if I'm a user and have tweetdeck on my phone, the twitter app on my ipad and something else open on my desktop all querying my timeline endpoint, they all share the 60/hr limit? That's only 20/hr/app if I've got 3 apps, right?
Also, with the streaming API, I seem to be able to connect to the 1.1 api using only username & pw (like with the 1.0 API). Is this a bug or will the streaming API continue to not require OAUTH?
All of the rate limits are in 15 minute windows. It's best to think of them within those groups of time instead of as hourly or minutely limits.
Unlike in API v1, all rate limiting is per user per app -- IP address has no involvement in rate limiting consideration. Rate limits are completely partitioned between applications.
The Streaming API still allows basic auth. It won't forever. Use OAuth for better future proofing.
In "GET users/lookup", you state:
"You are strongly encouraged to use a POST for larger (up to 100 screen names) requests."
If that's the case, why even support and document GET? Why not just support and document POST? There's another API call (GET users/show) if you just want to GET a single user. My guess is that most applications will be using this to acquire large lists of user objects, 100 at a time.
GETs are still easier to perform for many users. There are many use cases for all of our APIs, and while most bulk uses of users/lookup will want to use POST, there are less-heavy usages where the simplicity of a GET may be preferred.
The "recently updated" page contains 7 pages of results for September 4th and 5th, with no indication of what the deltas are:
https://dev.twitter.com/docs/recent
It would be really handy to know how each API endpoint changed. Is it just the rate limit? Or did the payload get some new information? Are there new parameters for the GET & POST? Are there any new endpoints? What endpoints have gone away?
Or are the changes limited to what's been stated in the overview?
https://dev.twitter.com/docs/api/1.1/overview
Finally, I don't see any mention of what the new base URL will be: is it "https://api.twitter.com/1.1/"? And is this endpoint live so we can start testing against it?
Consider that every resource in API v1.1 has changed in some way and read the documentation for the resources you make use of. In some cases the response body may have changed slightly, in other cases they have a per-endpoint rate limit defined for them that they did not have before, in some cases the resource URL itself has changed, in some cases previously available methods are no longer existant. Many changes in the API were across the board for all methods that supported entities or retweets (most objects support these). API v1.1 is a mostly new API and should be evaluated with new eyes.
API v1.1 is live. The methods documented in https://dev.twitter.com/docs/api/1.1 are the methods that exist in API v1.1 and can all be found at https://api.twitter.com/1.1/* -- except for upload.twitter.com/1.1/statuses/upload_with_media.
Thanks Taylor, looking at it in detail now.
Is the endpoint for OAuth requests still "https://api.twitter.com/oauth/", or should this be something like "https://api.twitter.com/1.1/oauth"?
Thanks, Craig. I know it's a lot to take in and overall it's pretty similar to 1.0 and the pre-1.0 days, but there are enough subtle differences that I think it's worthwhile to dive in completely. Happy to help you derive any specific nuances you have questions about.
The process of obtaining authorization and OAuth token negotiation are independent of REST API versioning and you needn't (and shouldn't) provide a version element to the path when accessing. Stick with https://api.twitter.com/oauth/* for all of those operations whether using 1.0 or 1.1 API calls ultimately. The access tokens they yield are the same regardless.
How can we see how many user tokens we have out? Also, is there a way to proactively expire them from the app side? If we have 120k tokens out there, but only 25% active users then unless the non-active users remove our token on the Twitter.com side, we would be out of luck.
At this time there's no easy way to determine what your current token count is on an application. We'll provide an easier way to make this determination in the future.
There's currently no way to revoke access on behalf of your users.
This would be really helpful :)
Yes, it will be easier to make business decisions if we have an idea of where we are rather than navigating in the fog
Agreed, we find that people typically don't invalidate their tokens when they stop using an app. As application owners, we will need tools to manage that if you are going to limit how many tokens we have.
Optionally, you can limit the number of active tokens we have, where active is defined as being used within the last 30 days.
Hi, I have a question about the changes.
I currently have a subscription to the RSS feed http://twitter.com/statuses/user_timeline/bitoclass.rss, which I use to keep a local archive of all my tweets (I never think it sensible to put any data, photos or anything else only in the cloud without a local backup).
So, does your API change only affect developers, or will my end-user RSS subscription cease to work on 5 March 2013 as well?
Thanks!
URLs of that style are pretty outdated now. RSS as an output format is considered part of the API, and that URL is what we consider a "version 0" URL. It will also cease functioning on March 5, 2013.
RSS and ATOM formats are relatively unused, and the OAuth requirement for all endpoints make utilizing them more difficult (or nearly impossible) for end-users. Tweets are better represented with the complete metadata available in the JSON responses available to you.
Thank you for your quick reply, much appreciated!
Pasting the above RSS URL into a 'subscribe to feed' box in an RSS reader couldn't be further from "nearly impossible" for me, and I'm an end-user. I'm not really sure what the tech jargon about "endpoints" etc means but I'm very disappointed you're turning off this functionality.
So just to be certain I'm understanding this, the consequence is that there will from March be no way for me to subscribe to my own tweets in an RSS reader in order to keep an offline copy of them? This is literally (really) the worst news I've ever heard about Twitter.
Perhaps I'm just being naive, but what exactly is the logic behind blocking RSS in this form? Several blogging applications rely on RSS in order to pull in Tweets (see sidebars), and RSS is the one solution that does this type of syndication in a relatively easy to manage manner.
If the goal is to prevent abuse or rate limit evasion, perhaps a delay on the published RSS feed would suffice? In the case of archival, as long as the original publication date is preserved, you could in theory not update feeds but once a day. This would prevent abuse whilst not cutting off the channel completely.
Tweets are more than the 140 character strings that make up their textual component. RSS is a great container for lightly referencing content but we prefer the full Tweet to be wherever a Tweet goes. We've chosen the JSON format to be our programmatic representation of tweets.
You may prefer that, but as one of your end-users I prefer to have the option of subscribing to my tweets via RSS. When companies start putting what they prefer ahead of what their users prefer, it's the beginning of the end. Good luck with that.
Why is there no API call to grab a user's tweets & retweets without requiring an OAUTH/User Context, if the requested user has a public account? I saw statuses/user_timeline.json, and it requires a user context.
RSS feeds performed this function already. You've said that the RSS feeds are going away. So, I won't be able to do this anymore, and the only replacement is the API call that returns JSON.
I looked at and saw that it requires user context. But wait, isn't this a downgrade of accessibility? And will the Application context be provided soon enough so that I can get some minor consolation, versus requiring to OAuth my own single-user apps?
As mentioned above, we are working on a variation of authentication for API v1.1 that will allow for an application-only context not requiring a user context. When that feature is made available, methods not requiring a user context will have their documentation updated to indicate as such (until then, all methods require it). If access-without-a-user-context is important to you, you may want to wait to implement 1.1 until those features are available.
Will old features (EG: RSS) that have access-without-a-user-context be postponed removal until a time after Application-only context's implementation?
On the whole these changes look pretty good. Thanks for the much friendlier wording the blog posts!
I'm a bit worried about the limitations in loading social graph data. It looks like large accounts will have to load significantly more slowly than before. The rate limitations on users/lookup also means caching will be more important that ever before for loading smaller accounts. Often users/lookup was the bottleneck in loading small accounts.
On the whole though, there are no show stoppers. The rate limiting on search is pretty exciting - making it per account based removes a lot of the uncertainty around per IP limits.
I do have one question at this point. In the Developer Rules of the Road it states "If your application will eventually need more than 1 million user tokens, or you expect your embedded Tweets and embedded timelines to exceed 10 million daily impressions, you will need to talk to us directly about your access to the Twitter API as you may be subject to additional terms." How do we do this - who should we contact? Is there any information available about the additional terms?
Thanks Taylor.
We're working to make that communication process easy to initiate. Stay tuned.
Hello,
To construct a user's timeline up to the max 3,200 available using "statuses/user_timeline" 16 calls would be needed. (200 max returned each call) To my understanding this is now limited to 15 calls per 15 minutes. Does this make it impossible to pull down all available Tweets when first encountering a user without hitting the rate limit? What is the guidance here?
Thanks.
Does this mean the end of the IP address-based whitelisting that was previously grandfathered? I'm referring to the 20K/hour rate that older apps might enjoy.
I think its sad, and a crazy business decision, that Twitter is removing the ability of an inexperienced developer to take some javascript and css and incorporate their tweets into a blog or website as they please. Now they must either use the cards or a much higher level of development to effectively recreate the cards because the display guidelines are so strict. This was a big part of the growth of twitter and leaves a gap for another service to fill. Instagram has to be most likely, all they need to do is create a text-only message option and will take a big slice from Twitter
Most of the api limits are not unreasonable but there are now so many rules and vague guidelines, and more will follow in the future. eg
What does "No other social or 3rd party actions similar to Follow, Reply, Retweet and Favorite may be attached to a Tweet." really mean?
"Tweets that are grouped together into a timeline must not be rendered with non-Twitter content." The BBC will often put tweets in a timeline of mixed content when covering sporting events. They will presumably have to remove this now and reduce the spread of tweets.
If you need to make money, just put an advert on the website, everyone else does that
Looking at the rate limits for lists, is it true that lists/statuses is in the 15 calls / 15 minute bucket?
I mean that way, you can only fetch tweets for one list per minute on average?
That would severely cripple the lists feature, wouldn't it? Just imagine you've got more than 15 lists ...
It is true. We'd love more feedback on how higher limits would be made use of. There are a lot of use cases for lists, and honestly I wish the rate limiting scenario for them were able to be proportionate to the number of lists a given user were subscribed to or owned. The likelihood of that level of sophistication at scale is low right now, but welcome feedback on this.
Gravity is automatically fetching lists for each user you're looking up, for example.
That makes lists a lot more interesting.
When you immediately see the lists that a user has created or is following, you get a better understanding of that user.
These kind of features won't work with the new limits anymore, I think.
By the way, is the API 1.1 live already? Can you test it out right away?
I find the rate limiting of lists/statuses to be too low, or not designed right. Each list acts like a single timeline, and each one of them should receive 15 calls / window. I can see how this would be very hard to implement, but it would really help a lot of people.
For example, performing social analysis to obtain someone's interests by looking at each list would be extremely difficult to perform with this 15 calls / window limitation. For example, if we were to fetch latest 800 tweets for each list, it would cost 4 API calls per list. If a user have or subscribed to more than 3 lists (which is very common for lists users), we will have to delay the process of analysis before getting back to the user with a result, which severely degrades the user experience. Sure, we can fetch all members (lists/members rate limit is too low by the way. lists/memberIds would help a lot.) and simulate the result of the call by using various other APIs, but I thought the lists APIs were implemented to ease the pain of developers implementing lists-like features manually, and to endorse the server side maintenance of alternate timelines.
I can see how this could affect scalability severely, but I really think it is important to re-think this part of the rate limit design.
Thank you,
Hi @aki,
I definitely recognize the value in raising the limits for lists/statuses and hope to have good things to announce about its rate limit in 1.1 soon.
Thanks a lot for the quick response! I'm glad to know that you guys are aware of the issue :)
Is there a best practice way to use JSONP to make requests to API v1.1?
By playing on the command line I see adding &callback= does generate a JSONP request, but I don't want to put all my oauth logic in Javascript (for obvious security reasons)
Also how do I send oauth headers to a JSONP request?
Help please!
JSON-P is going to be difficult in API v1.1. There may be a day in the future when we support OAuth 2, which would make JSON-P an easier prospect again.
OAuth 1.0a cannot be performed securely in client-side Javascript. Server-side integrations are pretty much a must to meet the auth requirements of API v1.1. If you're able to securely pipeline and prepare signed requests to a secure server-side signing endpoint that cannot be exploited by others, you could perform JSON-P that way -- but I would not recommend it.
You should be able to just redirect to the signed API requests from the JSON-P request to your own server.
That way the server only does the signing (shouldn't take too much resources) and the web browser will issue the real API request to Twitter ...
Now that the @Anywhere API is being removed in December, there doesn't seem to be a method by which a developer could implement an entirely front end sign in mechanism. With @Anywhere I could simply include the library then get an authenticated user via:
Now, I'm required to perform server side oAuth for this simple procedure? Is there no client-side-only alternative?
There are no client-side only solutions besides what we offer in our Twitter for Websites family of products at this time. If you want to use the API itself, you'll need to use server-side OAuth. If you want to engage with the platform client-side, you'll need to use the Tweet Button, Follow Button, Embeddable Timelines, Embeddable Tweets, the tweet, follow, favorite, and retweet web intents, and their accompanying javascript events. We're still working on this client-side family of products and there may be further enhancements to the API's auth model in the future that would make client-side development more palatable.
RT @billbrown Does this mean the end of the IP address-based whitelisting that was previously grandfathered? I'm referring to the 20K/hour rate that older apps might enjoy.
I would be interested too on how current whitelisted IPs are affected with API v1.1
The rate limiting model of API v1.1 does not factor IP addresses into the equation. As such, IP address whitelisting will have no effect on rate limiting in 1.1. Some applications will have to change their model to match the more dynamic world of 15 minute windows and per-method rate limiting. Some things will take longer. Some things can be done more rapidly. Some formal avenues to request elevated access will be provided soon.
So if I understand you correctly, whitelisting will disappear in 1.1? If so, I would love to know how to get elevated access for certain API calls.
For my application (and I assume numerous others), whitelisting/elevated access rate is crucial, as the API endpoints and rate limits are tailored towards certain use cases like clients or Tweet analysis tools. And if a tool does something else (in my case network analysis/management), the API endpoints and rate limits simply do not fit.
For example, if a network analysis needs the following lists of all the people a user follows and you would like to generate an answer in the minute range, it would work
- with whitelisting for all but the biggest accounts (up to 19,798 following)
- with 1.0 limits only for quite small accounts (up to 345 following)
- with 1.1 limits for almost no account (up to 14 following)
A couple of things that could help here:
- The obvious one: Whitelisting as before.
- Whitelisting the friends/ids and followers/ids calls only.
- An endpoint allowing to batch retrieval of following/follower id lists (like users/lookup for users). It would be OK if this one would only work for cursor=-1.
- Making the time window a lot bigger for friends/ids and followers/ids. For a network analysis, it is often fine to cache data for a week or so, as following lists do not change much over time. But to collect the data takes way too long with a 15 minute window. If an app could make a week's worth of calls for a user in a minute, it could then cache it and do analysis based on that data for the next week.
API v1.1 responds to unauthenticated requests with
400 Bad Requestinstead of401 Unauthorized.Compare:
curl -is https://api.twitter.com/1/statuses/mentions_timeline.jsonvs.
curl -is https://api.twitter.com/1.1/statuses/mentions_timeline.jsonIt would be helpful if you clarified this in https://dev.twitter.com/docs/error-codes-responses.