Site Streams

Updated on Mon, 2012-01-09 15:33

Currently in beta.

04/19/2011

Overview

Site Streams, a new feature on the Streaming API, is now available for beta testing. Site Streams allows services, such as web sites or mobile push services, to receive real-time updates for a large number of users without any of the hassles of managing REST API rate limits. Events can be streamed for all users that have OAuthed your application. The following events are streamed without rate limits: Home Timelines, Mentions Timelines, User Timelines, Direct Messages, Mentions, Follows, Favorites, Tweets, Retweets, Profile changes, and List changes.

Important Items

  • During the beta testing period, sites are encouraged to first pull and cross-check test data from both Site Streams and the REST API to develop confidence in correctness and robustness. Once your Site Streams client implementation is deemed sufficiently stable, the streaming data could, depending on your risk-tolerance, become the primary production data source. Once this feature graduates to production, Site Streams applications must then only use the REST API for data that is not available through Site Streams or use the REST API as a fall-back data source.
  • Be meticulous about privacy implications: As with all authenticated Twitter APIs, protected account data may be delivered on Site Streams. Do not display Site Streams to users who do not present the corresponding authentication token unless the public data is extracted in future-proof manner:
    • Publish only data that can be verified as public.
    • Do not attempt to remove data that appears to be protected, as new message types or other changes may escape a removal filter, resulting in an inadvertent privacy leak.
    • Do not expose or otherwise draw attention to status deletions or unfollows to users, other than to update displays.
  • Desktop applications must use the User Streams endpoint. Services that are opening streams for more than a few dozen users must use the Site Streams endpoint.
  • Read the User Streams and User Streams Suggestions documentation. Site Streams are very similar to User Streams — other than multiplexing many User Streams over a connection, almost all of the guidelines outlined also apply to Site Streams.
  • Consult the Streaming API documentation, and double-check the pre-launch checklist.
  • Ensure that your implementation respects the limitations below, as well as all Streaming API policies, Verify that your client falls back to the REST API if and only if Site Streams are unavailable.
  • Do not draw attention to or otherwise expose status deletion or unfollow events, other than to update displays and keep your data stores synchronized. Highlighting these events is against the Twitter TOS and may cause immediate suspension on Site Streams, and may even cause your access token to be permanently revoked across all Twitter APIs.
  • Twitter API developers must follow the very low-volume Twitter API Announce mailing list. This list is updated only when changes to the Twitter API are released or are pending release.
  • Site Streams developers should follow @sitestreams account for changes and operational updates during the beta test period.
  • Please provide feedback! Feedback, bug reports, and feature requests should be directed to Discussions.

Limitations

The following limitations must be respected during the beta period. These limitations may be changed with little advance notice. We intend to increase or remove these various limitations as we move from beta test into full production:

  • Limit the follow count to 100 users per stream while connecting. Use Control Streams to add up to 1,000 users per connection. See Control Streams for Site Streams.
  • Open no more than 25 new connections per second and exponentially back-off on errors.
  • If you intend to open more than roughly a thousand connections, you should coordinate your testing and launch with the Twitter Platform team.
  • Manage your connections with Control Streams for Site Streams.

API Overview

  • Site Streams can be accessed during the beta period at https://sitestream.twitter.com/2b/site.json. The endpoint and host will change as we move to full-production, so be prepared to change both portions of the URL.
  • Publicly display only data that can be verified as public. All other data must require matching authentication.
  • Unlike User Streams, track, loc, and other filters are not supported.
  • You must authenticate with Site Streams using OAuth as the owner of an application. For example if I, userid 5 create an application called SiteFoo, I need to authenticate using the application SiteFoo, with an access token given to userid 5.
  • To specify a list of users to include in the stream, use the follow parameter as with the filtered streams, e.g. https://sitestream.twitter.com/2b/site.json?follow=1,2,3,4,5. The server will check these userids against your stored access tokens, and return a friends message for each user that has a valid access token for your app.
  • Currently the count parameter is not supported. This feature would be very useful for covering reconnection gaps, but support for this feature in currently uncertain.
  • Only JSON is supported for Site Streams.
  • Each JSON message is wrapped in an envelope that indicates the user it is intended for, e.g. 
    1.   {\"for_user\":\"1888\",\"message\":{\"friends\":[]}}
    2.   {\"for_user\":\"9160152\",\"message\":{\"friends\":[]}}
    3.  
  • If a message should be routed to multiple users, you will receive multiple messages, each with a different for_user element.
  • The with parameter controls the types of messages received. The default for Site Streams is with=user, which sends only event about the user. The value with=followings also sends events about the user's followings -- basically the user's home timeline.
  • Upon connection, you will receive a new message type keyed on "control" and containing a "control_uri". These are part of Control Streams for Site Streams, a feature allowing for modifying the stream's criteria without reconnecting. See that documentation for more details. 
    1. {"control":
    2.   {
    3.     "control_uri": "/2b/site/c/01_225167_334389048B872A533002B34D73F8C29FD09EFC50"
    4.   }
    5. }

Implementation Suggestions

Connection Management

As new users authorize your application, you will need to add them to your streams. Our suggestion is to open a series of small streams for recently added users. Once you have a sufficient number of small streams open, create a larger stream that contains all recent users, then disconnect the small streams. Overall, minimize the total number of connections and minimize the number of new connections formed. Typically, your client should be opening connections only a few times per minute. Follow the connection suggestions in User Stream Connections and User Streams Algorithms.

Clients should not attempt to open all connections at once. For example, on client service restart, or Streaming API restart, the new connection rate must be controlled. The aggregate number of new connections from a service must be respected to prevent overloading the beta cluster. When partitioning your clients across multiple hosts, verify that the total number of new connections per second is still less than 25.

Be sure and read about Control Streams for Site Streams for more information on advanced stream management with control streams.

Asynchronous Processing

Site Streams may deliver hundreds of messages per second to a client, and each stream may consume significant (> 1 Mbit/sec) bandwidth. Your processing of tweets should be asynchronous, with appropriate buffers in place to handle spikes of 3x normal throughput. Note that slow reading clients are automatically terminated.

Apply for Access

  1. Read all documents: Site Streams, User Streams, User Streams Suggestions and Streaming API. Understand all connection heuristics and policies.
  2. Sign up for the required communications channels: @twitterapi, @sitestreams. Post feedback in Discussions.
  3. Implement a single-user scenario using User Streams develop all associated functionality for consuming user-centric data and events, including appropriate queueing, processing, and REST fallback support.
  4. Apply for beta access to the Site Streams API by filling out the form at http://bit.ly/sitestreams.

The needs of this beta program evolve and applicants will be selected depending on availability & current requirements. Developers that are accepted into the beta will receive an email with further instructions — waiting times vary and due to constraints, not every applicant will be provided beta access.

Site Streams does not support any of the search/track features of the User Streams, so if your application requires these capabilities, Site Streams may not be the right fit. Site Streams access does not provide a greater percentage of the firehose than the self-serve options available today.

There is currently no estimated date for Site Streams to exit beta.