Summary: This issue effects Search and REST API methods. When it happens you may see a truncated response with a Content-Length value higher than the actual response you received. Our engineers are investigating.
Search API sometimes returns malformed i.e. "cut off in the middle" JSON response.
Example scenario:
Request curl -XGET 'http://search.twitter.com/search.json?q=pic.twitter.com&lang=en&result_type=recent&rpp=100&include_entities=true' issued periodically every 30 secs failed (unparsable due to the incompleteness, example response is attached at the end) 10x from the total of 352 trials. Volume of failures increases proportionally with request frequency.
Our system first logged this problem at 2012-07-02 19:30 UTC. Since then it reappeared occasionally but in the last few days the volume of failed requests has raised drastically.
This is not network or hardware issue. We tested it on two different machines connected to two independent networks (DCs in two different countries (EU)).
Example of incomplete response:
{"completed_in":0.787,"max_id":227107620979351552,"max_id_str":"227107620979351552","next_page":"?page=2&max_id=227107620979351552&q=pic.twitter.com&lang=en&rpp=100&include_entities=1&result_type=recent","page":1,"query":"pic.twitter.com","refresh_url":"?since_id=227107620979351552&q=pic.twitter.com&lang=en&result_type=recent&include_entities=1","results":[{"created_at":"Sun, 22 Jul 2012 18:27:31 +0000","entities":{"hashtags":[],"urls":[],"user_mentions":[],"media":[{"id":227107621033893888,"id_str":"227107621033893888","indices":[31,51],"media_url":"http:\/\/p.twimg.com\/AybZLZWCYAA07fJ.jpg","media_url_https":"https:\/\/p.twimg.com\/AybZLZWCYAA07fJ.jpg","url":"http:\/\/t.co\/feM5ucuT","display_url":"pic.twitter.com\/feM5ucuT","expanded_url":"http:\/\/twitter.com\/Rachaelwbu\/status\/227107620979351552\/photo\/1","type":"photo","sizes":{"thumb":{"w":150,"h":150,"resize":"crop"},"large":{"w":487,"h":720,"resize":"fit"},"orig":{"w":487,"h":720,"resize":"fit"},"medium":{"w":487,"h":720,"resize":"fit"},"small":{"w":340,"h":503,"resize":"fit"}}}]},"from_user":"Rachaelwbu","from_user_id":528688411,"from_user_id_str":"528688411","from_user_name":"~\u2665\u263aStewie griffin\u263a\u2665~","geo":null,"id":227107620979351552,"id_str":"227107620979351552","iso_language_code":"en","metadata":{"result_type":"recent"},"profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/2412536418\/Hillingdon-20120719-00794_normal.jpg","profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/2412536418\/Hillingdon-20120719-00794_normal.jpg","source":"<a href="http:\/\/twitter.com\/">web<\/a>","text":"haha omg i love these pictures http:\/\/t.co\/feM5ucuT","to_user":null,"to_user_id":0,"to_user_id_str":"0","to_user_name":null},{"created_at":"Sun, 22 Jul 2012 18:27:30 +0000","entities":{"hashtags":[],"urls":[],"user_mentions":[{"screen_name":"DannyTaughtYou","name":"Danny \u2654","id":580390398,"id_str":"580390398","indices":[3,18]}],"media":[{"id":227100476280274944,"id_str":"227100476280274944","indices":[47,67],"media_url":"http:\/\/p.twimg.com\/AybSrhECAAAfhnw.jpg","media_url_https":"https:\/\/p.twimg.com\/AybSrhECAAAfhnw.jpg","url":"http:\/\/t.co\/Xqlf2mrX","display_url":"pic.twitter.com\/Xqlf2mrX","expanded_url":"http:\/\/twitter.com\/DannyTaughtYou\/status\/227100476267692032\/photo\/1","type":"photo","sizes":{"medium":{"w":600,"h":600,"resize":"fit"},"thumb":{"w":150,"h":150,"resize":"crop"},"
cURL output:
curl: (18) transfer closed with 147797 bytes remaining to read
Updates
- Currently investigating reports of this problem on the Search API and other REST-based APIs. Thanks for the report.
- Update 2012-07-24: We're still unable to reproduce this. If you're experiencing this issue, please capture a connection attempt reflecting this problem, including the full HTTP request & response cycle and post it in the forums. Thanks!
- Update 2012-08-03: Thank you all for the continued reports of this. Engineers are investigating and working to resolution. No ETA at this time for a fix.