Python Infrastructure Status

Python Infrastructure

Update - We have increased the rolling brownouts to the first 20 minutes of every hour, plus the first 20 minutes after each half-hour (:00-:20 and :30-:50).
Apr 6, 11:43 UTC

Update - We have increased the rolling brownouts to the first 15 minutes of every hour, plus the first 15 minutes after each half-hour (:00-:15 and :30-:45).
Apr 3, 11:45 UTC

Update - We have increased the rolling brownouts to the first 15 minutes of every hour, plus the first 10 minutes after each half-hour (:00-:15 and :30-:40).
Mar 31, 16:00 UTC

Update - We have increased the rolling brownouts to the first 15 minutes of every hour, plus the first 5 minutes after each half-hour (:00-:15 and :30-:35).
Mar 28, 16:33 UTC

Update - We are currently enforcing TLSv1.2 during the first 15 minutes of each hour. You can follow the planning and discussion of this maintenance at https://github.com/pypa/warehouse/issues/3411
Mar 27, 16:46 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 22, 18:59 UTC

Scheduled - In preparation for our CDN provider deprecating TLSv1.0 and TLSv1.1 protocols, we have begun rolling brownouts for these protocols for the first ten (10) minutes of each hour.

During that window, clients accessing pypi.python.org with clients that do not support TLSv1.2 will receive an HTTP 403 with the error message "This is a brown out of TLSv1 support. TLSv1 support is going away soon, upgrade to a TLSv1.2+ capable client.".

For users of pip on MacOS/OS X, upgrading to the latest pip should resolve the issue. pip users on other platforms should upgrade their OpenSSL to a version which supports TLSv1.2.

If you have trouble with getting your PyPI client working, file an issue at https://github.com/pypa/packaging-problems/issues.

python.org Operational

pypi.python.org Operational

pypi.org Operational

docs.python.org Operational

hg.python.org Operational

bugs.python.org Operational

wiki.python.org Operational

mail.python.org Operational

Mailing Lists and Archives Operational

Message Handling Services Operational

pypy.org Operational

speed.pypy.org Operational

psfmember.org Operational

Content Delivery Network Operational

Fastly US East (JFK) Operational

Fastly Asia/Pacific (HK) Operational

Fastly US East (IAD) Operational

Fastly US East (ATL) Operational

Fastly US East (MIA) Operational

Fastly US Central (DEN) Operational

Fastly US Central (DFW) Operational

Fastly US Central (ORD) Operational

Fastly US West (LAX) Operational

Fastly US West (SEA) Operational

Fastly US West (SJC) Operational

Fastly Europe (FRA) Operational

Fastly Europe (AMS) Operational

Fastly Europe (LCY) Operational

Fastly Europe (LHR) Operational

Fastly Asia/Pacific (SYD) Operational

Fastly Asia/Pacific (SIN) Operational

Fastly Asia/Pacific (TYO) Operational

Fastly Asia/Pacific (NZ) Operational

us.pycon.org Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

System Metrics Month Week Day

PyPI Errors

Fetching

Warehouse Errors

Fetching

PyPI CDN Miss Latency

Fetching

Warehouse CDN Miss Latency

Fetching

Past Incidents

Apr 5, 2018

No incidents reported.

Apr 4, 2018

Hosting issue

An administrative issue has occurred which required staff intervention for bugs.python.org. The hosting company has turned the infrastructure service back on.
Apr 4, 13:55 UTC

Apr 2, 2018

No incidents reported.

Apr 1, 2018

No incidents reported.

Mar 30, 2018

No incidents reported.

Mar 29, 2018

No incidents reported.

Mar 28, 2018

Denial Of Service

Resolved - We're weathering the incoming floods much more gracefully now and a full mitigation will be in place as soon as DNS propagation completes. Resolving for now!
Mar 28, 15:53 UTC

Monitoring - We've tuned the backend that was bearing the brunt of the attack to attempt to mitigate the incoming floods of bogus traffic and are monitoring the efficacy of our measures.
Mar 28, 14:43 UTC

Identified - We've configured the redirect from python.org => www.python.org to take affect at the CDN edge rather than our backend servers. This should reduce the impact of trivial DoS floods, but will take up to 24 hours to propagate. We're monitoring and tuning our backends to try to reduce the impact in the interim.
Mar 28, 14:24 UTC

Investigating - We are working to mitigate a Denial of Service attack against our infrastructure, sites hosted on the target are currently experiencing intermittent outages.
Mar 28, 14:02 UTC

Mar 26, 2018

No incidents reported.

Mar 25, 2018

https://pypi.org load testing underway

Resolved - This testing period has concluded. No new issues were reported!
Mar 25, 22:13 UTC

Identified - Identified - A fraction of traffic for `https://pypi.python.org/simple` is being redirected to `https://pypi.org/simple` as a load test in preparation to begin final rollout of the new codebase.

This will mainly affect `pip` and other Python packaging clients which do not support TLSv1.2, or cannot follow an HTTP 302 redirect from pypi.python.org to pypi.org.

During this period, if you notice any issues with your results from `/simple` or experience problems with the redirect, please notify the PyPI team at https://github.com/pypa/warehouse/issues
Mar 25, 18:53 UTC

Mar 24, 2018

No incidents reported.

Mar 23, 2018

https://pypi.org load testing underway

Resolved - We've completed this testing for the day.

One report of an OS X/MacOS user was found, upgrading to pip 9.0.3+ resolved their issue.

Otherwise we had two users who were behind TLS Man-In-The-Middle terminators or HTTP Proxies that reported issues with their configuration.
Mar 23, 22:58 UTC

Identified - A fraction of traffic for `https://pypi.python.org/simple` is being redirected to `https://pypi.org/simple` as a load test in preparation to begin final rollout of the new codebase.

This will mainly affect `pip` and other Python packaging clients which do not support TLSv1.2, or cannot follow an HTTP 302 redirect from pypi.python.org to pypi.org.

During this period, if you notice any issues with your results from `/simple` or experience problems with the redirect, please notify the PyPI team at https://github.com/pypa/warehouse/issues
Mar 23, 20:57 UTC

PyPI Database Maintenance

Completed - We're working to resolve one minor issue this created, but impact is minimal. Maintenance complete!
Mar 23, 11:22 UTC

Verifying - Our maintenance is complete! We're going to monitor and verify before completing this scheduled maintenance window.
Mar 23, 10:36 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 23, 10:00 UTC

Scheduled - In preparation for our upcoming rollout of the all new https://pypi.org, we'll be taking down both https://pypi.org and https://pypi.python.org temporarily to perform some database maintenance. pip installs will continue to work during this time, but new release uploads and XMLRPC cache misses will fail.
Mar 23, 01:56 UTC

Subscribe to Incident