Python Infrastructure Status

Python Infrastructure

Update - We are currently enforcing TLSv1.2 during the first 15 minutes of each hour. You can follow the planning and discussion of this maintenance at https://github.com/pypa/warehouse/issues/3411
Mar 27, 16:46 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 22, 18:59 UTC

Scheduled - In preparation for our CDN provider deprecating TLSv1.0 and TLSv1.1 protocols, we have begun rolling brownouts for these protocols for the first ten (10) minutes of each hour.

During that window, clients accessing pypi.python.org with clients that do not support TLSv1.2 will receive an HTTP 403 with the error message "This is a brown out of TLSv1 support. TLSv1 support is going away soon, upgrade to a TLSv1.2+ capable client.".

For users of pip on MacOS/OS X, upgrading to the latest pip should resolve the issue. pip users on other platforms should upgrade their OpenSSL to a version which supports TLSv1.2.

If you have trouble with getting your PyPI client working, file an issue at https://github.com/pypa/packaging-problems/issues.

python.org Operational

pypi.python.org Operational

pypi.org Operational

docs.python.org Operational

hg.python.org Operational

bugs.python.org Operational

wiki.python.org Operational

mail.python.org Operational

Mailing Lists and Archives Operational

Message Handling Services Operational

pypy.org Operational

speed.pypy.org Operational

psfmember.org Operational

Content Delivery Network Operational

Fastly US East (JFK) Operational

Fastly Asia/Pacific (HK) Operational

Fastly US East (IAD) Operational

Fastly US East (ATL) Operational

Fastly US East (MIA) Operational

Fastly US Central (DEN) Operational

Fastly US Central (DFW) Operational

Fastly US Central (ORD) Operational

Fastly US West (LAX) Operational

Fastly US West (SEA) Operational

Fastly US West (SJC) Operational

Fastly Europe (FRA) Operational

Fastly Europe (AMS) Operational

Fastly Europe (LCY) Operational

Fastly Europe (LHR) Operational

Fastly Asia/Pacific (SYD) Operational

Fastly Asia/Pacific (SIN) Operational

Fastly Asia/Pacific (TYO) Operational

Fastly Asia/Pacific (NZ) Operational

us.pycon.org Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

System Metrics Month Week Day

PyPI Errors

Fetching

Warehouse Errors

Fetching

PyPI CDN Miss Latency

Fetching

Warehouse CDN Miss Latency

Fetching

Past Incidents

Mar 26, 2018

No incidents reported.

Mar 25, 2018

https://pypi.org load testing underway

Resolved - This testing period has concluded. No new issues were reported!
Mar 25, 22:13 UTC

Identified - Identified - A fraction of traffic for `https://pypi.python.org/simple` is being redirected to `https://pypi.org/simple` as a load test in preparation to begin final rollout of the new codebase.

This will mainly affect `pip` and other Python packaging clients which do not support TLSv1.2, or cannot follow an HTTP 302 redirect from pypi.python.org to pypi.org.

During this period, if you notice any issues with your results from `/simple` or experience problems with the redirect, please notify the PyPI team at https://github.com/pypa/warehouse/issues
Mar 25, 18:53 UTC

Mar 24, 2018

No incidents reported.

Mar 23, 2018

https://pypi.org load testing underway

Resolved - We've completed this testing for the day.

One report of an OS X/MacOS user was found, upgrading to pip 9.0.3+ resolved their issue.

Otherwise we had two users who were behind TLS Man-In-The-Middle terminators or HTTP Proxies that reported issues with their configuration.
Mar 23, 22:58 UTC

Identified - A fraction of traffic for `https://pypi.python.org/simple` is being redirected to `https://pypi.org/simple` as a load test in preparation to begin final rollout of the new codebase.

This will mainly affect `pip` and other Python packaging clients which do not support TLSv1.2, or cannot follow an HTTP 302 redirect from pypi.python.org to pypi.org.

During this period, if you notice any issues with your results from `/simple` or experience problems with the redirect, please notify the PyPI team at https://github.com/pypa/warehouse/issues
Mar 23, 20:57 UTC

PyPI Database Maintenance

Completed - We're working to resolve one minor issue this created, but impact is minimal. Maintenance complete!
Mar 23, 11:22 UTC

Verifying - Our maintenance is complete! We're going to monitor and verify before completing this scheduled maintenance window.
Mar 23, 10:36 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 23, 10:00 UTC

Scheduled - In preparation for our upcoming rollout of the all new https://pypi.org, we'll be taking down both https://pypi.org and https://pypi.python.org temporarily to perform some database maintenance. pip installs will continue to work during this time, but new release uploads and XMLRPC cache misses will fail.
Mar 23, 01:56 UTC

Mar 22, 2018

pypi.org email send delays

Resolved - This incident has been resolved.
Mar 22, 03:10 UTC

Monitoring - We've scaled our backend worker pool and are monitoring to ensure timely delivery.
Mar 22, 02:55 UTC

Identified - pypi.org is delayed in sending emails due to overwhelmed backend workers. we're working on a fix and will update when we're caught up.
Mar 22, 01:07 UTC

Mar 21, 2018

No incidents reported.

Mar 20, 2018

No incidents reported.

Mar 19, 2018

No incidents reported.

Mar 18, 2018

No incidents reported.

Mar 17, 2018

https://pypi.org load testing underway

Resolved - Complete!
Mar 17, 00:29 UTC

Investigating - A fraction of traffic for XMLRPC requests to `https://pypi.python.org/pypi` is being transparently directed to `https://pypi.org/pypi` as a load test in preparation to begin final rollout of the new codebase.

During this period, if you notice any issues with your results for XMLRPC requests to `/pypi`, please notify the PyPI team at https://github.com/pypa/warehouse/issues
Mar 16, 23:10 UTC

Mar 16, 2018

https://pypi.org load testing underway

Resolved - We've concluded load splitting for the (US Eastern) morning. Things are stil looking great. Some new reports are in regarding TLS configuration, but our fix relating to old versions of pip was proven successful. We'll be assessing mitigation paths for the TLS issues.

If you'd like to opt in to the new changes, please set your clients to send a value for the `Force-Warehouse-Redirect` header to experience the new PyPI on /simple
Mar 16, 15:57 UTC

Identified - The issue has been identified and a fix is being implemented.
Mar 16, 14:44 UTC

Investigating - A fraction of traffic for `https://pypi.python.org/simple` is being redirected to `https://pypi.org/simple` as a load test in preparation to begin final rollout of the new codebase.

During this period, if you notice any issues with your results from `/simple` or experience problems with the redirect, please notify the PyPI team at https://github.com/pypa/warehouse/issues
Mar 16, 14:44 UTC

https://pypi.org load testing underway

Resolved - We've concluded load splitting for the evening. We squashed a few bugs in the new codebase and infrastructure, but things are looking great. Some clients, notably Artifactory and ancient versions of pip didn't enjoy the new redirect. We'll be assessing

If you'd like to opt in to the new changes, please set your clients to send a value for the `Force-Warehouse-Redirect` header to experience the new PyPI on /simple.
Mar 16, 00:10 UTC

Monitoring - A subset of traffic for `https://pypi.python.org/simple` is being redirected to `https://pypi.org/simple` as a load test in preparation to begin final rollout of the new codebase.

During this period, if you notice any issues with your results from `/simple` or experience problems with the redirect, please notify the PyPI team at https://github.com/pypa/warehouse/issues
Mar 15, 21:34 UTC

Mar 14, 2018

No incidents reported.

Mar 13, 2018

No incidents reported.

Subscribe to Incident