Python Infrastructure Status

Python Infrastructure

All Systems Operational

python.org Operational

pypi.python.org Operational

pypi.org Operational

docs.python.org Operational

hg.python.org Operational

bugs.python.org Operational

wiki.python.org Operational

mail.python.org Operational

Mailing Lists and Archives Operational

Message Handling Services Operational

pypy.org Operational

speed.pypy.org Operational

psfmember.org Operational

Content Delivery Network Operational

Fastly US East (JFK) Operational

Fastly Asia/Pacific (HK) Operational

Fastly US East (IAD) Operational

Fastly US East (ATL) Operational

Fastly US East (MIA) Operational

Fastly US Central (DEN) Operational

Fastly US Central (DFW) Operational

Fastly US Central (ORD) Operational

Fastly US West (LAX) Operational

Fastly US West (SEA) Operational

Fastly US West (SJC) Operational

Fastly Europe (FRA) Operational

Fastly Europe (AMS) Operational

Fastly Europe (LCY) Operational

Fastly Europe (LHR) Operational

Fastly Asia/Pacific (SYD) Operational

Fastly Asia/Pacific (SIN) Operational

Fastly Asia/Pacific (TYO) Operational

Fastly Asia/Pacific (NZ) Operational

us.pycon.org Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

System Metrics Month Week Day

PyPI Errors

Fetching

Warehouse Errors

Fetching

PyPI CDN Miss Latency

Fetching

Warehouse CDN Miss Latency

Fetching

Past Incidents

Apr 17, 2018

No incidents reported today.

Apr 16, 2018

Next Generation PyPI Rollout

Postmortem - Read details
Apr 16, 19:21 UTC

Resolved - We have completed our rollout! If you have any issues, please reach out at https://github.com/pypa/warehouse/issues or #pypa-dev on the freenode IRC network.
Apr 16, 19:19 UTC

Update - We're preparing an Incident Report on the files.pythonhosted.org incident now. All services are stable, and we'll be completing transition of XMLRPC later this afternoon.
Apr 16, 17:31 UTC

Update - We've completed the migration to the intended state for all services. We're continuing to monitor our own telemetry and all support channels.
Apr 16, 17:06 UTC

Update - We have gotten file hosting back into a stable place and are currently working towards a full resolution.
Apr 16, 16:26 UTC

Update - We're continuing to work on resolving the files.pythonhosted.org package service.
Apr 16, 16:05 UTC

Update - Now investigating widespread 404s for files.pythonhosted.org downloads.
Apr 16, 15:45 UTC

Update - We've rolled out a fix for the redirect issue on files.pythonhosted.org for the new PyPI and are monitoring.
Apr 16, 15:35 UTC

Update - files.pythonhosted.org is currently under maintenance to resolve redirect loops.
Apr 16, 15:29 UTC

Update - We are currently investigating redirect loops for some package installs.
Apr 16, 15:22 UTC

Update - pypi.org is live!
Apr 16, 15:04 UTC

Monitoring - Over the next few hours we'll be cutting over to the new https://pypi.org for all traffic to https://pypi.python.org!

Most requests will be redirected or rerouted correctly at our CDN edge; users should not need to perform any specific changes.

The old service will be available at https://legacy.pypi.org for users who find immediate issues they cannot address. We plan to leave this endpoint available until the end of the month when the previous generation PyPI's backend services are spun down.

If you experience any issues, please note them at https://github.com/pypa/warehouse/issues or join #pypa-dev on the freenode IRC network.
Apr 16, 14:20 UTC

Apr 15, 2018

pypi.org failover testing

Resolved - work is complete!
Apr 15, 16:55 UTC

Identified - We are currently testing failover for pypi.org to static mirrors. this should not impact users of the simple index such as `pip`, or json document api such as `pipenv` but may make pypi.org unavailable for users of the web ui. Uploads will not be impacted.
Apr 15, 16:20 UTC

Apr 14, 2018

pypi.org load testing underway

Resolved - Load testing complete for the day!
Apr 14, 20:57 UTC

Update - A fraction of traffic for `https://pypi.python.org/simple` is also being redirected to `https://pypi.org/simple` as a load test in preparation to begin final rollout of the new codebase.

This will mainly affect `pip` and other Python packaging clients that cannot follow an HTTP 302 redirect from pypi.python.org to pypi.org or are incompatible with the `/simple` resources provided by the new PyPI.

During this period, if you notice any issues with your results from `/simple` or experience problems with the redirect, please notify the PyPI team at https://github.com/pypa/warehouse/issues
Apr 14, 17:03 UTC

Investigating - A fraction of traffic for PyPI's XMLRPC API is being transparently served by to `https://pypi.org` as a load test in preparation to begin final rollout of the new codebase.

During this period, if you notice any issues with your results from the XMLRPC API, please notify the PyPI team at https://github.com/pypa/warehouse/issues
Apr 14, 15:57 UTC

Apr 13, 2018

pypi.org load testing underway

Resolved - load testing complete.
Apr 13, 10:59 UTC

Investigating - Identified - A fraction of traffic for PyPI's XMLRPC API is being transparently served by to `https://pypi.org` as a load test in preparation to begin final rollout of the new codebase.

During this period, if you notice any issues with your results from the XMLRPC API, please notify the PyPI team at https://github.com/pypa/warehouse/issues
Apr 13, 10:33 UTC

Apr 12, 2018

pypi.org load testing underway

Resolved - Load testing pypi.org is complete for the day.
Apr 12, 21:09 UTC

Investigating - Identified - A fraction of traffic for `https://pypi.python.org/simple` is being redirected to `https://pypi.org/simple` as a load test in preparation to begin final rollout of the new codebase.

This will mainly affect `pip` and other Python packaging clients that cannot follow an HTTP 302 redirect from pypi.python.org to pypi.org or are incompatible with the `/simple` resources provided by the new PyPI.

During this period, if you notice any issues with your results from `/simple` or experience problems with the redirect, please notify the PyPI team at https://github.com/pypa/warehouse/issues
Apr 12, 13:29 UTC

Apr 11, 2018

TLSv1.0/TLSv1.1 rolling brownouts

Completed - The rolling brownouts are finished, and TLSv1.0 and TLSv1.1 have been disabled.
Apr 11, 15:37 UTC

Update - The rolling brownouts have been upgraded to a blackout, TLSv1.0 and TLSv1.1 will be rejected with a HTTP 403 at all times.
Apr 8, 15:49 UTC

Update - We have increased the rolling brownouts to the first 20 minutes of every hour, plus the first 20 minutes after each half-hour (:00-:20 and :30-:50).
Apr 6, 11:43 UTC

Update - We have increased the rolling brownouts to the first 15 minutes of every hour, plus the first 15 minutes after each half-hour (:00-:15 and :30-:45).
Apr 3, 11:45 UTC

Update - We have increased the rolling brownouts to the first 15 minutes of every hour, plus the first 10 minutes after each half-hour (:00-:15 and :30-:40).
Mar 31, 16:00 UTC

Update - We have increased the rolling brownouts to the first 15 minutes of every hour, plus the first 5 minutes after each half-hour (:00-:15 and :30-:35).
Mar 28, 16:33 UTC

Update - We are currently enforcing TLSv1.2 during the first 15 minutes of each hour. You can follow the planning and discussion of this maintenance at https://github.com/pypa/warehouse/issues/3411
Mar 27, 16:46 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 22, 18:59 UTC

Scheduled - In preparation for our CDN provider deprecating TLSv1.0 and TLSv1.1 protocols, we have begun rolling brownouts for these protocols for the first ten (10) minutes of each hour.

During that window, clients accessing pypi.python.org with clients that do not support TLSv1.2 will receive an HTTP 403 with the error message "This is a brown out of TLSv1 support. TLSv1 support is going away soon, upgrade to a TLSv1.2+ capable client.".

For users of pip on MacOS/OS X, upgrading to the latest pip should resolve the issue. pip users on other platforms should upgrade their OpenSSL to a version which supports TLSv1.2.

If you have trouble with getting your PyPI client working, file an issue at https://github.com/pypa/packaging-problems/issues.
Mar 22, 18:51 UTC

pypi.org mail outage

A misconfiguration accidentally disabled mail sent from https://pypi.org between 2018-04-11T07:00:00Z and 2018-04-11T10:00:00Z. Email sends enqueued during that period will be retried, but some users may need to reinitiate password resets and account verification emails.

We're working hard to get some of the final pieces in place before launching the new PyPI on April 16th!
Apr 11, 10:37 UTC

Apr 10, 2018

No incidents reported.

Apr 9, 2018

No incidents reported.

Apr 7, 2018

No incidents reported.

Apr 5, 2018

No incidents reported.

Apr 4, 2018

Hosting issue

An administrative issue has occurred which required staff intervention for bugs.python.org. The hosting company has turned the infrastructure service back on.
Apr 4, 13:55 UTC