Python Infrastructure Status

Python Infrastructure

All Systems Operational

python.org Operational

pypi.python.org Operational

docs.python.org Operational

hg.python.org Operational

bugs.python.org Operational

wiki.python.org Operational

mail.python.org Operational

Mailing Lists and Archives Operational

Message Handling Services Operational

pypy.org Operational

speed.pypy.org Operational

psfmember.org Operational

Content Delivery Network Operational

Fastly US East (JFK) Operational

Fastly Asia/Pacific (HK) Operational

Fastly US East (IAD) Operational

Fastly US East (ATL) Operational

Fastly US East (MIA) Operational

Fastly US Central (DEN) Operational

Fastly US Central (DFW) Operational

Fastly US Central (ORD) Operational

Fastly US West (LAX) Operational

Fastly US West (SEA) Operational

Fastly US West (SJC) Operational

Fastly Europe (FRA) Operational

Fastly Europe (AMS) Operational

Fastly Europe (LCY) Operational

Fastly Europe (LHR) Operational

Fastly Asia/Pacific (SYD) Operational

Fastly Asia/Pacific (SIN) Operational

Fastly Asia/Pacific (TYO) Operational

Fastly Asia/Pacific (NZ) Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

System Metrics Month Week Day

PyPI Errors

Fetching

PyPI CDN Miss Latency

Fetching

Past Incidents

Jan 3, 2018

No incidents reported today.

Jan 2, 2018

No incidents reported.

Jan 1, 2018

No incidents reported.

Dec 31, 2017

Bugs proxy error

Resolved - This incident has been resolved.
Dec 31, 04:40 UTC

Identified - The bugs.python.org disk was full, we've cleared up some space and monitoring the situation.
Dec 31, 04:24 UTC

Investigating - We are currently investigating an issue with bugs.python.org.
Dec 31, 01:17 UTC

Dec 30, 2017

No incidents reported.

Dec 29, 2017

No incidents reported.

Dec 28, 2017

No incidents reported.

Dec 27, 2017

OSUOSL Ganeti Cluster Maintenance

Completed - The scheduled maintenance has been completed.
Dec 27, 23:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 27, 17:00 UTC

Scheduled - OSUOSL Staff will be performing maintenance and updates to the Ganeti Cluster. During this time, PSF infrastructure at OSUOSL maybe intermittently inaccessible.
Dec 24, 23:39 UTC

Dec 26, 2017

PyPI: Elevated Error Rates for /packages

Resolved - Errors for /packages urls have subsided. During the course of responding to this incident, we determined that the mirror that is intended to act as a failover in situations like this had run out of disk space.

We have begun creating a new mirror with the necessary storage and verifying that failover can take place.

Additionally, we've added monitoring to alert when mirror space is low which should avoid surprises like this in the future.
Dec 26, 20:30 UTC

Monitoring - Errors have subsided, we're continuing to monitor and take other preventative measures.
Dec 26, 19:56 UTC

Identified - The object store that hosts PyPI's files is experiencing intermittent issues serving packages. The error rate is minimal, but we are continuing to work to resolve the issue.
Dec 26, 19:47 UTC

Investigating - Our automated systems detected an elevated error rate for `/packages` urls. We are investigating the issue and will update shortly.
Dec 26, 19:39 UTC

Dec 25, 2017

No incidents reported.

Dec 24, 2017

No incidents reported.

Dec 23, 2017

PyPI Outage

Resolved - Incident is resolved. During the database failover 37,889 requests received an error at our CDN edge and 3,723 requests to backend servers resulted in an error.

No data loss or ongoing impact to PyPI was detected.
Dec 23, 14:45 UTC

Monitoring - PyPI service is restored and we are actively monitoring and ensuring data consistency. All looks good and this incident will be marked resolved shortly.
Dec 23, 14:42 UTC

Identified - Our high availability database system underwent a failover event causing all backend web servers to stop serving requests.

The database system failover took approximately six minutes but has completed and servers are coming back online.
Dec 23, 14:40 UTC

Investigating - We are investigating hard outage of PyPI's backend servers.

Simple Index and Package installs should continue to be served by the CDN while we investigate.
Dec 23, 14:38 UTC

Dec 22, 2017

No incidents reported.

Dec 21, 2017

OSUOSL Ganeti Cluster Upgrade

Completed - The scheduled maintenance has been completed.
Dec 21, 20:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 21, 17:00 UTC

Scheduled - OSUOSL Staff will be performing a Ganeti Cluster Upgrade from Ganeti 2.6.2 to 2.15.2. During this time VM's at OSUOSL maybe intermittently down as the upgrade is completed.

A list of hosts that maybe affected at OSUOSL is located here: https://github.com/python/psf-chef/blob/master/hostnames.
Dec 17, 23:30 UTC

Dec 20, 2017

No incidents reported.