Python Infrastructure
Update - There have been a few attempts at submitting new spam to PyPI over the last 36 hours. We are monitoring and using some rapid-response tooling to remove spam as it arrives. This seems to be reducing the frequency of attempts.

Development has begun on our longer term strategies and we'll update here as those progress through the week.
Feb 20, 16:27 UTC
Update - No new spam has arrived on PyPI since reenabling new Project registration. We are monitoring new project registration closely and planning our next steps. Based on information from the team behind https://www.npmjs.com, these spammers are rather ruthless and we expect them to return.

Here's a quick summary of what's been done so far:

- Require at least one verified email address per user in order to register new projects
- Admin feature flags to allow for quickly disabling new project registration and new user registration
- Admin features for spam User and Project cleanup
- Disabled User registration on https://pypi.python.org and direct users to register at https://pypi.org

We're working on automated spam classification of Projects and Releases along with Admin features for us to train the model taking into account User reports of spam as well.
Feb 19, 13:12 UTC
Monitoring - We've completed initial cleanup and are preparing for next phase of features related to automated classification, community reporting, and admin verification of spam.

New Project registrations are re-enabled for the time being to determine the efficacy of our existing prevention methods.
Feb 19, 05:08 UTC
Update - We've begun cleanup of spam from the first volley. Once that is complete, we'll assess and consider re-enabling new Project registration. Thanks for your patience as we work through this unfortunate situation.
Feb 19, 04:02 UTC
Update - The PyPI Administrators have decided to temporarily disable new Project registration on PyPI.

We understand that this will be disruptive for some users, but have decided to take this action to allow for two things:
- A brief rest for the PyPI administrators
- Perhaps convince the spammers to leave PyPI alone

See https://pypi.org/help/#admin-intervention for more details.

Note: Existing Projects will still be able to upload new releases.

We have a few more mitigations in the works, but have decided that protecting our contributors from burnout is of higher importance than resolving this issue in a haphazard or partial manner.
Feb 18, 18:24 UTC
Update - We've taken additional action to dissuade spammers by disabling registration via https://pypi.python.org and forcing user registration to take place via https://pypi.org (the new codebase) which has a CAPTCHA.

We'll begin disabling attacker accounts after this change goes live to see if they return.
Feb 18, 18:01 UTC
Update - We have shipped the first of a handful of spam mitigations, PyPI users are now required to have a verified email address in order to register new projects.

See https://pypi.org/help/#verified-email for details on the change and how you can retroactively verify your email address.

We're still waiting to determine the efficacy of our spam prevention techniques before undertaking cleanup, as it is costly in time for the administrators of PyPI and best done in large batches rather than repeating the process multiple times.
Feb 18, 15:05 UTC
Update - First pass of our initial spam prevention measures are feature complete, but require review. We'll still be waiting to clean up the spam until after they ship and we're able to reduce the rate of incoming spam, which should occur tomorrow.
Feb 18, 01:44 UTC
Identified - Two spam prevention measures are nearly ready for review and deploy. After those are out, we'll begin analysis and cleanup of today's volley of Spam.
Feb 18, 00:25 UTC
Investigating - PyPI is currently being spammed in a relatively well designed way. We are working to put measures in place to stop new spam before beginning cleanup efforts.
Feb 17, 19:22 UTC
python.org   Operational
pypi.python.org   Operational
docs.python.org   ? Operational
hg.python.org   Operational
bugs.python.org   Operational
wiki.python.org   Operational
mail.python.org Operational
Mailing Lists and Archives   Operational
Message Handling Services   Operational
pypy.org   Operational
speed.pypy.org   Operational
psfmember.org   Operational
Content Delivery Network Operational
Fastly US East (JFK)   Operational
Fastly Asia/Pacific (HK)   Operational
Fastly US East (IAD)   Operational
Fastly US East (ATL)   Operational
Fastly US East (MIA)   Operational
Fastly US Central (DEN)   Operational
Fastly US Central (DFW)   Operational
Fastly US Central (ORD)   Operational
Fastly US West (LAX)   Operational
Fastly US West (SEA)   Operational
Fastly US West (SJC)   Operational
Fastly Europe (FRA)   Operational
Fastly Europe (AMS)   Operational
Fastly Europe (LCY)   Operational
Fastly Europe (LHR)   Operational
Fastly Asia/Pacific (SYD)   Operational
Fastly Asia/Pacific (SIN)   Operational
Fastly Asia/Pacific (TYO)   Operational
Fastly Asia/Pacific (NZ)   Operational
us.pycon.org   Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
System Metrics Month Week Day
PyPI Errors ?
Fetching
PyPI CDN Miss Latency ?
Fetching
Past Incidents
Feb 22, 2018

No incidents reported today.

Feb 21, 2018
Resolved - pypi.org and upload.pypi.org backends are stable, we're continuing to monitor and will address and tune as necessary.
Feb 21, 20:16 UTC
Monitoring - We've deployed our solution to and are monitoring the results!
Feb 21, 19:08 UTC
Update - We have a fix going out that will stop our backends from running out of memory! This should lead to a much more stable pypi.org and upload.pypi.org endpoints!
Feb 21, 18:57 UTC
Identified - we are experiencing issues with pypi.org and the upload.pypi.org endpoint. we're actively working to resolve the issue and will update here as we're able.
Feb 21, 16:33 UTC
Feb 16, 2018

No incidents reported.

Feb 15, 2018

No incidents reported.

Feb 14, 2018

No incidents reported.

Feb 13, 2018

No incidents reported.

Feb 12, 2018

No incidents reported.

Feb 11, 2018

No incidents reported.

Feb 10, 2018

No incidents reported.

Feb 9, 2018

No incidents reported.

Feb 8, 2018

No incidents reported.