Resolved: This incident has been resolved.
Python Status’s Tweets
Monitoring: We're seeing improvements after reports from our backend providers that their incident is resolved.
1
1
2
Show this thread
Investigating: We're continuing to investigate with our CDN, but found a potentially related outage published by our backend that we are now tracking as well.
1
1
Show this thread
Investigating: We are working with our CDN provider to determine what is contributing to these latencies.
1
1
Show this thread
Investigating: We are investigating issues with files.pythonhosted.org causing larger files to experience large latencies and timeouts.
2
2
Show this thread
Monitoring: Beginning on Nov 6th, 2022, PyPI began to become inundated with failed authentication attempts due to a dictionary attack (en.wikipedia.org/wiki/Dictionar). At its peak, the attack was attempting more than 260K passwords per hour,…
9
16
We published a note on our October 26th database maintenance:
1
1
1
Show this thread
Completed: All systems are online and we are seeing successful operations and good performance from the backends. This maintenance is complete.
1
5
7
Show this thread
Verifying: Our validation steps have completed and we are currently rolling out a deployment to move to the new database server. This will slowly bring PyPI out of read-only mode. We are monitoring to ensure systems are online and sta…
1
3
2
Show this thread
In Progress: The database restore has finished. Indexes are being built and constraints applied which will take 10-15 minutes. After that we will begin validating consistency, updating database planner statistics, and preparing to swi…
1
1
Show this thread
In Progress: The largest portion of the data migration has completed and we are working on estimating the remaining time before we can begin validation and final steps to bring PyPI out of read-only mode.
1
Show this thread
In Progress: Based on current estimate for completion we have extended this maintenance window by an additional hour. The data migration is progressing and we hope that after the current table completes the remainder will proceed at t…
1
Show this thread
In Progress: We are continuing to monitor the progress of the database migration and will update after its next milestone.
1
Show this thread
In Progress: Based on current estimate for completion we have extended this maintenance window by one hour.
1
3
1
Show this thread
In Progress: The data migration is still in progress, but progressing at a slower rate than expected based on testing. We will have an update on ETA soon but expect to use the entire maintenance window at this time.
1
3
3
Show this thread
In Progress: PyPI has been placed in read-only mode and the database upgrade is now in process. We expect this phase to take 60-90 minutes.
1
4
9
Show this thread
In Progress: Scheduled maintenance is currently in progress. We will provide updates as necessary.
1
Show this thread
Scheduled (Oct 28, 2022, 12:00 UTC): Maintenance will begin as scheduled in 60 minutes.
1
1
Show this thread
Completed: The scheduled maintenance has been completed.
Show this thread
In Progress: Scheduled maintenance is currently in progress. We will provide updates as necessary.
1
Show this thread
Scheduled (Oct 27, 2022, 14:00 UTC): Maintenance will begin as scheduled in 60 minutes.
1
1
Show this thread
Scheduled (Oct 27, 2022, 14:00 UTC): In preparation for a database upgrade scheduled for Oct 28, 2022 12:00-14:00 UTC PyPI will be set to Read Only mode to ensure that impact to the service is as we expect.
During this time, uploads …
2
3
1
Show this thread
Scheduled (Oct 28, 2022, 12:00 UTC): PyPI's databases will be undergoing a major version upgrade during this period. At the beginning of the maintenance window, PyPI will be set to a read only mode to ensure database consistency as da…
1
4
6
Show this thread
We published a postmortem on our October 24th incident:
2
7
Show this thread
Resolved: This incident has been resolved.
1
Show this thread
Monitoring: We have identified and resolved the reason for the outage and are monitoring to ensure it remains stable.
1
Show this thread
Investigating: We are currently investigating this issue. stspg.io/nft2x7gqp8t0?u
3
Investigating: The backend that hosts PyPI and associated services is experiencing a major outage, we are investigating.
1
1
Show this thread
Resolved: All CSS appears to be loading, we will investigate what occurred that caused CSS to fail to load during deployment.
1
2
Show this thread
Monitoring: Some users experienced missing CSS files during a deployment of PyPI. Those files now appear to be serving, we are monitoring for further reports.
1
1
1
Show this thread
Investigating: We are currently investigating this issue.
1
1
2
Show this thread
Resolved: This maintenance is complete. Please file an issue at github.com/pypa/pypi-supp if you experience issues with test.pypi.org after this migration.
1
1
Show this thread
Monitoring: TestPyPI may be unaccessible for up to an hour while we complete migration of the CDN configurations. Services may also be impacted after the migration due to changes in the configuration to better match …
Show this thread
Resolved: This incident has been resolved. stspg.io/m6ptq5bd09hz?u
1
Show this thread
Identified: test.pypi.org's infrastructure is currently under maintenance. File downloads may be impacted while we migrate our CDN for the service.
1
1
1
Show this thread
Resolved: This incident has been resolved.
2
Show this thread
Identified: The underlying cause is that the host is running on a host with degraded storage performance. We will be taking the host back offline to migrate it to a new underlying system.
1
2
Show this thread
Monitoring: Host is back online and we are monitoring the status of the services it backs.
1
1
Show this thread
Identified: Our CDN will continue to serve pages that are cached, but some docs.python.org URLs will not be able to load. In the event of cache expiry some pages that were previously cached may become unavailable.
1
1
Show this thread
Identified: The backends for docs.python.org are currently unavailable. They became unresponsive at approximately 12:00 UTC and we have been unable to bring the cloud server back online via administration panel. We are working…
1
1
3
Show this thread