Incident with Webhooks
Incident
Report for GitHub
Resolved
On October 9th, 2025, between 14:35 UTC and 15:21 UTC, a network device in maintenance mode that was undergoing repairs was brought back into production before repairs were completed. Network traffic traversing this device experienced significant packet loss.
Authenticated users of the github.com UI experienced increased latency during the first 5 minutes of the incident. API users experienced up to 7.3% error rates, after which it stabilized to about 0.05% until mitigated. Actions service experienced 24% of runs being delayed for an average of 13 minutes. Large File Storage (LFS) requests experienced minimally increased error rate, with 0.038% of requests erroring.
To prevent similar issues, we are enhancing the validation process for device repairs of this category.
Authenticated users of the github.com UI experienced increased latency during the first 5 minutes of the incident. API users experienced up to 7.3% error rates, after which it stabilized to about 0.05% until mitigated. Actions service experienced 24% of runs being delayed for an average of 13 minutes. Large File Storage (LFS) requests experienced minimally increased error rate, with 0.038% of requests erroring.
To prevent similar issues, we are enhancing the validation process for device repairs of this category.
Update
All services have fully recovered.
Update
Actions has fully recovered but Notifications is still experiencing delays. We will continue to update as the system is fully restored to normal operation.
Update
Actions is operating normally.
Update
Pages is operating normally.
Update
Git Operations is operating normally.
Update
Actions and Notifications are still experiencing delays as we process the backlog. We will continue to update as the system is fully restored to normal operation.
Update
Pull Requests is operating normally.
Update
Actions is experiencing degraded performance. We are continuing to investigate.
Update
We are seeing full recovery in many of our systems, but delays are still expected for actions. We will continue to update as the system is fully restored to normal operation.
Update
Webhooks is operating normally.
Update
Webhooks is experiencing degraded performance. We are continuing to investigate.
Update
Issues is operating normally.
Update
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Update
API Requests is operating normally.
Update
We identified a faulty network component and have removed it from the infrastructure. Recovery has started and we expect full recovery shortly.
Update
Pull Requests is experiencing degraded availability. We are continuing to investigate.
Update
Git Operations is experiencing degraded performance. We are continuing to investigate.
Update
Actions is experiencing degraded availability. We are continuing to investigate.
Update
We are investigating widespread reports of delays and increased latency in various services. We will continue to keep users updated on progress toward mitigation.
Update
Issues is experiencing degraded availability. We are continuing to investigate.
Update
API Requests is experiencing degraded performance. We are continuing to investigate.
Update
Pages is experiencing degraded performance. We are continuing to investigate.
Update
Actions is experiencing degraded performance. We are continuing to investigate.
Investigating
We are investigating reports of degraded availability for Webhooks
This incident affected: Git Operations, Webhooks, API Requests, Issues, Pull Requests, Actions, and Pages.
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe