Railway - Notice history

All proxies are back up in all regions. We are still monitoring as everything is coming back up.

Europe and US-East proxies are back up

Southeast Asia (Singapore) is back up

Most of the traffic appears to be back up, but we're still monitoring.

Our fix is rolling out and we are seeing recovery for some customers.

All applications on the impacted host have resumed.

The host recovered and applications on it are coming back online. It may take a few minutes for all applications on this box to be available.

This incident has been resolved. We found a bug in our scheduling algorithm which resulted in an instance in the US East region to schedule replica workloads on non-unique hosts, resulting in over subscription. To mitigate this, we have provisioned extra capacity in our non-primary regions (US East, Europe, and Southeast Asia)

Additionally, we will be digging into the scheduling issue to prevent this in the future.

The impacted host has been successfully restarted and all affected workloads are back online. We are investigating the root failure cause and monitoring the host closely.

We've begun a failover process to recover the impacted host.

Impacted regions have been rolled back and responses should be correct now.

We implemented a fix and are currently monitoring the result.

We are adding capacity to the us-west region.

Dashboard latency has recovered.

We've seen error rates drop since 22:30 UTC and docker have moved their incident status to monitoring. Resolving.

This has been confirmed as an upstream outage on DockerHub

This incident has been resolved.

Traffic is now back online. All impacted users should have their services restored. We are seeing high load on our US East proxy and are currently adding further mitigations to prevent this from happening again.

We've isolated this incident to workloads on our new Edge Proxy only.

Builds are becoming fully operational again.