Railway - We are investigating issues with elevated network latency and slow deploys – Incident details

All systems operational

We are investigating issues with elevated network latency and slow deploys

Monitoring
Major outage
Started about 5 hours ago

Affected

Deployments (Railway Metal)

Degraded performance from 7:10 AM to 7:37 AM, Major outage from 7:37 AM to 8:06 AM, Partial outage from 8:06 AM to 8:30 AM, Degraded performance from 8:30 AM to 9:08 AM, Partial outage from 9:08 AM to 11:00 AM, Degraded performance from 10:30 AM to 11:00 AM, Operational from 11:00 AM to 12:00 AM

US West (Metal / California, USA)

Degraded performance from 7:10 AM to 7:37 AM, Major outage from 7:37 AM to 8:06 AM, Partial outage from 8:06 AM to 8:30 AM, Degraded performance from 8:30 AM to 9:08 AM, Partial outage from 9:08 AM to 10:30 AM, Degraded performance from 10:30 AM to 11:00 AM, Operational from 11:00 AM to 12:00 AM

US East (Metal / Virginia, USA)

Degraded performance from 7:10 AM to 7:37 AM, Major outage from 7:37 AM to 8:06 AM, Partial outage from 8:06 AM to 8:30 AM, Degraded performance from 8:30 AM to 9:08 AM, Partial outage from 9:08 AM to 10:30 AM, Degraded performance from 10:30 AM to 11:00 AM, Operational from 11:00 AM to 12:00 AM

EU West (Metal / Amsterdam, Netherlands)

Degraded performance from 7:10 AM to 7:37 AM, Major outage from 7:37 AM to 8:06 AM, Partial outage from 8:06 AM to 8:30 AM, Degraded performance from 8:30 AM to 9:08 AM, Partial outage from 9:08 AM to 10:30 AM, Degraded performance from 10:30 AM to 11:00 AM, Operational from 11:00 AM to 12:00 AM

Southeast Asia (Metal / SIngapore)

Degraded performance from 7:10 AM to 7:37 AM, Major outage from 7:37 AM to 8:06 AM, Partial outage from 8:06 AM to 8:30 AM, Degraded performance from 8:30 AM to 9:08 AM, Partial outage from 9:08 AM to 11:00 AM, Operational from 11:00 AM to 12:00 AM

Private Networking

Degraded performance from 7:10 AM to 7:37 AM, Operational from 7:10 AM to 7:37 AM, Major outage from 7:37 AM to 8:06 AM, Partial outage from 8:06 AM to 8:30 AM, Degraded performance from 8:30 AM to 9:08 AM, Partial outage from 9:08 AM to 10:30 AM, Degraded performance from 10:30 AM to 11:00 AM, Operational from 11:00 AM to 12:00 AM

Updates
  • Update
    Update

    We have seen widespread recovery and are continuing to monitor this incident.

  • Update
    Update

    We are still monitoring this incident.

  • Update
    Update

    We are seeing further degradation, particularly in EU regions. We are continuing to work on a permanent fix.

  • Update
    Update

    We have seen recovery in all regions. We are continuing to monitor this incident.

  • Monitoring
    Monitoring

    We are rolling out a fix and are seeing recovery in all regions. Deploys have been re-enabled

  • Identified
    Identified

    We continue to see degradation across all regions, and are working on a fix. We have paused new deploys as we investigate.

  • Investigating
    Investigating
    We are currently investigating this incident.