Railway - Issues with networking on a subset of hosts – Incident details

All systems operational

Issues with networking on a subset of hosts

Resolved
Major outage
Started about 8 hours agoLasted 37 minutes

Affected

Deployments (Railway Metal)

Partial outage from 9:27 AM to 9:52 AM, Degraded performance from 9:52 AM to 10:04 AM

US West (Metal / California, USA)

Partial outage from 9:27 AM to 9:52 AM, Degraded performance from 9:52 AM to 10:04 AM

US East (Metal / Virginia, USA)

Partial outage from 9:27 AM to 9:52 AM, Degraded performance from 9:52 AM to 10:04 AM

EU West (Metal / Amsterdam, Netherlands)

Partial outage from 9:27 AM to 9:52 AM, Degraded performance from 9:52 AM to 10:04 AM

Southeast Asia (Metal / SIngapore)

Partial outage from 9:27 AM to 9:52 AM, Degraded performance from 9:52 AM to 10:04 AM

Private Networking

Operational from 9:27 AM to 9:42 AM, Partial outage from 9:42 AM to 9:52 AM, Degraded performance from 9:52 AM to 10:04 AM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    Identified

    We have identified the root cause and are working on a fix

  • Update
    Update

    We were able to confirm impact to be limited to part of the fleet, but to be affecting all networking for affected deployments.

  • Investigating
    Investigating

    We are currently investigating failures to reach the internet for a subset of deployments