Railway - Issue with deployments – Incident details

All systems operational

Issue with deployments

Resolved
Major outage
Started about 6 hours agoLasted about 1 hour

Affected

Builds (Railway Metal)

Degraded performance from 11:05 PM to 11:27 PM, Partial outage from 11:27 PM to 11:32 PM, Degraded performance from 11:32 PM to 11:51 PM, Partial outage from 11:51 PM to 12:16 AM, Operational from 12:16 AM to 12:00 AM

Image Registry (Metal)

Degraded performance from 11:05 PM to 11:27 PM, Partial outage from 11:27 PM to 11:32 PM, Degraded performance from 11:32 PM to 11:51 PM, Partial outage from 11:51 PM to 12:16 AM, Operational from 12:16 AM to 12:00 AM

Build Machines (Metal)

Degraded performance from 11:05 PM to 11:27 PM, Partial outage from 11:27 PM to 11:32 PM, Degraded performance from 11:32 PM to 11:51 PM, Partial outage from 11:51 PM to 12:16 AM, Operational from 12:16 AM to 12:00 AM

Deployments (Railway Metal)

Operational from 11:05 PM to 11:52 PM, Partial outage from 11:52 PM to 12:16 AM, Operational from 12:16 AM to 12:00 AM

US West (Metal / California, USA)

Operational from 11:05 PM to 11:52 PM, Partial outage from 11:52 PM to 12:16 AM, Operational from 12:16 AM to 12:00 AM

US East (Metal / Virginia, USA)

Operational from 11:05 PM to 11:52 PM, Partial outage from 11:52 PM to 12:16 AM, Operational from 12:16 AM to 12:00 AM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Update
    Update

    Hobby, free and trial deployments have been re-enabled. We are still monitoring, but have seen full recovery on our deploy systems.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result. Non-pro deploys remain disabled until we catch up with the deploy queue.

  • Update
    Update
    We are continuing to work on a fix for this incident.
  • Update
    Update
    We are continuing to work on a fix for this incident.
  • Identified
    Identified

    We've identified the root cause of the issue and are working on a fix. Free, Trial and Hobby provisions remain disabled at this time.

  • Update
    Update

    We've paused free, trial and hobby plan deploys as we continue investigating the issue.

  • Investigating
    Investigating

    We're investigating an issue causing slowdowns in builds.