Railway - Deployment slowness – Incident details

Deployment slowness

Monitoring
Partial outage
Started about 5 hours ago

Affected

Deployments (Railway Metal)

Degraded performance from 3:19 PM to 6:58 PM, Partial outage from 6:58 PM to 7:51 PM, Degraded performance from 7:51 PM to 8:07 PM, Operational from 8:07 PM to 12:00 AM

US West (Metal / California, USA)

Degraded performance from 3:19 PM to 6:58 PM, Partial outage from 6:58 PM to 7:51 PM, Degraded performance from 7:51 PM to 8:07 PM, Operational from 8:07 PM to 12:00 AM

US East (Metal / Virginia, USA)

Degraded performance from 3:19 PM to 6:58 PM, Partial outage from 6:58 PM to 7:51 PM, Degraded performance from 7:51 PM to 8:07 PM, Operational from 8:07 PM to 12:00 AM

EU West (Metal / Amsterdam, Netherlands)

Degraded performance from 3:19 PM to 6:58 PM, Partial outage from 6:58 PM to 7:51 PM, Degraded performance from 7:51 PM to 8:07 PM, Operational from 8:07 PM to 12:00 AM

Southeast Asia (Metal / SIngapore)

Degraded performance from 3:19 PM to 6:58 PM, Partial outage from 6:58 PM to 7:51 PM, Degraded performance from 7:51 PM to 8:07 PM, Operational from 8:07 PM to 12:00 AM

Updates
  • Update
    Update

    We are seeing improvement — deployments are now processing without being queued. We continue to monitor the situation and will provide a further update once we are confident the issue is fully resolved

  • Update
    Update

    We've successfully eliminated slowness from our deployment pipeline, and are re-enabling deployments for all plans as the backpressure is recovering.

  • Update
    Update

    Elevated load continues to cause slowness on our system. We' have restored all Pro deployments back to nominal, we are turning on Hobby slowly to assist with deployment backpressure.

  • Update
    Update

    Elevated load continues to cause slowness on our system. We're pausing Free, Trial and Hobby deployments to assist with deployment backpressure.

  • Update
    Update

    Deployment slowness is persisting. Deploys are completing but with longer than normal queue and build times. Our team is actively working on a resolution.

  • Update
    Update

    Our team continues to work on this incident. Deploys are still slow, but not failing.

  • Update
    Update

    We are continuing to monitor this incident. Deployments may be queued, but are going through

  • Monitoring
    Monitoring

    We're seeing recovery, with delays primarily impacting trial deployments. We are monitoring for full recovery.

  • Identified
    Identified

    We've pushed out a fix and are seeing slow recovery.

  • Investigating
    Investigating

    We are currently investigating an issue causing deployments to take longer than normal