Railway - Notice history

All systems operational

Dashboard (railway.com) - Operational

100% - uptime
Sep 2023 · 99.99%Oct · 100.0%Nov · 99.98%

Previous page

Next page

Sep 2023
Oct 2023
Nov 2023
100% - uptime

DNS - Operational

100% - uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2023
Oct 2023
Nov 2023

Traffic - Operational

100% - uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2023
Oct 2023
Nov 2023

Central Station (station.railway.com) - Operational

100% - uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2023
Oct 2023
Nov 2023

Notice history

Nov 2023

Workloads unavailable on Stacker 50
  • Resolved
    Resolved

    This incident has been resolved.

  • Update
    Update

    The machine is currently back online and serving workloads once again. The Railway team will monitor this machine as workloads fully restore. We appreciate your patience.

  • Monitoring
    Monitoring

    The team is currently restarting the affected machine and bringing on workloads back in a staged manner, some workloads may continue to be unavailable.

  • Identified
    Identified

    Workloads on one machine hit a IO limit causing workloads to be unreachable for around 1 hour or so. We advise users to trigger a re-deployment if they haven't. For affected legacy plugins, migrating to a V2 database should resolve the issue.

A small percentage of provisions related to volumes are failing
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We're seeing a very small spike in provision errors for volumes in the US. Looking into it.

Deployments are slow to go out
  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

  • Resolved
    Resolved

    This incident has been resolved.

Deployments are failing to go out
  • Update
    Update

    This incident has been resolved.

  • Resolved
    Resolved

    We had a spike in deploy requests and a small delay for some deploys as they got backlogged.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

404 Flickers on Dashboard for some users
  • Resolved
    Resolved

    This incident has been resolved.

Oct 2023

Volume and database provisions are failing
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

[US West] A small percentage of builds is failing to go out
  • Resolved
    Resolved

    This incident has been resolved. If you run into "Cannot create code snapshot" error, please issue a new build.

  • Monitoring
    Monitoring

    One of our build hosts in US West is down. A small percentage of builds routed to that builder may be failing with "Cannot create code snapshot" errors.

    We've taken the broken builder out of rotation and is currently monitoring the results. Please try issuing a new build.

Volumes are failing to be created in uswest-1
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    Identified

    We are in the process of rolling out a fix. It will be live in a few minutes.

  • Investigating
    Investigating

    We are currently investigating this incident.

A percentage of deployments are failing to go out
  • Resolved
    Resolved

    When we cycled our regional builder fleet, the machine image they booted with had some outdated config. This had the unintended consequence of pushing all images from regional builders to the us-west1 registry which in turn caused deployments to any other region to fail.

    To ensure this doesn't happen again, we have removed the outdated config from our machine images and will be making the config path safer.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

Railway up is returning 504
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

Sep 2023

Deployments are failing to go out
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Update
    Update

    We are currently investigating this incident.

  • Investigating
    Investigating

    We are currently investigating this incident.

Builds are slow to go out
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

A small subset of requests ingested at our us-east edge failed with a 503 error
  • Resolved
    Resolved

    We experienced a high volume of connections from AWS directed at a single tenant via our us-east edge between 19:55 UTC and 20:20UTC. This traffic caused resource exhaustion on our proxies and led to a small subset of connections being terminated. We have mitigated this temporarily and are provisioning more proxy nodes to handle the load in us-east.

Delayed route propagation and failing Private Network DNS lookups
  • Resolved
    Resolved

    This incident has been resolved. We've updated the incident title to match the full scope of the incident.

  • Monitoring
    Monitoring

    The rollback has been deployed and services are back in normal operation. DNS resolution for private networking was also impacted between 20:57:30 UTC and 21:05:00 UTC.

  • Investigating
    Investigating

    An fault in our routing control plane may cause domain updates for some deployments to be delayed. A rollback is in progress.

We're investigating reports of delayed deploy and build logs
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We identified a bottleneck in our log ingestion pipeline that delayed ingest by up to a minute. We've rolled out a fix and log should now be flowing as normal.

  • Investigating
    Investigating

    We are currently investigating this incident.

Sep 2023 to Nov 2023

Next