Railway - Notice history

All systems operational

Dashboard (railway.com) - Operational

100% - uptime
Aug 2023 · 99.99%Sep · 99.99%Oct · 100.0%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023
100% - uptime

DNS - Operational

100% - uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023

Traffic - Operational

100% - uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023

Central Station (station.railway.com) - Operational

100% - uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023

Notice history

Oct 2023

Volume and database provisions are failing
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

[US West] A small percentage of builds is failing to go out
  • Resolved
    Resolved

    This incident has been resolved. If you run into "Cannot create code snapshot" error, please issue a new build.

  • Monitoring
    Monitoring

    One of our build hosts in US West is down. A small percentage of builds routed to that builder may be failing with "Cannot create code snapshot" errors.

    We've taken the broken builder out of rotation and is currently monitoring the results. Please try issuing a new build.

Volumes are failing to be created in uswest-1
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    Identified

    We are in the process of rolling out a fix. It will be live in a few minutes.

  • Investigating
    Investigating

    We are currently investigating this incident.

A percentage of deployments are failing to go out
  • Resolved
    Resolved

    When we cycled our regional builder fleet, the machine image they booted with had some outdated config. This had the unintended consequence of pushing all images from regional builders to the us-west1 registry which in turn caused deployments to any other region to fail.

    To ensure this doesn't happen again, we have removed the outdated config from our machine images and will be making the config path safer.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

Railway up is returning 504
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

Sep 2023

Deployments are failing to go out
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Update
    Update

    We are currently investigating this incident.

  • Investigating
    Investigating

    We are currently investigating this incident.

Builds are slow to go out
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

A small subset of requests ingested at our us-east edge failed with a 503 error
  • Resolved
    Resolved

    We experienced a high volume of connections from AWS directed at a single tenant via our us-east edge between 19:55 UTC and 20:20UTC. This traffic caused resource exhaustion on our proxies and led to a small subset of connections being terminated. We have mitigated this temporarily and are provisioning more proxy nodes to handle the load in us-east.

Delayed route propagation and failing Private Network DNS lookups
  • Resolved
    Resolved

    This incident has been resolved. We've updated the incident title to match the full scope of the incident.

  • Monitoring
    Monitoring

    The rollback has been deployed and services are back in normal operation. DNS resolution for private networking was also impacted between 20:57:30 UTC and 21:05:00 UTC.

  • Investigating
    Investigating

    An fault in our routing control plane may cause domain updates for some deployments to be delayed. A rollback is in progress.

We're investigating reports of delayed deploy and build logs
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We identified a bottleneck in our log ingestion pipeline that delayed ingest by up to a minute. We've rolled out a fix and log should now be flowing as normal.

  • Investigating
    Investigating

    We are currently investigating this incident.

Aug 2023

A percentage of builds are failing to go through
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

GitHub-triggered deploys may be delayed for some users due to an upstream issue
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    GitHub is currently experiencing issues with their web hooks. Refer to https://www.githubstatus.com/ for more details.

Some builds may be failing to go out
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

We're investigating reports of build failures on a subset of builder hosts
  • Resolved
    Resolved

    The affected build node has been cycled. A critical daemon on the node crashed and did not correctly update the nodes healthcheck. Any work landing on that node was thus instantly failing. We're taking measures to fix the healthcheck to ensure only healthy build nodes take on work.

  • Identified
    Identified

    We've identified the affected builder instance and are cycling it out.

  • Investigating
    Investigating

    We're investigating reports of build failures on a subset of builder hosts

A percentage of users are seeing an issue with builds during snapshot creation
  • Resolved
    Resolved

    We've pushed a fix and builds are going through fine

  • Investigating
    Investigating

    A percentage of users are seeing an issue with builds during snapshot creation

Aug 2023 to Oct 2023

Next