Railway - Notice history

All systems operational

Dashboard (railway.com) - Operational

100% - uptime
Apr 2024 · 100.0%May · 100.0%Jun · 99.93%

Previous page

Next page

Apr 2024
May 2024
Jun 2024
100% - uptime

DNS - Operational

100% - uptime
Apr 2024 · 100.0%May · 100.0%Jun · 99.63%

Previous page

Next page

Apr 2024
May 2024
Jun 2024

Traffic - Operational

100% - uptime
Apr 2024 · 100.0%May · 100.0%Jun · 99.63%

Previous page

Next page

Apr 2024
May 2024
Jun 2024

Notice history

Jun 2024

New volume provisioning in us-west region might be delayed
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

    We are adding capacity to the us-west region.

  • Investigating
    Investigating
    We are currently investigating this incident.
Dashboard loading slowly for some users
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Dashboard latency has recovered.

  • Identified
    Identified
    We are continuing to work on a fix for this incident.
  • Investigating
    Investigating
    We are currently investigating this incident.
Increased error rates for pulls from dockerhub
  • Resolved
    Resolved

    We've seen error rates drop since 22:30 UTC and docker have moved their incident status to monitoring. Resolving.

  • Identified
    Identified

    This has been confirmed as an upstream outage on DockerHub

  • Investigating
    Investigating

    We're seeing elevated errors for image pulls from dockerhub. Any deploys referencing dockerhub images or builds relying on dockerhub images as their base may fail or be delayed.

Traffic routed over US East to Railway workloads are currently down
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Traffic is now back online. All impacted users should have their services restored. We are seeing high load on our US East proxy and are currently adding further mitigations to prevent this from happening again.

  • Identified
    Identified

    We've isolated this incident to workloads on our new Edge Proxy only.

  • Investigating
    Investigating

    We are currently investigating this incident. Any users connecting to Railway-hosted applications going through our US East edge network is impacted.

Slightly delayed builds. We’re looking into it
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Builds are becoming fully operational again.

  • Investigating
    Investigating
    Slightly delayed builds. We’re looking into it

May 2024

we're investigating the failure of a single host in the us-west1 region
  • Resolved
    Resolved

    All workloads on the affected host have been fully restored.

  • Monitoring
    Monitoring

    All workloads on the affected host have been restored. We're continuing to monitor.

  • Identified
    Identified

    We're continuing to recover the failed host; the replacement is provisioning.

  • Investigating
    Investigating

    A single host in the us-west1 region has encountered a hardware failure. We're investigating and recovering the host.

Dashboard may be slow or failing to load for some users
  • Resolved
    Resolved

    This incident has been resolved. Our database was temporarily unreachable from a subset of our API backend hosts, resulting in slower load times and errors for some users. We have performed a rolling restart of those hosts and are seeing full recovery.

  • Monitoring
    Monitoring

    We have implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating
    We are currently investigating this incident.
Deployments are failing to go out in the Europe region
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating
    Deployments are failing to go out in the Europe region
Builds in us-west are stuck initializing
  • Resolved
    Resolved

    The incident has been resolved. Deploys are building normally now.

  • Monitoring
    Monitoring

    Some of our builders are back online and deployments are slowly progressing

  • Update
    Update

    We are still affected by the GCP outage and are monitoring it closely. https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#wVCaQTRV9VEch1ZTHVQn

  • Identified
    Identified

    GCP Virtual Private Cloud (VPC) is currently having an incident that is affecting our builders (https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#wVCaQTRV9VEch1ZTHVQn). We are working on mitigations

  • Investigating
    Investigating
    We are currently investigating this incident.
Builds may be delayed in EU West
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We have identified and implemented a fix for this issue, and are continuing to monitor the results. Our builders in EU West was under-provisioned leading to slower builds. We have scaled it up, and builds should now be going through normally.

Apr 2024

Builds are delayed
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    New builds are taking longer to process than usual

Healthchecks on user workloads failing with Service Unavailable
  • Resolved
    Resolved

    This is a retrospective incident: Healthchecks on user workloads were failing for the period between 7:10 and 7:40am UTC due to the implementation of an overrestrictive firewall rule. This caused new deployments on services with a healthcheck configured to fail due to a failed healthcheck. We reverted the rule, and healthchecks began operating normally.

Apr 2024 to Jun 2024

Next