Railway - Notice history

All systems operational

Dashboard (railway.com) - Operational

100% - uptime
Jun 2024 · 99.93%Jul · 99.99%Aug · 99.90%

Previous page

Next page

Jun 2024
Jul 2024
Aug 2024
100% - uptime

DNS - Operational

100% - uptime
Jun 2024 · 99.63%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2024
Jul 2024
Aug 2024

Traffic - Operational

100% - uptime
Jun 2024 · 99.63%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2024
Jul 2024
Aug 2024

Central Station (station.railway.com) - Operational

100% - uptime
Jun 2024 · 100.0%Jul · 100.0%Aug · 100.0%

Previous page

Next page

Jun 2024
Jul 2024
Aug 2024

Notice history

Aug 2024

We're seeing degraded performance on our Asia proxy
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We've restarted the impacted boxes and restored service. We are still monitoring
  • Investigating
    Investigating
    We're seeing degraded performance on our Asia proxy
Asia edge proxy (v2) service degradation
  • Resolved
    Resolved

    This is a retroactive incident about severe service degradation that happened between 12:40 PM UTC and 13:10 PM UTC on our edge proxies in south-east Asia. We are conducting an investigation into what happened and will update this incident once it's complete.

Edge Proxy outage
  • Resolved
    Resolved
    This incident has been resolved.
  • Update
    Update

    All proxies are back up in all regions. We are still monitoring as everything is coming back up.

  • Update
    Update

    Europe and US-East proxies are back up

  • Update
    Update

    Southeast Asia (Singapore) is back up

  • Monitoring
    Monitoring

    Most of the traffic appears to be back up, but we're still monitoring.

  • Update
    Update

    Our fix is rolling out and we are seeing recovery for some customers.

  • Identified
    Identified

    Our new edge proxy is experiencing a major outage. We have identified the issue and are working on a fix

Some requests from our edge proxy in Europe may have experienced higher latency than normal
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    Our proxy in Europe continued to process requests but from August 25th, 20:30 to August 26th, 15:00, HTTP latency may have been elevated as we dealt with a sudden increase in traffic from this region.

Builds are queuing up due to a bad rollout on the workers. We're reverting. Builds triggered in the last 10 minutes will start to go out without needing to be retriggered
  • Resolved
    Resolved
    Builds are going out nominally.
  • Monitoring
    Monitoring
    We have restored the build queue and are now monitoring the health of remaining builds.
  • Investigating
    Investigating
    Builds are queuing up due to a bad rollout on the workers. We're reverting. Builds triggered in the last 10 minutes will start to go out without needing to be retriggered

Jul 2024

a single host encountered a hardware failure and is restarting, a small portion of deploys in us-west may be unavailable
  • Resolved
    Resolved

    All applications on the impacted host have resumed.

  • Monitoring
    Monitoring

    The host recovered and applications on it are coming back online. It may take a few minutes for all applications on this box to be available.

  • Investigating
    Investigating
    We are currently investigating this incident.
We're investigating routing failures for deployments using static IPs for egress.
  • Resolved
    Resolved
    An update to our routing control plane incorrectly propagated a bad route for regional egress proxies, this led to the drop of static IP egress traffic from 21:28 UTC to approximately 22:00 UTC. Our static IP monitoring was configured globally in our largest region. We're adding regional static IP monitoring to catch this in staged rollouts (aka, much quicker).
  • Monitoring
    Monitoring
    We've reverted the routing config and static IPs traffic is flowing normally.
  • Identified
    Identified
    we've identified a configuration change in our network routing config that looks to have caused the problem. We're rolling out a fix globally.
  • Investigating
    Investigating
    We're investigating routing failures for deployments using static IPs for egress.
Elevated image pull times for GitHub Container Registry
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We are experiencing elevated pull times for images hosted in GitHub Container Registry (GHCR). This appears to be related to a third-party incident on GHCR's CDN (https://www.fastlystatus.com/incident/376717).

We're investigating a single host failure in the us-east4 region, some deployments and volumes in the region may be inaccessible.
  • Resolved
    Resolved

    This incident has been resolved. We found a bug in our scheduling algorithm which resulted in an instance in the US East region to schedule replica workloads on non-unique hosts, resulting in over subscription. To mitigate this, we have provisioned extra capacity in our non-primary regions (US East, Europe, and Southeast Asia)

    Additionally, we will be digging into the scheduling issue to prevent this in the future.

  • Monitoring
    Monitoring

    The impacted host has been successfully restarted and all affected workloads are back online. We are investigating the root failure cause and monitoring the host closely.

  • Identified
    Identified

    We've begun a failover process to recover the impacted host.

  • Investigating
    Investigating
    We're investigating a single host failure in the us-east4 region, some deployments and volumes in the region may be inaccessible.
Some requests from v2 Proxy in US-West and Asia returning incorrectly encoded responses
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Impacted regions have been rolled back and responses should be correct now.

  • Identified
    Identified

    The recent change is reverting.

Jun 2024

New volume provisioning in us-west region might be delayed
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

    We are adding capacity to the us-west region.

  • Investigating
    Investigating
    We are currently investigating this incident.
Dashboard loading slowly for some users
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Dashboard latency has recovered.

  • Identified
    Identified
    We are continuing to work on a fix for this incident.
  • Investigating
    Investigating
    We are currently investigating this incident.
Increased error rates for pulls from dockerhub
  • Resolved
    Resolved

    We've seen error rates drop since 22:30 UTC and docker have moved their incident status to monitoring. Resolving.

  • Identified
    Identified

    This has been confirmed as an upstream outage on DockerHub

  • Investigating
    Investigating

    We're seeing elevated errors for image pulls from dockerhub. Any deploys referencing dockerhub images or builds relying on dockerhub images as their base may fail or be delayed.

Traffic routed over US East to Railway workloads are currently down
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Traffic is now back online. All impacted users should have their services restored. We are seeing high load on our US East proxy and are currently adding further mitigations to prevent this from happening again.

  • Identified
    Identified

    We've isolated this incident to workloads on our new Edge Proxy only.

  • Investigating
    Investigating

    We are currently investigating this incident. Any users connecting to Railway-hosted applications going through our US East edge network is impacted.

Slightly delayed builds. We’re looking into it
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Builds are becoming fully operational again.

  • Investigating
    Investigating
    Slightly delayed builds. We’re looking into it

Jun 2024 to Aug 2024

Next