Railway - Notice history

All systems operational

Dashboard (railway.com) - Operational

100% - uptime
May 2025 · 99.99%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2025
Jun 2025
Jul 2025
100% - uptime

DNS - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2025
Jun 2025
Jul 2025

Traffic - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2025
Jun 2025
Jul 2025

Central Station (station.railway.com) - Operational

100% - uptime
May 2025 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2025
Jun 2025
Jul 2025

Notice history

Jul 2025

Canvas might be taking longer than normal to load. We are rolling out a fix
  • Resolved
    Resolved
    the canvas is back to normal
  • Monitoring
    Monitoring
    The canvas is loading normally. We are monitoring the fix
  • Update
    Update
    We are rolling out a fix to improve canvas loading speed
  • Identified
    Identified
    We are continuing to work on a fix
  • Investigating
    Investigating
    Canvas might be taking longer than normal to load. We are rolling out a fix
Canvas may be delayed to show up and function. We are investigating
  • Resolved
    Resolved
    The incident has been resolved and we are seeing performance back to normal levels.
  • Monitoring
    Monitoring
    We are seeing recovery as we continue to monitor the system.
  • Identified
    Identified
    We are noticing some issues with the stability of the canvas. We are continuing with the fix
  • Monitoring
    Monitoring
    The canvas is back to normal. We are monitoring the rollout
  • Update
    Update
    we are still investigating. builds and deploys are working normally. only the visual canvas in the dashboard is slow to load
  • Investigating
    Investigating
    Canvas may be delayed to show up and function. We are investigating
A single host in US East has failed
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    The host has restarted and workloads should be coming back online. We are still monitoring this incident.

  • Identified
    Identified
    We are continuing to work on a fix for this incident.
  • Investigating
    Investigating
    A single host in US East has failed
Elevated deployment latency
  • Resolved
    Resolved

    Our deployment pipeline throughput was impacted by a resource constraint on the database backing our Temporal cluster. We have increased cluster resources and upgraded the database; we have seen deployment throughput restored.

  • Monitoring
    Monitoring

    Queue backpresssure has significantly reduced, we have re-enabled non-pro deployments and are monitoring

  • Identified
    Identified

    Deployments are now processing as we're reducing the queue backpressure. We have halted some background operations to relieve deployment delays.

  • Update
    Update

    We continue to investigate the issue and have halted some active background operations to relieve the deployment delays

  • Investigating
    Investigating

    We are investigating a number of deploys that are stuck initializing. Non-Pro deployments have been temporarily paused.

A single host in EU West is failing over
  • Resolved
    Resolved

    This incident has been resolved. All workloads have been successfully restored.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating

    We are currently investigating this incident. Workloads located on that host may be unavailable during that time.

Jun 2025

Slow builds
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Builds have been delayed over the last ~30 minutes. We're seeing recovery and are monitoring the incident.

Degraded read/write performance on Railway Volumes
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are continuing to monitor, but have seen recovery.

  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We're seeing recovery across the fleet. We are continuing to monitor this incident.

  • Update
    Update

    Users may experience higher read/write latency on Railway Volumes. We are investigating this incident.

  • Investigating
    Investigating
    We are currently investigating this incident.
Degraded read/write performance on Railway Volumes
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    All volumes should be seeing recovery back to nominal latency. We are continuing to monitor the fleet latency.

  • Update
    Update

    Most volumes have recovered to nominal latency. We are continuing to investigate any remaining ones that have not recovered yet.

  • Update
    Update

    Some hosts are showing signs of recovery. We are continuing to investigate.

  • Investigating
    Investigating
    Users may experience higher read/write latency on Railway Volumes. We are investigating this incident.
Elevated Packet Loss on Global Edge Network
  • Resolved
    Resolved
    Due to an erroneous PR that got past health checks, there was an elevated rate of dropped packets on the global edge network for about 20 minutes from 07:45 UTC to 08:05 UTC. Even though we stopped the rollout immediately after observing this failure, we estimate that about 10% of packets were dropped worldwide for this period.
GitHub Access is Down
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    We have found that GitHub access is down. Deployments with GH as a source will not deploy. We are continuing to work on a fix for this incident.

May 2025

we're temporarily pausing deployments while we investgate delays in network propagation
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We have shipped a remedy for network discovery. We have re-enabled deployments. We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating
    we're temporarily pausing deployments while we investgate delays in network propagation
Elevated latency on Edge Network
  • Resolved
    Resolved

    During a routine update to BGP configuration for Railway's Anycast network 66.33.22.0/23, a small proportion (approximately 8%) of upstream ISPs failed to propagate a routing update - this led to a small amount of traffic to be dropped for a period less than 5 mins.

    Once routes propagated, some ISPs were routed in a suboptimal manner leading to elevated latency for some users. We have reverted the routing configuration and are liaising with our vendors to root cause the issue. We will also be creating maintenance windows for future routing config changes and staging their rollout to minimize risk.

  • Monitoring
    Monitoring

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

  • Investigating
    Investigating

    As we're rolling out a routing reconfiguration of our metal edge, a small subset of users may notice latency issues while the new routing configuration is being propagated to all ISPs.

GCP edge network connectivity issues in the EU region
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We are seeing recovery and continuing to monitor the situation.

  • Identified
    Identified

    Attack targeting EU edge identified. Mitigation efforts underway.

  • Investigating
    Investigating
    We are currently investigating this incident.
[Retroactive] Magic Login Outage
  • Resolved
    Resolved

    (This is a retroactive incident) magic login links failed to go out since this morning

Small % of Project 404's
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    We rolled out a change where we dropped a column unsafely. While the new deploy was rolling out (~5 minutes), old deploys, the ORM we used automatically queried the old field. As a result, some small % of users who hard refreshed (and had to load project state) may have seen a 404 briefly.

    We're rolling out a lint rule to catch this in the future. Apologies

May 2025 to Jul 2025

Next