Railway - Notice history

Dashboard (railway.com) experiencing partial outage

Dashboard (railway.com) - Partial outage

100% - uptime
Aug 2024 · 99.90%Sep · 99.68%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024
100% - uptime

DNS - Operational

100% - uptime
Aug 2024 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024

Traffic - Operational

100% - uptime
Aug 2024 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024

Central Station (station.railway.com) - Operational

100% - uptime
Aug 2024 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024

Notice history

Oct 2024

We are investigating an issue with the edge network proxy
  • Resolved
    Resolved

    We have resolved this issue.

  • Monitoring
    Monitoring

    A fix has been implemented and we're currently monitoring the edge network proxy.

  • Identified
    Identified

    We have identified the issue as a DDoS attack. We're in the process of applying additional mitigations

  • Investigating
    Investigating

    We are currently investigating this incident.

A host in the us-west region experienced an unexpected restarts. We're recovering the last workloads on this host that failed to restore automatically.
  • Resolved
    Resolved
    The host has fully recovered and all impacted workloads are back online.
  • Investigating
    Investigating
    A host in the us-west region experienced an unexpected restarts. We're recovering the last workloads on this host that failed to restore automatically.
[Retroactive] A single host in US East has restarted
  • Resolved
    Resolved

    (This is a retroactive incident) During this time we noticed a single host in the US East 1 region had restarted and we quickly worked to bring containers back online, users may have not been able to dial their deployments or connect to databases during the restart efforts. This incident has been resolved.

[Retroactive] Two hosts in EU West experienced degraded performance
  • Resolved
    Resolved

    This is a retroactive incident which is resolved. For the period of the incident (11:45-12:35 UTC), some workloads may have exhibited connection problems. The incident was due to resource exhaustion on the two hosts due to unaccounted resource thresholds. We have improved our monitoring to better respond to this type of incident and are working on preventative measures in our orchestration.

We have observed degraded performance of the proxy layer. Users may experience dropped connections. We are investigating.
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We have taken action to mitigate the issue and are monitoring.
  • Investigating
    Investigating

    We have observed degraded performance of the edge proxy layer. Users may experience dropped connections. We are investigating.

Sep 2024

[External] Dockerhub Image pull returns 401 Unauthorized
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    We are currently investigating this incident. For more info see Docker's status page: https://www.dockerstatus.com/

Failing to create volumes
  • Resolved
    Resolved

    We have traced the issue to a guard that was trying to prevent users to deploying volumes on Railway Metal. We have since applied the fix and systems are working normally.

  • Investigating
    Investigating
    Failing to create volumes
Dockerhub is currently undergoing maintenance.
  • Resolved
    Resolved

    Dockerhub has completed their maintenance

  • Identified
    Identified

    During this period, builds might fail, displaying a "401 Unauthorized" message in the logs.

    Please check Dockerhub's status page for more information.

    https://www.dockerstatus.com/

Edge Network Hardware Upgrades
  • Completed
    September 19, 2024 at 9:08 PM
    Completed
    September 19, 2024 at 9:08 PM

    We have successfully upgraded our edge network hardware. You should not experience any further connection disconnects.

    Our monitors during this time show no degradation to TCP connections or HTTP requests failures while we performed this across our fleet but as mentioned, closing your connection to shutdown the machine was unavoidable. Your clients would reconnect and continue to work as expected.

  • Update
    September 19, 2024 at 8:17 PM
    In progress
    September 19, 2024 at 8:17 PM

    We are upgrading our final set of machines ahead of schedule.

  • In progress
    September 19, 2024 at 6:43 PM
    In progress
    September 19, 2024 at 6:43 PM

    This upgrade is in progress.

  • Planned
    September 19, 2024 at 6:00 PM
    Planned
    September 19, 2024 at 6:00 PM

    We are upgrading the hardware for our edge network that powers TCP and HTTP proxying. During this time requests to boxes being upgraded will have 5 minutes to complete before being disconnected. These clients will be able to reconnect and will be routed to another box that will handle the traffic indefinitely.

Builds are currently in a queueing state. We're lookinig into it
  • Resolved
    Resolved
    Queue back to zero
  • Monitoring
    Monitoring
    Queued backpressure is almost drained. Most queued deploys have been rolled out
  • Identified
    Identified
    Deploys are going out and the queue is draining. We will move to monitoring when there are no queued builds
  • Investigating
    Investigating
    Builds are currently in a queueing state. We're lookinig into it

Aug 2024

We're seeing degraded performance on our Asia proxy
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We've restarted the impacted boxes and restored service. We are still monitoring
  • Investigating
    Investigating
    We're seeing degraded performance on our Asia proxy
Asia edge proxy (v2) service degradation
  • Resolved
    Resolved

    This is a retroactive incident about severe service degradation that happened between 12:40 PM UTC and 13:10 PM UTC on our edge proxies in south-east Asia. We are conducting an investigation into what happened and will update this incident once it's complete.

Edge Proxy outage
  • Resolved
    Resolved
    This incident has been resolved.
  • Update
    Update

    All proxies are back up in all regions. We are still monitoring as everything is coming back up.

  • Update
    Update

    Europe and US-East proxies are back up

  • Update
    Update

    Southeast Asia (Singapore) is back up

  • Monitoring
    Monitoring

    Most of the traffic appears to be back up, but we're still monitoring.

  • Update
    Update

    Our fix is rolling out and we are seeing recovery for some customers.

  • Identified
    Identified

    Our new edge proxy is experiencing a major outage. We have identified the issue and are working on a fix

Some requests from our edge proxy in Europe may have experienced higher latency than normal
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    Our proxy in Europe continued to process requests but from August 25th, 20:30 to August 26th, 15:00, HTTP latency may have been elevated as we dealt with a sudden increase in traffic from this region.

Builds are queuing up due to a bad rollout on the workers. We're reverting. Builds triggered in the last 10 minutes will start to go out without needing to be retriggered
  • Resolved
    Resolved
    Builds are going out nominally.
  • Monitoring
    Monitoring
    We have restored the build queue and are now monitoring the health of remaining builds.
  • Investigating
    Investigating
    Builds are queuing up due to a bad rollout on the workers. We're reverting. Builds triggered in the last 10 minutes will start to go out without needing to be retriggered

Aug 2024 to Oct 2024

Next