Railway - Notice history

All systems operational

Dashboard (railway.com) - Operational

100% - uptime
Jul 2024 · 99.99%Aug · 99.90%Sep · 99.68%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024
100% - uptime

DNS - Operational

100% - uptime
Jul 2024 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

Traffic - Operational

100% - uptime
Jul 2024 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

Central Station (station.railway.com) - Operational

100% - uptime
Jul 2024 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

Notice history

Sep 2024

[External] Dockerhub Image pull returns 401 Unauthorized
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    We are currently investigating this incident. For more info see Docker's status page: https://www.dockerstatus.com/

Failing to create volumes
  • Resolved
    Resolved

    We have traced the issue to a guard that was trying to prevent users to deploying volumes on Railway Metal. We have since applied the fix and systems are working normally.

  • Investigating
    Investigating
    Failing to create volumes
Dockerhub is currently undergoing maintenance.
  • Resolved
    Resolved

    Dockerhub has completed their maintenance

  • Identified
    Identified

    During this period, builds might fail, displaying a "401 Unauthorized" message in the logs.

    Please check Dockerhub's status page for more information.

    https://www.dockerstatus.com/

Edge Network Hardware Upgrades
  • Completed
    September 19, 2024 at 9:08 PM
    Completed
    September 19, 2024 at 9:08 PM

    We have successfully upgraded our edge network hardware. You should not experience any further connection disconnects.

    Our monitors during this time show no degradation to TCP connections or HTTP requests failures while we performed this across our fleet but as mentioned, closing your connection to shutdown the machine was unavoidable. Your clients would reconnect and continue to work as expected.

  • Update
    September 19, 2024 at 8:17 PM
    Update
    September 19, 2024 at 8:17 PM

    We are upgrading our final set of machines ahead of schedule.

  • In progress
    September 19, 2024 at 6:43 PM
    In progress
    September 19, 2024 at 6:43 PM

    This upgrade is in progress.

  • Planned
    September 19, 2024 at 6:00 PM
    Planned
    September 19, 2024 at 6:00 PM

    We are upgrading the hardware for our edge network that powers TCP and HTTP proxying. During this time requests to boxes being upgraded will have 5 minutes to complete before being disconnected. These clients will be able to reconnect and will be routed to another box that will handle the traffic indefinitely.

Builds are currently in a queueing state. We're lookinig into it
  • Resolved
    Resolved
    Queue back to zero
  • Monitoring
    Monitoring
    Queued backpressure is almost drained. Most queued deploys have been rolled out
  • Identified
    Identified
    Deploys are going out and the queue is draining. We will move to monitoring when there are no queued builds
  • Investigating
    Investigating
    Builds are currently in a queueing state. We're lookinig into it

Aug 2024

We're seeing degraded performance on our Asia proxy
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We've restarted the impacted boxes and restored service. We are still monitoring
  • Investigating
    Investigating
    We're seeing degraded performance on our Asia proxy
Asia edge proxy (v2) service degradation
  • Resolved
    Resolved

    This is a retroactive incident about severe service degradation that happened between 12:40 PM UTC and 13:10 PM UTC on our edge proxies in south-east Asia. We are conducting an investigation into what happened and will update this incident once it's complete.

Edge Proxy outage
  • Resolved
    Resolved
    This incident has been resolved.
  • Update
    Update

    All proxies are back up in all regions. We are still monitoring as everything is coming back up.

  • Update
    Update

    Europe and US-East proxies are back up

  • Update
    Update

    Southeast Asia (Singapore) is back up

  • Monitoring
    Monitoring

    Most of the traffic appears to be back up, but we're still monitoring.

  • Update
    Update

    Our fix is rolling out and we are seeing recovery for some customers.

  • Identified
    Identified

    Our new edge proxy is experiencing a major outage. We have identified the issue and are working on a fix

Some requests from our edge proxy in Europe may have experienced higher latency than normal
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    Our proxy in Europe continued to process requests but from August 25th, 20:30 to August 26th, 15:00, HTTP latency may have been elevated as we dealt with a sudden increase in traffic from this region.

Builds are queuing up due to a bad rollout on the workers. We're reverting. Builds triggered in the last 10 minutes will start to go out without needing to be retriggered
  • Resolved
    Resolved
    Builds are going out nominally.
  • Monitoring
    Monitoring
    We have restored the build queue and are now monitoring the health of remaining builds.
  • Investigating
    Investigating
    Builds are queuing up due to a bad rollout on the workers. We're reverting. Builds triggered in the last 10 minutes will start to go out without needing to be retriggered

Jul 2024

a single host encountered a hardware failure and is restarting, a small portion of deploys in us-west may be unavailable
  • Resolved
    Resolved

    All applications on the impacted host have resumed.

  • Monitoring
    Monitoring

    The host recovered and applications on it are coming back online. It may take a few minutes for all applications on this box to be available.

  • Investigating
    Investigating
    We are currently investigating this incident.
We're investigating routing failures for deployments using static IPs for egress.
  • Resolved
    Resolved
    An update to our routing control plane incorrectly propagated a bad route for regional egress proxies, this led to the drop of static IP egress traffic from 21:28 UTC to approximately 22:00 UTC. Our static IP monitoring was configured globally in our largest region. We're adding regional static IP monitoring to catch this in staged rollouts (aka, much quicker).
  • Monitoring
    Monitoring
    We've reverted the routing config and static IPs traffic is flowing normally.
  • Identified
    Identified
    we've identified a configuration change in our network routing config that looks to have caused the problem. We're rolling out a fix globally.
  • Investigating
    Investigating
    We're investigating routing failures for deployments using static IPs for egress.
Elevated image pull times for GitHub Container Registry
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We are experiencing elevated pull times for images hosted in GitHub Container Registry (GHCR). This appears to be related to a third-party incident on GHCR's CDN (https://www.fastlystatus.com/incident/376717).

We're investigating a single host failure in the us-east4 region, some deployments and volumes in the region may be inaccessible.
  • Resolved
    Resolved

    This incident has been resolved. We found a bug in our scheduling algorithm which resulted in an instance in the US East region to schedule replica workloads on non-unique hosts, resulting in over subscription. To mitigate this, we have provisioned extra capacity in our non-primary regions (US East, Europe, and Southeast Asia)

    Additionally, we will be digging into the scheduling issue to prevent this in the future.

  • Monitoring
    Monitoring

    The impacted host has been successfully restarted and all affected workloads are back online. We are investigating the root failure cause and monitoring the host closely.

  • Identified
    Identified

    We've begun a failover process to recover the impacted host.

  • Investigating
    Investigating
    We're investigating a single host failure in the us-east4 region, some deployments and volumes in the region may be inaccessible.
Some requests from v2 Proxy in US-West and Asia returning incorrectly encoded responses
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Impacted regions have been rolled back and responses should be correct now.

  • Identified
    Identified

    The recent change is reverting.

Jul 2024 to Sep 2024

Next