Railway - Notice history

All systems operational

Dashboard - Operational

100% - uptime
Sep 2024 · 100%Oct · 99.68%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024
100% - uptime

Builders - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

Image Registry - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

Third-Party: npm Registry - Operational

Third-Party: Github Integrations - Operational

Third Party: Docker Hub - Operational

Third-Party: GitHub Container Registry - Operational

100% - uptime

EU West (europe-west4 / Amsterdam, Netherlands) - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 99.98%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

US East (us-east4 / Virginia, USA) - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

US West (us-west1 / Oregon, USA) - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 99.99%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

Southeast Asia (asia-southeast1 / Singapore) - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024
100% - uptime

US West (us-west1 / Oregon, USA) - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

US East (us-east4 / Virginia, USA) - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

EU West (europe-west4 / Amsterdam, Netherlands) - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

Southeast Asia (asia-southeast1 / Singapore) - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

TCP Proxy - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024
100% - uptime

US West (us-west1 / Oregon, USA) - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

US East (us-east4 / Virginia, USA) - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

EU West (europe-west4 / Amsterdam, Netherlands) - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

Southeast Asia (asia-southeast1 / Singapore) - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024
100% - uptime

DNS - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

Traffic - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024
100% - uptime

Logs - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

Metrics - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024
100% - uptime

Payments - Operational

100% - uptime
Sep 2024 · 100%Oct · 100.0%Nov · 100.0%

Previous page

Next page

Sep 2024
Oct 2024
Nov 2024

Notice history

Nov 2024

Some projects seeing 404s. We've identified the issue and rolling back the bad commit
  • Resolved
    Resolved
    Resolved
  • Identified
    Identified
    Rollback should be live in 2 minutes
  • Investigating
    Investigating
    Some projects seeing 404s. We've identified the issue and rolling back the bad commit
Elevated error rates on edge proxy in asia-southeast1
  • Resolved
    Resolved

    This incident has been resolved. We successfully mitigated a DDoS attack originating from our asia-southeast1 region. Users may have experienced difficulties connecting to Railway-hosted workloads in Asia between 1122UTC~1149UTC.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating

    We are currently investigating this incident. Users connecting to Railway-hosted workloads from Asia may experience increased latency during this period.

A single host in US West is experiencing increased latency due to load on the box.
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating
    We are currently investigating this incident.
A single host in US West has restarted
  • Resolved
    Resolved

    All applications on the host are running.

  • Identified
    Identified

    The host is online and applications are resuming.

  • Investigating
    Investigating

    We are currently investigating this incident.

A single host in US West has restarted
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    All applications on this host are back on online.

Oct 2024

We are investigating an issue with the edge network proxy
  • Resolved
    Resolved

    We have resolved this issue.

  • Monitoring
    Monitoring

    A fix has been implemented and we're currently monitoring the edge network proxy.

  • Identified
    Identified

    We have identified the issue as a DDoS attack. We're in the process of applying additional mitigations

  • Investigating
    Investigating

    We are currently investigating this incident.

A host in the us-west region experienced an unexpected restarts. We're recovering the last workloads on this host that failed to restore automatically.
  • Resolved
    Resolved
    The host has fully recovered and all impacted workloads are back online.
  • Investigating
    Investigating
    A host in the us-west region experienced an unexpected restarts. We're recovering the last workloads on this host that failed to restore automatically.
[Retroactive] A single host in US East has restarted
  • Resolved
    Resolved

    (This is a retroactive incident) During this time we noticed a single host in the US East 1 region had restarted and we quickly worked to bring containers back online, users may have not been able to dial their deployments or connect to databases during the restart efforts. This incident has been resolved.

[Retroactive] Two hosts in EU West experienced degraded performance
  • Resolved
    Resolved

    This is a retroactive incident which is resolved. For the period of the incident (11:45-12:35 UTC), some workloads may have exhibited connection problems. The incident was due to resource exhaustion on the two hosts due to unaccounted resource thresholds. We have improved our monitoring to better respond to this type of incident and are working on preventative measures in our orchestration.

We have observed degraded performance of the proxy layer. Users may experience dropped connections. We are investigating.
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We have taken action to mitigate the issue and are monitoring.
  • Investigating
    Investigating

    We have observed degraded performance of the edge proxy layer. Users may experience dropped connections. We are investigating.

Sep 2024

[External] Dockerhub Image pull returns 401 Unauthorized
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    We are currently investigating this incident. For more info see Docker's status page: https://www.dockerstatus.com/

Failing to create volumes
  • Resolved
    Resolved

    We have traced the issue to a guard that was trying to prevent users to deploying volumes on Railway Metal. We have since applied the fix and systems are working normally.

  • Investigating
    Investigating
    Failing to create volumes
Dockerhub is currently undergoing maintenance.
  • Resolved
    Resolved

    Dockerhub has completed their maintenance

  • Identified
    Identified

    During this period, builds might fail, displaying a "401 Unauthorized" message in the logs.

    Please check Dockerhub's status page for more information.

    https://www.dockerstatus.com/

Edge Network Hardware Upgrades
  • Completed
    September 19, 2024 at 9:08 PM
    Completed
    September 19, 2024 at 9:08 PM

    We have successfully upgraded our edge network hardware. You should not experience any further connection disconnects.

    Our monitors during this time show no degradation to TCP connections or HTTP requests failures while we performed this across our fleet but as mentioned, closing your connection to shutdown the machine was unavoidable. Your clients would reconnect and continue to work as expected.

  • Update
    September 19, 2024 at 8:17 PM
    In progress
    September 19, 2024 at 8:17 PM

    We are upgrading our final set of machines ahead of schedule.

  • In progress
    September 19, 2024 at 6:43 PM
    In progress
    September 19, 2024 at 6:43 PM

    This upgrade is in progress.

  • Planned
    September 19, 2024 at 6:00 PM
    Planned
    September 19, 2024 at 6:00 PM

    We are upgrading the hardware for our edge network that powers TCP and HTTP proxying. During this time requests to boxes being upgraded will have 5 minutes to complete before being disconnected. These clients will be able to reconnect and will be routed to another box that will handle the traffic indefinitely.

Builds are currently in a queueing state. We're lookinig into it
  • Resolved
    Resolved
    Queue back to zero
  • Monitoring
    Monitoring
    Queued backpressure is almost drained. Most queued deploys have been rolled out
  • Identified
    Identified
    Deploys are going out and the queue is draining. We will move to monitoring when there are no queued builds
  • Investigating
    Investigating
    Builds are currently in a queueing state. We're lookinig into it

Sep 2024 to Nov 2024

Next