Railway - Notice history

All systems operational

Dashboard (railway.com) - Operational

100% - uptime
Oct 2023 · 100.0%Nov · 99.98%Dec · 100.0%

Previous page

Next page

Oct 2023
Nov 2023
Dec 2023
100% - uptime

DNS - Operational

100% - uptime
Oct 2023 · 100.0%Nov · 100.0%Dec · 100.0%

Previous page

Next page

Oct 2023
Nov 2023
Dec 2023

Traffic - Operational

100% - uptime
Oct 2023 · 100.0%Nov · 100.0%Dec · 100.0%

Previous page

Next page

Oct 2023
Nov 2023
Dec 2023

Notice history

Dec 2023

A single host in the europe region has encountered a hardware error and is being failed over
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    The affected host is back up and running. All deploys have been restarted.

  • Identified
    Identified

    deployments on the affected hosts are starting to come back up, we expect full resolution within the next 10 minutes.

  • Investigating
    Investigating

    A single host in the europe region has encountered a hardware error - we're failing over that node. A small proportion of services in the europe region may be briefly unavailable.

Elevated request latency and error rates on edge network
  • Resolved
    Resolved

    This incident is now resolved. Between 1:54am UTC and 2:06am UTC we saw a extremely high volume of traffic across our edge proxies in us-east and asia-southeast1 due to a DoS attack - our mitigations restricted this traffic, but a portion of edge requests originating from these regions failed / experienced higher latency during this time.

  • Monitoring
    Monitoring

    Between 1:54am UTC and 2:06am UTC we saw a extremely high volume of traffic across all our edge proxies due to a DoS attack - our mitigations restricted this traffic, but a portion of edge requests failed / had higher latency during this time. Traffic has now recovered, and we're continuing to monitor.

  • Investigating
    Investigating

    Elevated request latency and error rates on edge network

Observed latency spikes on dashboard. Looking into it
  • Resolved
    Resolved

    This incident has been resolved.

  • Update
    Update

    The issue seems to have been abated. During our discovery, we saw timeouts to GitHub and a few other external services. Given we don't use any egress products other than Cloud NAT from Google, we've escalated this to Google and are awaiting a response

  • Monitoring
    Monitoring

    Moving back to monitoring as we've observing it again

  • Resolved
    Resolved

    We've identified the issue as a bottleneck in our service mesh under heavier-than-average load which causes higher-than-normal latency.

  • Monitoring
    Monitoring

    Recovered. Initial response appears to be a flood of deployments with lots of variables put temporary lag on the database, which propagated to the dashboard. We will monitor as we continue to dig in

  • Investigating
    Investigating

    Observed latency spikes on dashboard. Looking into it

A subset of projects aren't loading in the dashboard
  • Update
    Update

    The issue has been resolved.

  • Resolved
    Resolved

    The issue has been resolved.

  • Monitoring
    Monitoring

    We have pushed a fix and are now monitoring.

  • Investigating
    Investigating

    A subset of projects aren't loading in the dashboard

Deploys failing to go out for some users & environment variables may not render correctly
  • Resolved
    Resolved

    This incident has been resolved.

  • Update
    Update

    We implemented a fix and are currently monitoring the result.

  • Monitoring
    Monitoring

    We have rolled out a fix and are currently monitoring the results.

    Variables that were edited during the incident (within the past 1 hour) are currently being restored into their encrypted state.

Nov 2023

Workloads unavailable on Stacker 50
  • Resolved
    Resolved

    This incident has been resolved.

  • Update
    Update

    The machine is currently back online and serving workloads once again. The Railway team will monitor this machine as workloads fully restore. We appreciate your patience.

  • Monitoring
    Monitoring

    The team is currently restarting the affected machine and bringing on workloads back in a staged manner, some workloads may continue to be unavailable.

  • Identified
    Identified

    Workloads on one machine hit a IO limit causing workloads to be unreachable for around 1 hour or so. We advise users to trigger a re-deployment if they haven't. For affected legacy plugins, migrating to a V2 database should resolve the issue.

A small percentage of provisions related to volumes are failing
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We're seeing a very small spike in provision errors for volumes in the US. Looking into it.

Deployments are slow to go out
  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

  • Resolved
    Resolved

    This incident has been resolved.

Deployments are failing to go out
  • Update
    Update

    This incident has been resolved.

  • Resolved
    Resolved

    We had a spike in deploy requests and a small delay for some deploys as they got backlogged.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

404 Flickers on Dashboard for some users
  • Resolved
    Resolved

    This incident has been resolved.

Oct 2023

Volume and database provisions are failing
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

[US West] A small percentage of builds is failing to go out
  • Resolved
    Resolved

    This incident has been resolved. If you run into "Cannot create code snapshot" error, please issue a new build.

  • Monitoring
    Monitoring

    One of our build hosts in US West is down. A small percentage of builds routed to that builder may be failing with "Cannot create code snapshot" errors.

    We've taken the broken builder out of rotation and is currently monitoring the results. Please try issuing a new build.

Volumes are failing to be created in uswest-1
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    Identified

    We are in the process of rolling out a fix. It will be live in a few minutes.

  • Investigating
    Investigating

    We are currently investigating this incident.

A percentage of deployments are failing to go out
  • Resolved
    Resolved

    When we cycled our regional builder fleet, the machine image they booted with had some outdated config. This had the unintended consequence of pushing all images from regional builders to the us-west1 registry which in turn caused deployments to any other region to fail.

    To ensure this doesn't happen again, we have removed the outdated config from our machine images and will be making the config path safer.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

Railway up is returning 504
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

Oct 2023 to Dec 2023

Next