Railway - Notice history

All systems operational

Dashboard (railway.com) - Operational

100% - uptime
Oct 2024 · 100.0%Nov · 100.0%Dec · 100.0%

Previous page

Next page

Oct 2024
Nov 2024
Dec 2024
100% - uptime

DNS - Operational

100% - uptime
Oct 2024 · 100.0%Nov · 100.0%Dec · 100.0%

Previous page

Next page

Oct 2024
Nov 2024
Dec 2024

Traffic - Operational

100% - uptime
Oct 2024 · 100.0%Nov · 100.0%Dec · 100.0%

Previous page

Next page

Oct 2024
Nov 2024
Dec 2024

Notice history

Dec 2024

A single host in US East is unresponsive due to load on the box.
  • Resolved
    Resolved

    All applications on the affected host have been restored.

  • Identified
    Identified

    Host is being recovered by the team.

  • Investigating
    Investigating
    We are currently investigating this incident.
Services Outage
  • Resolved
    Resolved

    This incident has been resolved. Railway was down for a total of ~10 minutes. We will publish a full retroactive on this incident shortly.

  • Monitoring
    Monitoring

    Services should be fully restored and we'll continue to monitor.

  • Identified
    Identified

    Issue has been identified.

  • Investigating
    Investigating
    We are currently investigating this incident.
Dashboard Maintenance
  • Completed
    December 16, 2024 at 5:30 AM
    Completed
    December 16, 2024 at 5:30 AM
    Maintenance has completed successfully
  • In progress
    December 16, 2024 at 5:00 AM
    In progress
    December 16, 2024 at 5:00 AM
    Maintenance is now in progress
  • Planned
    December 16, 2024 at 5:00 AM
    Planned
    December 16, 2024 at 5:00 AM

    We will be upgrading our high availability database on Sunday at midnight EST. We do not expect any interruption of service. While we do not expect it, during this time there may be dashboard "blips" and potentially delayed builds.

Some builds relying on GitHub fail with connection error
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified
    We are continuing to work on a fix for this incident.
Builds using NPM will fail during NPM’s scheduled maintenance.
  • Resolved
    Resolved
    NPM's maintenence window is completed.
  • Investigating
    Investigating
    Builds using NPM will fail during NPM’s scheduled maintenance.

Nov 2024

Singular Google Cloud host failing over. Users scheduled on this machine may notice request timeout/retries
  • Resolved
    Resolved
    Resolved
  • Monitoring
    Monitoring
    Host returned to nominal. Monitoring
  • Investigating
    Investigating
    Singular Google Cloud host failing over. Users scheduled on this machine may notice request timeout/retries
US-East Metal builds have stalled. We’re rolling out a fix
  • Resolved
    Resolved
    We've resolved the incident. All builds back to nominal
  • Investigating
    Investigating
    US-East Metal builds have stalled. We’re rolling out a fix
Users on project pages may see inverted 'dark mode'/'light mode' colors for service tiles. We've rolled out a patch which should be live in 5 minutes
  • Resolved
    Resolved
    Resolved
  • Investigating
    Investigating
    Users on project pages may see inverted 'dark mode'/'light mode' colors for service tiles. We've rolled out a patch which should be live in 5 minutes
DDOS attack being mitigated currently. A small subset of traffic may see a degredation
  • Resolved
    Resolved

    Resolved. A degraded serving instance failed to remove itself from the edge rotation. We're deploying a patch to prevent this from occuring again

  • Monitoring
    Monitoring
    We're stabilized the latency and are moving to monitoring
  • Investigating
    Investigating
    DDOS attack being mitigated currently. A small subset of traffic may see a degredation
We're experiencing elevated error rates on our builders. Deploys may fail to go live during this period
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating
    We're experiencing elevated error rates on our builders. Deploys may fail to go live during this period

Oct 2024

We are investigating an issue with the edge network proxy
  • Resolved
    Resolved

    We have resolved this issue.

  • Monitoring
    Monitoring

    A fix has been implemented and we're currently monitoring the edge network proxy.

  • Identified
    Identified

    We have identified the issue as a DDoS attack. We're in the process of applying additional mitigations

  • Investigating
    Investigating

    We are currently investigating this incident.

A host in the us-west region experienced an unexpected restarts. We're recovering the last workloads on this host that failed to restore automatically.
  • Resolved
    Resolved
    The host has fully recovered and all impacted workloads are back online.
  • Investigating
    Investigating
    A host in the us-west region experienced an unexpected restarts. We're recovering the last workloads on this host that failed to restore automatically.
[Retroactive] A single host in US East has restarted
  • Resolved
    Resolved

    (This is a retroactive incident) During this time we noticed a single host in the US East 1 region had restarted and we quickly worked to bring containers back online, users may have not been able to dial their deployments or connect to databases during the restart efforts. This incident has been resolved.

[Retroactive] Two hosts in EU West experienced degraded performance
  • Resolved
    Resolved

    This is a retroactive incident which is resolved. For the period of the incident (11:45-12:35 UTC), some workloads may have exhibited connection problems. The incident was due to resource exhaustion on the two hosts due to unaccounted resource thresholds. We have improved our monitoring to better respond to this type of incident and are working on preventative measures in our orchestration.

We have observed degraded performance of the proxy layer. Users may experience dropped connections. We are investigating.
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We have taken action to mitigate the issue and are monitoring.
  • Investigating
    Investigating

    We have observed degraded performance of the edge proxy layer. Users may experience dropped connections. We are investigating.

Oct 2024 to Dec 2024

Next