Railway - Elevated latency on Edge Network – Incident details

All systems operational

Elevated latency on Edge Network

Resolved
Operational
Started 15 days agoLasted 18 minutes

Affected

Edge Network (Railway Metal)

Operational from 7:30 AM to 7:48 AM

US West (Metal / California, USA)

Operational from 7:30 AM to 7:48 AM

US East (Metal / Virginia, USA)

Operational from 7:30 AM to 7:48 AM

EU West (Metal / Amsterdam, Netherlands)

Operational from 7:30 AM to 7:48 AM

Southeast Asia (Metal / Singapore)

Operational from 7:30 AM to 7:48 AM

Updates
  • Resolved
    Resolved

    During a routine update to BGP configuration for Railway's Anycast network 66.33.22.0/23, a small proportion (approximately 8%) of upstream ISPs failed to propagate a routing update - this led to a small amount of traffic to be dropped for a period less than 5 mins.

    Once routes propagated, some ISPs were routed in a suboptimal manner leading to elevated latency for some users. We have reverted the routing configuration and are liaising with our vendors to root cause the issue. We will also be creating maintenance windows for future routing config changes and staging their rollout to minimize risk.

  • Monitoring
    Monitoring

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

  • Investigating
    Investigating

    As we're rolling out a routing reconfiguration of our metal edge, a small subset of users may notice latency issues while the new routing configuration is being propagated to all ISPs.