Railway - We're investigating connection failures between the GCP and Metal interconnect in the us-east4 region – Incident details

All systems operational

We're investigating connection failures between the GCP and Metal interconnect in the us-east4 region

Resolved
Operational
Started 5 days agoLasted about 1 hour

Affected

Deployments

Partial outage from 9:07 AM to 9:59 AM, Degraded performance from 9:59 AM to 12:00 AM

US East (us-east4 / Virginia, USA)

Partial outage from 9:07 AM to 9:59 AM, Degraded performance from 9:59 AM to 12:00 AM

Updates
  • Resolved
    Resolved

    This incident has been resolved. We will be spinning up a secondary BGP Anycast route on metal to provide full dual redundancy, regardless of Google's failures.

  • Monitoring
    Monitoring
    Google Cloud failed over the interconnect, resulting in the primary failing over. The secondary did not have the correct config to propagate and as a result blackholed traffic for Hobby users into Metal US East. We have modified the config and are seeing recovery.
  • Update
    Update

    Redeploying in any other region will resolve the issue for affected users

  • Identified
    Identified

    Metal interconnect seems to have been taken down by Google Cloud. This incident affects any edge traffic to metal instances in us-east

  • Investigating
    Investigating

    We are currently investigating this incident