Network

How we cut anycast failover to under a second

When an edge location fails, BGP alone can take tens of seconds to move traffic. Here is how we combined BFD, pre-computed withdrawals and health-driven route maps to get it under one second.

Anycast is wonderful until a location fails. In theory, routers stop announcing the prefix and traffic flows to the next-nearest site. In practice, BGP convergence on the wider internet can take anywhere from a few seconds to a minute, and visitors notice every one of them.

Where the time goes

We instrumented a series of planned failovers and broke the delay into three parts: detecting the failure, withdrawing the route, and the rest of the internet noticing. Detection was the surprise: our health checks ran every five seconds and needed three failures before acting.

Faster detection with BFD

Bidirectional Forwarding Detection runs between our edge routers and the servers behind them at 50 ms intervals. When a server stops answering, the router withdraws its route locally in about 150 ms, long before our application-level checks would have noticed.

Pre-computed withdrawals

Instead of reacting by reconfiguring routers, every location already has the alternative paths installed with lower preference. A failure only removes the preferred path; nothing has to be computed under pressure.

Results

The long tail is now dominated by a handful of networks that dampen route changes. We are working with them directly.

Questions about this post? Write to the engineering team through our contact page.