Intermittent throughput collapse to Singapore from us-west2 replica (US destinations unaffected)

Question

## Summary

Our `us-west2` replica suffers repeated throughput collapse to Singapore: ~27 MB/s drops to as low as 36 KB/s for 30-90s, recovers, and recurs continuously. US-bound traffic from the same container is unaffected throughout. Two unrelated Singapore destinations are hit, and the only thing they share is transit via NTT (AS2914) San Jose -> Tokyo. No packet loss is detectable and RTT stays flat: the path is clean at probe rates and collapses only under load.

## Identifiers

Measurements are from staging, which we moved to us-west2 to reproduce in isolation.

## Only NTT-transiting traffic is affected

```
Destination                 Path                                       Result
Cloudflare (US)             Telia -> Cloudflare, Palo Alto             22-133 MB/s, always
Linode Singapore (AS63949)  Telia -> NTT (SJC->Tokyo) -> Akamai -> SG  collapses
DynamoDB ap-southeast-1     Telia -> NTT (SJC->Tokyo) -> AWS           collapses
```

The two affected destinations are unrelated networks that diverge after Tokyo. The healthy one never enters NTT.

## Throughput from inside the container (UTC, 2026-09-22)

```
            Singapore     US control
17:31:52     25.1  MB/s    124.1 MB/s
17:32:02      0.41 MB/s    127.4 MB/s
17:32:50      0.11 MB/s    133.2 MB/s
17:33:39     27.1  MB/s     83.7 MB/s
17:54:36      0.036 MB/s        -
```

Singapore: `speedtest.singapore.linode.com` (139.162.23.4), 100 MB. US: `speed.cloudflare.com`, 10 MB.

Across 17:31-18:00 the Singapore path oscillated between ~27 MB/s and under 1 MB/s roughly every 1-2 minutes. The US control never fell below 22 MB/s, including during every Singapore collapse.

## The path is clean at probe rates

mtr to the Singapore endpoint, captured during the 36 KB/s sample at 17:54:36:

```
 5. AS1299   railwaycorp-ic-396941.ip.twelve99-cust.net   0.0%    0.2 ms
 8. AS2914   ae-1.r27.snjsca04.us.bb.gin.ntt.net          0.0%    1.2 ms
 9. AS2914   ae-11.r34.tokyjp05.jp.bb.gin.ntt.net        30.0%  107.3 ms
10. AS2914   ae-0.a01.tokyjp09.jp.bb.gin.ntt.net          0.0%  108.0 ms
12. AS20940  ae3.r21.tyo01.mag.netarch.akamai.com         0.0%  104.4 ms
17. AS20940  ae23.gw1-sin1.netarch.akamai.com             0.0%  180.9 ms
21. AS63949  speedtest.singapore.linode.com               0.0%  184.4 ms
```

Five traces captured during five separate degraded windows are identical in this respect: 0.0% loss to the destination, RTT stable at 184.4 ms (StDev 0.1). We read hop 9's 30% as ICMP rate-limiting, since every hop downstream including the endpoint is clean; we are not claiming that hop is at fault.

So at low probe rates the path is perfect while bulk transfers over the same path collapse. That is consistent with a bandwidth limit that ICMP never triggers, and inconsistent with a lossy circuit.

## Impact

Our backend reads DynamoDB in ap-southeast-1. Reads needing several sequential round trips span the degraded windows and are severed mid-response:

```
Error: aborted
  at TLSSocket.socketCloseListener (node:_http_client:464:19)
  code: 'ECONNRESET', '$metadata': { attempts: 3, totalRetryDelay: 84 }
```

All three SDK retries fail. The socket dies after response headers arrive but before the body completes. This is causing user-facing failures in production.

## Questions

1. Is the us-west2 -> NTT (AS2914) trans-Pacific path capacity-constrained or subject to a committed-rate policer? A utilisation graph for that interconnect covering 2026-09-22 17:30-18:00 UTC maybe?
2. We saw the route through Japan shift between Osaka (osakjp02/03) and Tokyo (tokyjp05/09) across runs two minutes apart. ECMP, or route churn?

## Already ruled out

- **Our container** - US transfers from it run 22-133 MB/s during every collapse
- **AWS-specific** - a non-AWS Singapore destination degrades identically
- **Packet loss** - 0.0% to destination in five traces during active degradation
- **Latency / DNS** - RTT flat at 184.4 ms; DNS resolution ~44 ms