Summary

Since ~2026-07-01 06:30 UTC, a significant fraction of requests to our service stall at the edge layer for 5-25 seconds, while our upstream consistently responds in under 100ms. In our latest sampling (2026-07-06 ~03:03 UTC), 5 out of 12 identical requests took 8.9-22.8s; the rest completed in ~0.5s. All affected requests route through the hnd1 (Tokyo) edge per x-hikari-trace. This is breaking our mobile app, which uses a 10s client timeout.

Environment

• Service: doko-api (project doko, environment production)
• Service ID: 83c53ea5-8dce-412e-8e3e-71dd5926027b
• Region: asia-southeast1-eqsg3a, 1 replica
• Custom domain: api.doko.kr
• Clients are in Japan/Korea, so requests enter via the Tokyo (hnd1) edge

Evidence 1 — edge vs upstream timing (from Railway's own logs)

Same request, cross-referenced between our deploy logs (app-side) and HTTP logs (edge-side), deployment bf398dfa-3fab-439f-972f-10a471dd3633:

| Time (UTC) | Path | App responded in | Edge totalDuration |

|---|---|---|---|

| 2026-07-06 01:58:58 | GET /collections/2f60a15e-.../public | 28ms | 9,247ms |

| 2026-07-06 02:02:21 | same path | 30ms | 10,005ms (client aborted at its 10s deadline) |

| 2026-07-06 02:01:58 | GET /posts/feed | 81ms | 9,301ms |

upstreamErrors is empty on all of these; all returned HTTP 200. Service is idle (CPU ~2%, memory ~200MB).

Notably, even GET /i 302 redirects with no response body took 6.5-13.1s at the edge on 2026-07-05 ~14:51 UTC — so this is not payload-size or bandwidth related.

Evidence 2 — reproducible from outside with curl

Fetching the same public endpoint repeatedly (GET https://api.doko.kr/collections/2f60a15e-ebc0-476a-9221-03f3e151ca4a/public, ~324KB gzip / ~424KB identity) from a wired connection in Korea, captured 2026-07-06 ~03:03 UTC:

| time_total | x-hikari-trace | x-railway-request-id |

|---|---|---|

| 22.76s | hnd1.sx1d | UDozNSzOQOGHq4TcVOLIQQ |

| 20.51s | hnd1.sx1d | jEfmLdWsRqesVuqtJH0Vcg |

| 17.96s | hnd1.y5sc | woQqknPLTgG0QJIHmrpb1w |

| 17.62s | hnd1.2zrd | bQoGrmyYTqm4Y1qADcO5xA |

| 8.94s | hnd1.df23 | kS2HZWOoRr6pzS2dV7rehQ |

| 0.45-0.75s | (same instances on other tries) | — |

Additional observations:

• Both Accept-Encoding: gzip and identity are affected equally.
• TTFB is also elevated on slow requests (2-3s), then the body trickles for the remainder.
• The same x-hikari-trace instance can serve one request in 0.5s and the next in 20s, so this looks like an intermittent per-connection/path stall at or behind the hnd1 edge, not a single bad instance.
• Requests served via hnd1.4cm0 were consistently fast in our samples.

Evidence 3 — onset timeline (from HTTP logs, deployment c8f8fea9-4025-484f-b4a1-95f20f598ae8)

• 2026-06-30 23:01-23:56 UTC: 500 consecutive requests, max totalDuration 536ms — fully healthy
• 2026-07-01 06:33 UTC: first clear burst (/i requests at 3.2-9.8s)
• 2026-07-01 onward: every sampled window shows 1-7.4% of requests exceeding 3s, p99 of 5-26s
• Ongoing as of 2026-07-06

We also saw brief similar episodes on 06-18 ~08:35 UTC, 06-21 01:35-01:42 UTC, and 06-22 ~10:59 UTC (client-side error bursts in Sentry), but the sustained degradation started 07-01.

What we ruled out

• App: upstream duration is consistently <100ms in Railway's own logs; the service is idle; identical behavior across paths, including bodyless 302s.
• Client/network: reproducible with curl from multiple networks (our users' mobile devices in Japan, plus wired connections in Korea); affects multiple UAs (CFNetwork/iOS, Android, curl, node).
• Encoding/payload: gzip and identity equally affected; 302s with no body also stall.

Impact

Our production mobile app (users in Japan) uses a 10s request timeout, so every stalled request surfaces as a timeout/hang to end users. This has been degrading the app continuously for 5+ days.

Ask

Could you investigate the hnd1 edge → asia-southeast1 path for whatever changed around 2026-07-01 06:30 UTC? The request IDs in Evidence 2 are from stalled requests and should be directly traceable. Happy to run any further diagnostics you need.