Production outage, ~18:08–23:13 UTC on 2026-08-10, now resolved by redeploy. Reporting for root-cause confirmation and prevention.

Identifiers

Project: 7326f4d5-3f2f-44fc-be55-d9ae934d1821

Environment: production (70e8e0ae-1dda-4558-97c6-c34b7bb49790)

Service: sayora-voice (region EU West)

Affected deployment: 5c1feab8-14c1-4840-b844-7a9ef5725d14 (live since 2026-08-09 19:53 UTC, healthy for ~22h before the fault)

Recovery deployment: 71dddea2-d779-4326-868d-9b7b6bdb9067 (2026-08-10 23:13 UTC)

What happened

The container running deployment 5c1feab8 lost all outbound IPv4 connectivity mid-life, while inbound HTTP and internal DNS kept working, so the service showed Online throughout. Every outbound call from our app (Supabase, Twilio, Mixpanel) failed at TCP connect for roughly 4–5 hours.

Evidence, captured live from inside the affected container via railway ssh before we replaced it:

Raw TCP connects (bash /dev/tcp, 4s timeout) to every external IPv4 destination timed out: 1.1.1.1:443, 8.8.8.8:443, 140.82.121.6:443 (GitHub), 104.18.38.10:443 / 172.64.149.246:443 (Supabase via Cloudflare), 34.96.125.79:443 (Mixpanel/GCP), 18.239.68.8:443 (Twilio/CloudFront).

Node fetch from the same container: UND_ERR_CONNECT_TIMEOUT (attempted addresses: 104.18.38.10:443, 172.64.149.246:443, timeout: 10000ms).

DNS worked throughout via the internal resolver (nameserver fd12::10), and inbound requests served normally.

Our process was healthy: next-server at 25 open fds, 2 sockets in the container's /proc/net/tcp (1 LISTEN, 1 TIME_WAIT).

Host-global counters visible from the container (/proc/net/sockstat): TCP: inuse 0 orphan 1 tw 1 alloc 8817 mem 121205, i.e. ~8.8k TCP sockets allocated host-wide holding ~473MB of kernel TCP buffer memory. This looks like host TCP memory pressure or a broken NAT/conntrack path rather than anything in our workload.

All destination endpoints were confirmed healthy and fast from the public internet at the same time.

Timeline (UTC, 2026-08-10)

18:08 — last confirmed successful outbound request from the container

22:36 — first observed errors in service logs (100% outbound failures from then on; onset was somewhere between these two)

23:06 — redeploy attempt; new container's networking was fine (its pre-deploy step reached our database), deploy failed for an unrelated internal reason

23:13 — fresh deploy succeeded; outbound connectivity normal from the new container (Supabase reachable in 50ms). The old deployment was retroactively marked CRASHED.

Questions

Can you confirm from your side what happened to the host under deployment 5c1feab8 in that window (and the precise onset time from host/network metrics)?

Do you detect and auto-cordon hosts that lose egress like this? The service showed Online the whole time because inbound kept working, so nothing on the platform surfaced the fault.

Is there anything you recommend on our side to detect or mitigate this class of failure faster, beyond external uptime monitoring on our own health endpoint (already in progress)?