20 hours ago
Subject: Recurring outbound network failures (ETIMEDOUT/ENETUNREACH) in sfo region — now causing full app unresponsiveness, multiple days
Project: bglam-stock-transfer (project ID f93ce332-fd77-4fb6-b7cd-d4494c7186f5)
Service: bglam-stock-transfer (service ID ae1442af-f7f7-48ff-9c0c-0b2c5a3332c2)
Environment: production (726ee7d9-53d4-4720-914c-cdd8589b0379)
Region: sfo
We've had a recurring issue for several days where our service loses outbound network connectivity to all external hosts (not one specific provider) for extended periods. It has occurred on at least Aug 23, Aug 24, and repeatedly today, Aug 25.
Exact error signature (from our own logs):
connect ETIMEDOUT 23.227.38.74:443; connect ENETUNREACH 2620:127:f00f:e:::443 - Local (:::0)
Today's timeline (all UTC):
~10:27 — service booted cleanly after a routine deploy.
~10:27:53 onward — every outbound call (to our e-commerce and ERP integrations) began failing with the error above.
~10:29–10:35 — the fault escalated beyond just outbound calls: ordinary inbound page loads and API requests started taking 60–250+ seconds to complete, and logins began timing out entirely (clients gave up before we could respond).
10:38 — we restarted the service manually, which temporarily restored responsiveness (~9–11s response times).
~10:42 onward — degraded again; users are back to timeouts and multi-minute load times as of this message.
Impact: This is our production system used by staff across multiple physical retail locations for day-to-day operations. Today's incident has gone beyond background sync failures into a full user-facing outage — staff cannot log in or use the app.
What we're asking:
Please investigate outbound connectivity from the sfo region for our service/account around the timestamps above.
Please let us know if this is a known, ongoing regional issue, and if so, an ETA for resolution.
Given this has now recurred on 3+ separate days, we'd like to understand whether moving regions would help, or if this is expected to be resolved where we are.
Happy to provide our own request logs/timestamps in more detail if useful.
2 Replies
20 hours ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 20 hours ago
13 hours ago
Set short and precise timeouts for outbound http clients add a circuit breaker pattern after n consecutive failed outbound calls temporarily disable that integration and put it in the queue move outbound calls to an async background job removing them from the request response loop so that the login inbound api is not affected even if the external service slows down make the health check endpoint independent of outbound dependencies
gofretfkintank
Set short and precise timeouts for outbound http clients add a circuit breaker pattern after n consecutive failed outbound calls temporarily disable that integration and put it in the queue move outbound calls to an async background job removing them from the request response loop so that the login inbound api is not affected even if the external service slows down make the health check endpoint independent of outbound dependencies
2 hours ago
Thanks for the suggestion — outbound timeouts are already in place on our Shopify/Odoo clients (added after yesterday's incident). But today's recurrence shows something that doesn't fit "slow outbound call blocking the request loop": a plain static-file request (GET /, zero outbound dependencies) hung for 100–128s before the client gave up, and it reproduced on a brand-new container within under a minute of redeploying, and independently timed out from a completely separate network path we tested from. That points at something between your edge and the container, or something blocking our process before an outbound call is even in play — a stuck outbound HTTP client can't explain a request that never makes one.
Could someone check the edge/proxy-level timing for these specific windows and confirm whether the request even reached our container, or stalled earlier in your routing layer?
Service: bglam-stock-transfer (project f93ce332-fd77-4fb6-b7cd-d4494c7186f5)
Instance 1 (bc295e30-6470-4e57-a3ac-75d01774c7cc, deployed 03:17 UTC): hangs at 04:11:56–04:17:20 UTC, durations 2.3s–128s, all "client closed request before server could send a response"
Instance 2 (810c7018-5afd-474d-b416-4b2cfa40e4a5, fresh redeploy at 04:18:30 UTC): same pattern, first hang at 04:19:57 UTC, just 34s after boot
CPU and memory were flat/normal throughout on our side both times — no resource exhaustion, no crash
We're separately reviewing whether one of our own boot-time jobs could be blocking our process's event loop briefly, so this isn't meant as "it's definitely your infra" — but the static-file + cross-network evidence means it's not purely an outbound-timeout problem either. Appreciate a look at the routing/edge side for these windows.