Set short and precise timeouts for outbound http clients add a circuit breaker pattern after n consecutive failed outbound calls temporarily disable that integration and put it in the queue move outbound calls to an async background job removing them from the request response loop so that the login inbound api is not affected even if the external service slows down make the health check endpoint independent of outbound dependencies

Thanks for the suggestion — outbound timeouts are already in place on our Shopify/Odoo clients (added after yesterday's incident). But today's recurrence shows something that doesn't fit "slow outbound call blocking the request loop": a plain static-file request (GET /, zero outbound dependencies) hung for 100–128s before the client gave up, and it reproduced on a brand-new container within under a minute of redeploying, and independently timed out from a completely separate network path we tested from. That points at something between your edge and the container, or something blocking our process before an outbound call is even in play — a stuck outbound HTTP client can't explain a request that never makes one.

Could someone check the edge/proxy-level timing for these specific windows and confirm whether the request even reached our container, or stalled earlier in your routing layer?

Service: bglam-stock-transfer (project f93ce332-fd77-4fb6-b7cd-d4494c7186f5)

Instance 1 (bc295e30-6470-4e57-a3ac-75d01774c7cc, deployed 03:17 UTC): hangs at 04:11:56–04:17:20 UTC, durations 2.3s–128s, all "client closed request before server could send a response"

Instance 2 (810c7018-5afd-474d-b416-4b2cfa40e4a5, fresh redeploy at 04:18:30 UTC): same pattern, first hang at 04:19:57 UTC, just 34s after boot

CPU and memory were flat/normal throughout on our side both times — no resource exhaustion, no crash

We're separately reviewing whether one of our own boot-time jobs could be blocking our process's event loop briefly, so this isn't meant as "it's definitely your infra" — but the static-file + cross-network evidence means it's not purely an outbound-timeout problem either. Appreciate a look at the routing/edge side for these windows.

Thanks for the suggestions — those are good general practices and we'll look at adding them regardless. But I want to share what we found today, because it points away from an outbound-call timeout issue specifically.

It just happened again, on a completely fresh container, within minutes of deploying.

We deployed a new release at 06:14 UTC today. Build succeeded, container started cleanly, app logged "listening on http://localhost:8080" at 06:15:46 UTC — a normal, healthy boot. Within about 3 minutes, the exact same failure mode was back: requests hanging until the client gives up, logged as HTTP 499 "client has closed the request before the server could send a response."

This is the second time today we've watched a brand-new instance fail within minutes of a clean boot (the first was a redeploy at 04:18 UTC that failed 34 seconds after starting).

Critically, this is happening on requests that make zero outbound network calls. Two examples from just now:

• GET / — pure static file read from local disk (fs.readFile), no network dependency at all
• GET /api/products/:sku/photo for an unknown SKU — this 404s from a local DB lookup before any network call would even be attempted

Both hung for 30–95+ seconds before the client (curl, from an independent network, not even our own app's browser) gave up. If the problem were outbound HTTP calls not timing out fast enough, these two routes should return instantly regardless — there's nothing for them to wait on.

At the same time, inbound Shopify webhook deliveries to /webhook/shopify/mauritius are also failing with the same 499 pattern — dozens of them in the last few minutes. So this isn't isolated to one route or one kind of request; it looks like the container/networking path itself is failing to complete requests, not any particular endpoint's own logic.

Other things we've already ruled out today:

• Resource exhaustion — CPU and memory stayed normal/near-zero throughout every occurrence
• Client/location-specific issue — reproduced independently from our own network, not just end users'
• One bad container — reproduced on at least two separate fresh containers today after redeploys

Given it reproduces on zero-dependency requests immediately after a clean boot, could someone look at this from the routing/proxy or container-networking side rather than the application layer? Happy to share deployment IDs and timestamps for both incidents if that helps you correlate against infra logs on your end.

Reference details, in case useful:

• Project: bglam-stock-transfer (f93ce332-fd77-4fb6-b7cd-d4494c7186f5), environment: production, service: bglam-stock-transfer (ae1442af-f7f7-48ff-9c0c-0b2c5a3332c2), region: sfo
• Incident 1: redeploy at 2026-08-26 04:18:30 UTC (deployment 34cf7633-7b50-41af-9f5a-b97e22fb6889, instance 810c7018-...) — hung within 34 seconds of boot
• Incident 2 (today, most recent): deploy at 2026-08-26 06:14:56 UTC (deployment c29bbdf8-b6f2-4d3e-9d61-f50f5c02c136, instance 27d8ff9f-9115-4821-ad4a-74aecb46811b) — clean boot log at 06:15:46 UTC, first 499s observed by ~06:18 UTC on GET /, GET /api/products/:sku/photo (zero-dependency), and POST /webhook/shopify/mauritius (inbound Shopify webhooks)

Project: bglam-stock-transfer (f93ce332-fd77-4fb6-b7cd-d4494c7186f5)

Service: bglam-stock-transfer (ae1442af-f7f7-48ff-9c0c-0b2c5a3332c2)

Environment: production (726ee7d9-53d4-4720-914c-cdd8589b0379)

Healthcheck path: /api/health, timeout 300s

We've had repeated deploy failures today (2026-08-28) where the healthcheck reports "service unavailable" even though our application logs confirm it booted successfully and was listening for connections. Example:

Deployment beb8b99e-9bd8-4a8f-9c75-a2a02d575582 — started 10:03:38 UTC. Deploy logs show server.listen() fired at 10:05:54 UTC (app accepting connections), and the process continued running normally (processing background jobs) until Railway sent SIGTERM at 10:09:38 UTC — roughly 4 minutes of confirmed uptime. In that entire window, the HTTP request log for this deployment shows zero entries — not even a failed healthcheck request, just nothing. This suggests the healthcheck prober (or the proxy generally) never actually reached the container.

For contrast, a deployment of an older build on the same service, ff4a69b0-9227-4c8e-a005-0da846690471 (09:30-09:38 UTC), shows normal HTTP traffic being routed and served with 200 OK responses throughout its lifetime — so networking to this service does work under normal conditions.

We've seen this same "app confirmed up, healthcheck still fails, zero HTTP logged" pattern recur across multiple separate deployments today (793e5f18, 31a32bdd, e9d6c63b, d724dce2, 4836002f, 3e7c56d5, beb8b99e), spanning more than 5 hours and multiple different code builds, causing extended production downtime for our team. Could you check on your side whether there was a routing/proxy-layer incident affecting this service today, and why the healthcheck prober appears not to be reaching the container on these failed deploys?

Happy to provide any additional deployment IDs or timestamps you need.