15 days ago
Description of the issue:
Deploys of douleia-backend-preprod intermittently fail their healthcheck and get torn down, despite the application booting successfully every time. Recurred 6+ times across 2026-08-05 and 2026-08-06, each time taking production down until we manually restarted the last known-good deployment.
We've ruled out, with direct evidence:
- Application/settings failure (added temp log checkpoints through Django startup; all print successfully on failed deploys)
- Database connectivity (checked Postgres's own logs for the failure windows; zero connection attempts logged)
- A hung request inside the app (gunicorn's worker-timeout watchdog would kill/respawn a genuinely stuck worker every 30s; logs show exactly one worker boot across the full failure window, no timeout/kill events at all — the healthcheck request appears to never reach the app process)
- Port mismatch (your own Diagnosis feature suggested this once; domain targetPort already matched what gunicorn bound to, and we additionally pinned PORT=8080 explicitly — a subsequent deploy with that pin still failed identically)
Using your in-dashboard AI support chat to inspect deployment b1d132c7-0df2-4159-9767-e4ed33f3203f directly, it found that deployment's own build logs show "[1/1] Healthcheck succeeded!" — yet the deployment was marked failed and removed. That's your own two data sources (build logs vs. final deployment state) disagreeing with each other, not something we're inferring from limited visibility on our end.
Error messages:
"Attempt #1 failed with service unavailable. Continuing to retry for 4m59s" (repeats 14x) → "1/1 replicas never became healthy! Healthcheck failed!"
Deployment IDs: b1d132c7-0df2-4159-9767-e4ed33f3203f, e5280956-b9f1-4420-b20a-44cbc20550df
Logs: happy to attach full deploy logs for both failed deployments (settings-load checkpoints, gunicorn startup, full healthcheck retry log) — let me know the best way to attach them here.
Repo: private (Django/DRF backend on Railway, gunicorn + Railpack build) — can share specific config on request.
4 Replies
15 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 15 days ago
mayori
Make sure you bind the address to `0.0.0.0`
15 days ago
Thanks for looking into this. To confirm: gunicorn is already bound to 0.0.0.0, not a
narrower interface — this is in the original report, but to make it explicit:
[INFO] Listening at: http://0.0.0.0:8080 (1)
This is the exact log line from the container on every failed deployment. The healthcheck
still fails 14/14 attempts despite this. The evidence that points away from a bind-address
issue specifically: gunicorn's own worker-timeout watchdog never killed/respawned the
worker during any failure window (it would have, roughly every 30s, if a request had
actually reached it and hung) — meaning the healthcheck requests don't appear to reach the
container's network stack at all, not that they arrive and get rejected by a wrong bind
address.
Also worth re-flagging from the original report: deployment b1d132c7's own build logs show
"[1/1] Healthcheck succeeded!", yet that same deployment was marked failed and removed —
your own two data sources disagreeing with each other. That inconsistency seems like the
more promising thread to pull on. Happy to share full deployment logs if that would help.
15 days ago
On the healthcheckTimeout: we did have it at the default 300s for the first six failures
across 2026-08-05 and 2026-08-06 — every one of those ran the full 5-minute window with all
14 attempts failing, and the worker was never hit by a single request throughout (per the
gunicorn watchdog evidence above). Since a longer window produced the same zero-success
outcome every time, we shortened it to 60s specifically to reduce the outage window, not to
risk cutting off a deployment that might have eventually succeeded — we have direct evidence
across multiple full 300s windows that it doesn't. Happy to set it back to 300s temporarily
for a specific test attempt if that would give you better data to look at, just let me know.
15 days ago
Update: we found and fixed the root cause on our side, confirmed on a real deploy just now.
It was our own SECURE_SSL_REDIRECT setting (added in a recent release) — Django was redirecting
your healthcheck prober to https://... instead of returning 200, because that prober appears to
connect directly to the container rather than through the public edge that sets the header Django
uses to detect HTTPS. Added a specific exemption for our healthcheck path, redeployed, and it
passed cleanly (your own build log even showed "[1/1] Healthcheck succeeded!" this time, matching
the actual outcome).
The one thing from this thread I'd still flag as worth your team looking into separately: deployment
b1d132c7-0df2-4159-9767-e4ed33f3203f's build logs showed "Healthcheck succeeded!" while the
deployment itself was marked failed and removed — that inconsistency between your build logs and
final deployment state seems like a real, independent issue on your side, unrelated to our redirect
bug. Happy to share more detail on that specifically if useful, otherwise this thread can be closed
on our end.
Thanks for the help pointing us toward the bind-address/timeout checks along the way.
Status changed to Solved 0x5b62656e5d • 15 days ago