Deploy healthcheck fails intermittently; build logs show "Healthcheck succeeded" but deployment marked failed
trev
HOBBYOP

15 days ago

Description of the issue:

Deploys of douleia-backend-preprod intermittently fail their healthcheck and get torn down, despite the application booting successfully every time. Recurred 6+ times across 2026-08-05 and 2026-08-06, each time taking production down until we manually restarted the last known-good deployment.

We've ruled out, with direct evidence:

  • Application/settings failure (added temp log checkpoints through Django startup; all print successfully on failed deploys)
  • Database connectivity (checked Postgres's own logs for the failure windows; zero connection attempts logged)
  • A hung request inside the app (gunicorn's worker-timeout watchdog would kill/respawn a genuinely stuck worker every 30s; logs show exactly one worker boot across the full failure window, no timeout/kill events at all — the healthcheck request appears to never reach the app process)
  • Port mismatch (your own Diagnosis feature suggested this once; domain targetPort already matched what gunicorn bound to, and we additionally pinned PORT=8080 explicitly — a subsequent deploy with that pin still failed identically)

Using your in-dashboard AI support chat to inspect deployment b1d132c7-0df2-4159-9767-e4ed33f3203f directly, it found that deployment's own build logs show "[1/1] Healthcheck succeeded!" — yet the deployment was marked failed and removed. That's your own two data sources (build logs vs. final deployment state) disagreeing with each other, not something we're inferring from limited visibility on our end.

Error messages:

"Attempt #1 failed with service unavailable. Continuing to retry for 4m59s" (repeats 14x) → "1/1 replicas never became healthy! Healthcheck failed!"

Deployment IDs: b1d132c7-0df2-4159-9767-e4ed33f3203f, e5280956-b9f1-4420-b20a-44cbc20550df

Logs: happy to attach full deploy logs for both failed deployments (settings-load checkpoints, gunicorn startup, full healthcheck retry log) — let me know the best way to attach them here.

Repo: private (Django/DRF backend on Railway, gunicorn + Railpack build) — can share specific config on request.

Solved$10 Bounty

4 Replies

Railway
BOT

15 days ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 15 days ago


15 days ago

Make sure you bind the address to 0.0.0.0


mayori

Make sure you bind the address to `0.0.0.0`

trev
HOBBYOP

15 days ago

Thanks for looking into this. To confirm: gunicorn is already bound to 0.0.0.0, not a

narrower interface — this is in the original report, but to make it explicit:

[INFO] Listening at: http://0.0.0.0:8080 (1)

This is the exact log line from the container on every failed deployment. The healthcheck

still fails 14/14 attempts despite this. The evidence that points away from a bind-address

issue specifically: gunicorn's own worker-timeout watchdog never killed/respawned the

worker during any failure window (it would have, roughly every 30s, if a request had

actually reached it and hung) — meaning the healthcheck requests don't appear to reach the

container's network stack at all, not that they arrive and get rejected by a wrong bind

address.

Also worth re-flagging from the original report: deployment b1d132c7's own build logs show

"[1/1] Healthcheck succeeded!", yet that same deployment was marked failed and removed —

your own two data sources disagreeing with each other. That inconsistency seems like the

more promising thread to pull on. Happy to share full deployment logs if that would help.


trev
HOBBYOP

15 days ago

On the healthcheckTimeout: we did have it at the default 300s for the first six failures

across 2026-08-05 and 2026-08-06 — every one of those ran the full 5-minute window with all

14 attempts failing, and the worker was never hit by a single request throughout (per the

gunicorn watchdog evidence above). Since a longer window produced the same zero-success

outcome every time, we shortened it to 60s specifically to reduce the outage window, not to

risk cutting off a deployment that might have eventually succeeded — we have direct evidence

across multiple full 300s windows that it doesn't. Happy to set it back to 300s temporarily

for a specific test attempt if that would give you better data to look at, just let me know.


trev
HOBBYOP

15 days ago

Update: we found and fixed the root cause on our side, confirmed on a real deploy just now.

It was our own SECURE_SSL_REDIRECT setting (added in a recent release) — Django was redirecting

your healthcheck prober to https://... instead of returning 200, because that prober appears to

connect directly to the container rather than through the public edge that sets the header Django

uses to detect HTTPS. Added a specific exemption for our healthcheck path, redeployed, and it

passed cleanly (your own build log even showed "[1/1] Healthcheck succeeded!" this time, matching

the actual outcome).

The one thing from this thread I'd still flag as worth your team looking into separately: deployment

b1d132c7-0df2-4159-9767-e4ed33f3203f's build logs showed "Healthcheck succeeded!" while the

deployment itself was marked failed and removed — that inconsistency between your build logs and

final deployment state seems like a real, independent issue on your side, unrelated to our redirect

bug. Happy to share more detail on that specifically if useful, otherwise this thread can be closed

on our end.

Thanks for the help pointing us toward the bind-address/timeout checks along the way.


Status changed to Solved 0x5b62656e5d 15 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...