Healthcheck consistently fails even though the app is healthy (requests never reach the container)

xandomm

HOBBYOP

a month ago

Across multiple consecutive deploys with the same code, the healthcheck fails on all 8 attempts within the retry window (2 minutes), returning "service unavailable" — even though the application starts up normally.

Deploy Log confirms the app is healthy:

Operations to perform:

Apply all migrations: account, admin, architect, auth, contenttypes, mfa, project, sessions, sites

Running migrations:

No migrations to apply.

[INFO] Starting gunicorn 26.0.0

[INFO] Listening at: http://0.0.0.0:8080 (1)

[INFO] Using worker: gthread

[INFO] Booting worker with pid: 3

[INFO] Booting worker with pid: 4

Gunicorn correctly binds to the port provided via $PORT (8080 in this case).

What I've already ruled out on the application side:

Custom Start Command in the dashboard (already cleared, confirmed the Dockerfile's ENTRYPOINT is being used)

ALLOWED_HOSTS/Django DisallowedHost: I temporarily re-enabled logging for this specific exception to diagnose this — no log line at all appears in the Deploy Log during the 8 healthcheck attempts, neither success nor error. This indicates the healthcheck prober's requests are not reaching the gunicorn process at all.

TLS certificate: valid and working (www.acervo.arq.br, Let's Encrypt, confirmed via curl -v)

Environment variables: all set correctly (DATABASE_URL via reference to the Postgres service in the same project, SECRET_KEY, ALLOWED_HOSTS, etc.)

Question for support: since the application is confirmed to be up and listening on the correct port, but no healthcheck request ever reaches it (not even a logged rejection), I suspect a routing/networking issue between the healthcheck prober and the container in this environment/region. Could you please check on your end?

$10 Bounty

2 Replies

Railway

BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway • about 1 month ago

ve-jo

HOBBY

a month ago

The available evidence does not yet prove that the Railway healthcheck request is failing before reaching the container.

There are a few important points that still need to be verified.

Testing the custom domain does not test the candidate deployment.

Railway only switches public traffic to a new deployment after its configured healthcheck returns HTTP 200. Until then, the custom domain normally continues serving the previous active deployment. Therefore, the valid TLS certificate and a successful request to www.acervo.arq.br do not confirm connectivity to the deployment currently failing its healthcheck.

ALLOWED_HOSTS.

Railway sends deployment healthchecks using the hostname: healthcheck.railway.app

That exact hostname should be present in Django's ALLOWED_HOSTS:

ALLOWED_HOSTS = [
    "acervo.arq.br",
    "www.acervo.arq.br",
    "healthcheck.railway.app",
]

The absence of a DisallowedHost log is not sufficient to conclude that no request arrived. Django's host validation occurs when request.get_host() is evaluated, and the behaviour can depend on the middleware and code path used by the health endpoint.

I would first reproduce the Railway request locally:

curl -i \
  -H 'Host: healthcheck.railway.app' \
  http://127.0.0.1:8080/ACTUAL_HEALTHCHECK_PATH

It must return a direct HTTP 200. Check for common differences such as /health versus /health/; a redirect, authentication response, or 400 is not a successful Railway healthcheck.

I would also enable Gunicorn access logs for one deployment.

ve-jo

HOBBY

a month ago

The following settings should also be checked together:

the container receives PORT=8080;
Gunicorn listens on that same $PORT;
the Railway healthcheck path exactly matches the Django route;
the endpoint returns 200 without authentication, redirects, CSRF checks, or external dependencies;
any manually configured Networking Target Port also equals 8080.

If all of the following are true:

the simulated request with Host: healthcheck.railway.app returns 200;
the path and healthcheck port are confirmed;
Gunicorn access logging is enabled;
dual-stack binding has been tested;
no RailwayHealthCheck/1.0 request appears during any attempt;

then the infrastructure/routing hypothesis becomes credible.

At that point Railway would need the failed deployment IDs, environment ID, service ID, region, and exact timestamps to inspect the healthcheck proxy's internal routing.

Welcome!