Repeated healthcheck failures on newly provisioned containers while existing deployment remains healthy
digoncalves14-hue
FREEOP

a day ago

Summary:

We've experienced 4 consecutive deployment failures for the same application, all failing during the Healthcheck stage approximately 15–16 seconds after the new container starts, while the currently active deployment continues to serve traffic and access the same database normally.

What we observed:

  • Service: barton/api
  • Stack: Node.js / NestJS
  • Dockerfile-based deployment
  • Railway region: US East (Virginia)
  • Application port: 3001
  • Healthcheck path: /health
  • Database: PostgreSQL on Supabase, region sa-east-1, accessed through Supavisor Session Pooler

We had 4 consecutive deployment attempts over approximately 1.5 hours.

All 4 followed the same pattern:

  1. Build succeeded.
  2. Deploy/container startup succeeded.
  3. Application logs showed "Nest application successfully started".
  4. Shortly after startup, the new container began logging repeated "unhandled_exception" / "request errored" events.
  5. A background worker that performs a lightweight database operation every 5 seconds repeatedly logged "messaging_worker_claim_failed".
  6. Approximately 15–16 seconds later, Railway marked the deployment as failed at Network > Healthcheck.
  7. Railway correctly kept the previous deployment ACTIVE, so production remained available throughout.

The key comparison is that the currently ACTIVE container continues to:

  • return HTTP 200 from /health;
  • report database.status = ok;
  • serve production traffic normally;
  • communicate with the same Supabase database.

Meanwhile, newly provisioned containers consistently fail healthcheck and also show repeated failures from the background worker when it attempts database-related work.

We also performed a control redeploy:

  • same commit;
  • same original environment configuration;
  • no code changes;
  • no database changes;
  • no pooler changes;
  • no region changes.

That control deployment failed in the same way.

Because the control redeploy used the same commit and configuration as the currently healthy deployment, we could not identify an application-level configuration difference that explains why the existing container remains healthy while newly provisioned containers fail.

We also checked for a PORT/binding mismatch (a known cause of similar issues on Railway) and confirmed our app listens on 0.0.0.0:3001, matching both the Dockerfile EXPOSE and the Public Networking target port shown in Railway's own Settings — so this does not appear to be a port/host configuration issue on our side.

Could you please investigate whether there was an issue with:

  • networking or egress from newly provisioned containers;
  • container provisioning;
  • connectivity between new Railway containers in US East and our external PostgreSQL endpoint;
  • healthcheck networking/routing;
  • or another platform-level condition during these deployment attempts?

Production never went offline because Railway retained the previous healthy deployment.

Failed deployment IDs:

  1. a7b9e354
  2. 63d06a90
  3. c79c3b79-b91f-4ed0-a1b2-ec703bcc164b
  4. 046c0458-e020-413d-a4e7-7308a0a8306b

We can also provide sanitized startup logs, timestamps, and screenshots of the failed and active deployments.

Please let us know if there are any additional diagnostics we can collect without disrupting the currently healthy production deployment.

$10 Bounty

3 Replies

Railway
BOT

a day ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 1 day ago


Make sure that you have the PORT env variable set to 3001.


texloomstudio-ctrl
PRO

10 hours ago

The PORT/binding angle is already ruled out based on what you've shared, so here's a different theory worth checking: connection pool exhaustion during the deploy overlap window, specifically with Supavisor's Session Pooler.

During a rolling deploy, Railway keeps your old container ACTIVE and serving traffic while the new container boots and goes through its healthcheck — that's exactly why the old one stays healthy throughout. For that window, you have two containers alive simultaneously, both trying to hold connections against the same Supavisor pool.

Session mode (unlike Transaction mode) holds one dedicated backend connection per client for the life of that session. If your existing container is already holding most or all of the pooler's session slots — especially with a worker doing a DB operation every 5 seconds — the new container spinning up alongside it may not be able to acquire its own connections. That would produce exactly what you're seeing: the new container's worker can't claim work (messaging_worker_claim_failed), throws unhandled exceptions, and if your /health check includes a database.status check, the healthcheck fails — while the old container, whose connections were already established before the new one existed, keeps working.

Worth checking:

  1. Your Supavisor pool's configured max connections vs. your app's own DB client pool size (e.g. Prisma/TypeORM connectionLimit) — multiply that by 2 during the overlap window and see if it exceeds the pooler's limit.
  2. Whether your background worker's DB queries could run through the Transaction pooler instead of Session mode — transaction mode multiplexes far more clients over the same backend connections, which would make this kind of overlap non-issue.
  3. As a quick mitigation regardless of root cause: add retry-with-backoff around the worker's claim function instead of letting it hard-fail on the first attempt during startup — that alone might get you through the overlap window without failing the deploy.

If you can check Supavisor's dashboard for connection count during one of these failed deploy windows, that would confirm or rule this out directly.


digoncalves14-hue
FREEOP

3 hours ago

Thanks for this — really helpful angle, and different from anything we'd tested so far.

I checked our Supabase project's Connection Pooling settings (Settings > Database):

  • Connection pool size: 15 (backend connections to Postgres, per user+db)
  • Max client connections: 200

Since we're on Session mode (not Transaction), the number that matters for us is the 15, since each session-mode client holds one of those slots for its whole lifetime.

Our app doesn't set an explicit connection_limit on the Prisma side, so each container defaults to Prisma's formula (num_cpus * 2 + 1) — with 2 vCPU that's ~5 per container. During the deploy overlap window, that would be old container (~5) + new container (~5) = ~10, which is still under the 15 pool size limit on paper. So the simple arithmetic doesn't obviously confirm the theory, though it doesn't rule it out either — we don't have a connection count captured from Supavisor's own side during one of the actual failed windows (only a quiet-traffic snapshot from Postgres' pg_stat_activity taken separately, which isn't the same thing).

If you have thoughts on reconciling that gap (e.g., whether Session mode connections can pile up higher than the nominal connection_limit under real traffic, or another way the two containers together could exceed 15), I'd be glad to hear them. We can also try capturing Supavisor's connection count live during a redeploy attempt if that would help narrow it down further.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...