2 months ago
Support ticket text — paste as-is into station.railway.com "Get Help!"
Subject: Production service failing healthcheck since 03:23 UTC today, 21+ consecutive failed deploys, one exception on redeploy
Project: happy-luck (652c4485-1734-4fc0-9ebb-d7ea6ae28ab5)
Service: kioku (85d0a1b9-d2a4-425a-85a5-fd0a0c1520d0)
Environment: production (548c3b9d-352d-486c-b602-7e02c78a0247)
Region: us-west2, Runtime V2
Our production service (kioku) has been failing its healthcheck on almost every deploy since 2026-08-14 03:23:25 UTC (deployment f1771147-cbce-446c-8d6b-c3794c7156b2, commit 454dd0485e6a0d1cb8f8d125932308e3ece2c29c). Last known good deploy before that: commit 204d67d399c608e331d91ab040bd99e36699b9e7, live since 2026-08-13 23:59 UTC.
Symptom (identical on every failure): app boots cleanly — DB connects, Redis connects, all cron jobs register, server logs "listening on port 3000" — then runs for several minutes with the http log stream completely empty (zero incoming requests, ever), until Railway kills the container after healthcheck retries exhaust. healthcheckPath: /health, healthcheckTimeout: 60 in railway.json.
Timeline / count: 21 deploys in a row failed this way between 03:23 UTC and 15:33 UTC today, across many unrelated code changes (auth, i18n, scheduler, memory dedup, telegram linking, etc. — full list of deployment IDs available on request). Only ONE deploy succeeded in that window:
2e3775a1-4e21-4ca6-9ee3-304aca0fb318— redeploy of the OLD commit204d67d...at 15:28:53 UTC → SUCCESS at 15:30:38 UTC.a905652c-13c8-4314-b48f-cb3e921f4313— redeploy of our CURRENT HEAD commitbdfd4be...at 15:33:35 UTC → FAILED at 16:02:43 UTC.
Same environment, same redeploy action, ~5 minutes apart — old commit came up healthy, newer commit did not.
What we've ruled out on our side (with evidence, not guesses):
- Tailscale blocking startup — checked the very first failed deployment's full logs: Tailscale joined in 20s, well within the healthcheck window. Not the cause.
- IPv4-only bind vs IPv6 healthcheck prober — changed our server to bind
::(dual-stack) with IPv4 fallback. Logs confirm it bound successfully (serving on port 3000 (host=::), no fallback triggered) — deploy still failed identically with zero HTTP requests reaching the container.
Both fixes are deployed to main and did not resolve the issue, which points away from our application code and toward the platform/network layer between the healthcheck prober and our container.
Questions for Railway:
- Is there a known issue affecting healthcheck delivery to containers in
us-west2today (2026-08-14)? We noticed the "Logs and Metrics Delays" incident (started ~16:05 UTC) — could that be masking/related to an earlier, broader networking issue, or is it unrelated to our problem which started 03:23 UTC? - We also found a resolved incident from 2026-08-10 ("Connectivity issues in US West") noting some services needed a redeploy to fully recover — is there a similar unresolved condition on our service/region right now?
- Can you check server-side whether healthcheck requests are actually being sent to our container, and if so, whether they're reaching it? Our logs show literally zero inbound HTTP traffic on every failed deploy, which we can no longer be fully confident in given the logs/metrics incident today — can you confirm from Railway's side whether our container ever received the probe?
Happy to provide additional deployment IDs, full logs, or run further tests. This has been blocking our production deploys for over 12 hours.
4 Replies
2 months ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 2 months ago
2 months ago
already tried, that was the original binding, failed identically for 12h before we even touched networking code — see deployment history above
2 months ago
Update: we're now at 29 consecutive-window failed deploys (with exactly 1 success in between) since 2026-08-14 03:23 UTC, service kioku in project happy-luck.
Following the suggestion in this thread, we reverted our app to listen on 0.0.0.0 (it was already 0.0.0.0 for most of this window; we also independently tried :: dual-stack for a few hours, thinking IPv6 was the issue). Result: identical failure with both 0.0.0.0 and ::. This rules out the host-bind/IP-version theory entirely on our side.
Every failed deploy shows the same signature in logs: app boots clean (DB connects, Redis connects, HTTP server logs "serving on port 3000"), zero HTTP requests ever reach the container, healthcheck (/health, 60s window) times out and the deploy is marked FAILED.
Notably: a redeploy of an old, previously-successful commit (no code change at all) also failed on retry, then succeeded on a later retry — same commit, same image, different outcome. That's why we believe this is on the platform/cutover side, not in our application.
Could an engineer look directly at deploy f161aa35-ad98-4abd-9880-15a2eaaa48c3 (service 85d0a1b9-d2a4-425a-85a5-fd0a0c1520d0, project 652c4485-1734-4fc0-9ebb-d7ea6ae28ab5) — our latest failure, 2026-08-14 23:19:42 UTC — and tell us whether the healthcheck probe actually reached the container?
2 months ago
Update — resolved on our side, and the cause was ours. Apologies for the noise.
Our /health endpoint returns HTTP 503 whenever there are unapplied database migrations — it's a deliberate deploy gate on our end. Two migrations had been merged into our repo but never applied to the database, so from that moment every new container answered 503 to your healthcheck probe. The service unavailable in the deploy logs was literally our own 503, not your prober failing to reach the container.
That also explains the pattern that confused us: redeploying an older commit succeeded because it predates those migrations, while every newer commit failed regardless of what it changed.
We've applied the migrations; /health now returns 200 with "schema_drift":"none", and deploys pass normally with the healthcheck restored.
Nothing needed from your side — closing this out. Thanks for the earlier pointer about 0.0.0.0, and sorry for the time spent on our misdiagnosis.
Status changed to Solved 0x5b62656e5d • about 2 months ago