an hour ago
Support ticket text — paste as-is into station.railway.com "Get Help!"
Subject: Production service failing healthcheck since 03:23 UTC today, 21+ consecutive failed deploys, one exception on redeploy
Project: happy-luck (652c4485-1734-4fc0-9ebb-d7ea6ae28ab5)
Service: kioku (85d0a1b9-d2a4-425a-85a5-fd0a0c1520d0)
Environment: production (548c3b9d-352d-486c-b602-7e02c78a0247)
Region: us-west2, Runtime V2
Our production service (kioku) has been failing its healthcheck on almost every deploy since 2026-08-14 03:23:25 UTC (deployment f1771147-cbce-446c-8d6b-c3794c7156b2, commit 454dd0485e6a0d1cb8f8d125932308e3ece2c29c). Last known good deploy before that: commit 204d67d399c608e331d91ab040bd99e36699b9e7, live since 2026-08-13 23:59 UTC.
Symptom (identical on every failure): app boots cleanly — DB connects, Redis connects, all cron jobs register, server logs "listening on port 3000" — then runs for several minutes with the http log stream completely empty (zero incoming requests, ever), until Railway kills the container after healthcheck retries exhaust. healthcheckPath: /health, healthcheckTimeout: 60 in railway.json.
Timeline / count: 21 deploys in a row failed this way between 03:23 UTC and 15:33 UTC today, across many unrelated code changes (auth, i18n, scheduler, memory dedup, telegram linking, etc. — full list of deployment IDs available on request). Only ONE deploy succeeded in that window:
2e3775a1-4e21-4ca6-9ee3-304aca0fb318— redeploy of the OLD commit204d67d...at 15:28:53 UTC → SUCCESS at 15:30:38 UTC.a905652c-13c8-4314-b48f-cb3e921f4313— redeploy of our CURRENT HEAD commitbdfd4be...at 15:33:35 UTC → FAILED at 16:02:43 UTC.
Same environment, same redeploy action, ~5 minutes apart — old commit came up healthy, newer commit did not.
What we've ruled out on our side (with evidence, not guesses):
- Tailscale blocking startup — checked the very first failed deployment's full logs: Tailscale joined in 20s, well within the healthcheck window. Not the cause.
- IPv4-only bind vs IPv6 healthcheck prober — changed our server to bind
::(dual-stack) with IPv4 fallback. Logs confirm it bound successfully (serving on port 3000 (host=::), no fallback triggered) — deploy still failed identically with zero HTTP requests reaching the container.
Both fixes are deployed to main and did not resolve the issue, which points away from our application code and toward the platform/network layer between the healthcheck prober and our container.
Questions for Railway:
- Is there a known issue affecting healthcheck delivery to containers in
us-west2today (2026-08-14)? We noticed the "Logs and Metrics Delays" incident (started ~16:05 UTC) — could that be masking/related to an earlier, broader networking issue, or is it unrelated to our problem which started 03:23 UTC? - We also found a resolved incident from 2026-08-10 ("Connectivity issues in US West") noting some services needed a redeploy to fully recover — is there a similar unresolved condition on our service/region right now?
- Can you check server-side whether healthcheck requests are actually being sent to our container, and if so, whether they're reaching it? Our logs show literally zero inbound HTTP traffic on every failed deploy, which we can no longer be fully confident in given the logs/metrics incident today — can you confirm from Railway's side whether our container ever received the probe?
Happy to provide additional deployment IDs, full logs, or run further tests. This has been blocking our production deploys for over 12 hours.
1 Replies
an hour ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 hour ago