Deployments stuck at "Created" instance state with zero deploy logs — builds succeed, service never serves traffic
navjotschahal30-ai
HOBBYOP

a month ago

Project ID: dba4a9bb-23e9-44a9-8cd5-25d494195afc (remarkable-renewal)

Service: vern (d4ad9031-24d3-4f84-8dbc-9e5a0d0a97bd)

Environment: production (7f4a6c5a-140c-4be5-93a4-17400a8298c1)

Region: sfo

Plan: Hobby

Summary:

My service (a Node/Express app, Railpack builder, Node 22.23.1) has been unreachable since around 21:06 UTC on 2026-07-13 (Application failed to respond / 502 on every request, including /health). I've tried four separate remediation steps across ~4 hours and gotten a different-looking failure each time, but never a stable recovery:

  1. railway redeploy — reused the existing build. Instance stayed at status Created indefinitely, 0/1 replicas running, zero deploy logs (not even app startup output).
  2. railway down + railway redeploy --from-source — fully fresh build, skipBuildCache: true, no reused layers. Build succeeded (clean tsc compile, image pushed, ~130MB). Same result: instance stuck at Created, zero deploy logs.
  3. Manual restart + redeploy from the dashboard — this one briefly worked (/health returned 200 for a couple minutes), then the service dropped again on its own with a brand-new instance ID and the same zero-log/stuck-Created symptom. No traffic spike or deploy triggered the drop — it just went away on its own.
  4. Code fix: per your own troubleshooting docs, I checked target port config (railway domain list → targetPort: 8080, matches PORT env var, so that's not it) and then explicitly bound the Express server to 0.0.0.0 instead of relying on Node's default (this exact code had been running fine on Railway for weeks prior, unchanged). Pushed and redeployed (deployment 99bfe117-3654-497a-90cc-380809281bf3). Result: still 502/timeout on every request, but the Metrics tab now shows Memory jumping from 0 to a sustained ~200MB (unlike every prior attempt, which sat completely flat at 0 CPU/0 memory/0 network the whole time) — so this attempt may have actually gotten the process running, but it still isn't accepting/completing requests (responses hang for several seconds before failing rather than failing fast).

What I've ruled out:

  • Not a code-level crash: same commit runs and serves /health correctly in under 3 seconds when built and run locally with production env vars.
  • Not a target-port mismatch (confirmed via railway domain list).
  • Not an account/billing limit (railway usage shows well under any limit, not over).
  • Not a broad platform incident that I could find reported.

Deploy IDs for reference (chronological):

  • 5d99a4d2-4df9-40de-8822-4b89085e291a
  • 7609bdbb-aed2-4ce9-acfb-ee6c72c20fb8
  • 43346969-a8b4-435b-a973-55b100dae64e
  • 99bfe117-3654-497a-90cc-380809281bf3 (current)

Would appreciate help figuring out why the instance isn't transitioning out of Created / isn't producing any deploy logs, and why it briefly recovered once before dropping again unprompted. Happy to provide anything else needed.

$10 Bounty

3 Replies

Railway
BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway about 1 month ago


hygrivakondury
HOBBY

a month ago

Please try this if it helps:

These are actually two different failures — separating them is the key.

Attempts 1–3 (stuck at Created, zero logs, flat 0 metrics) = platform-side, not your code. No logs + no CPU/mem/network means the container never started — your process never ran, so there's nothing in your code to fix here. The unchanged commit that ran for weeks, plus attempt 3 self-dropping with a new instance ID and no trigger, both point to the orchestrator cycling a wedged deploy. Escalate this to Railway with your deploy IDs. Quick isolation test: redeploy to a non-sfo region and see if the Created symptom vanishes.

Attempt 4 (process runs ~200MB, but every request incl. /health hangs then fails) = app-side, and fixable. The jump to sustained memory means it started this time. "Hangs then fails" (not fail-fast) is the classic sign of a process blocking on a startup dependency it can't reach — a DB/Redis connection that's instant locally but hangs over Railway's private network. Check:

Anything await-ing a DB/cache connection before app.listen()? Bind the server first, connect lazily.

Does /health touch the DB? Make it a static 200 that touches nothing — if it then responds instantly while other routes hang, that isolates the culprit.

Verify any *.railway.internal hostnames the app dials.

The "bind first, connect lazily" pattern:

"

jsconst app = express();

// Health check touches nothing — always answers instantly

app.get("/health", (_req, res) => res.sendStatus(200));

app.use(routes);

// Bind FIRST so the platform sees a ready server immediately

app.listen(process.env.PORT, "0.0.0.0", () => {

console.log("listening on", process.env.PORT);

});

// Connect to dependencies AFTER, without blocking startup

initDb().catch((err) => console.error("db connect failed:", err));

"

If your DB/Redis connect is await-ed before listen(), a dependency that hangs takes the whole server down with it — exactly your attempt-4 symptom. Binding first means the process is reachable the moment it starts; a hanging dependency then fails loudly in logs instead of silently wedging the deploy.

TL;DR: escalate 1–3 to Railway (platform-side, nothing to fix in your code), and for attempt 4, decouple binding + /health from your dependencies.


navjotschahal30-ai
HOBBYOP

a month ago

@hygrivakondury: Per your earlier note, we fixed our app to bind explicitly to 0.0.0.0 instead of the Node default dual-stack ::, since that matched the "stuck at Created, zero logs, flapping 502s" symptom on attempts 1-3. That fix deployed at 00:54:58Z yesterday. It didn't help — the exact same reachability symptom continued on the new deployment for the next ~12 hours, then cleared on its own with no restart, redeploy, or code change on our end. This points to a proxy/networking issue on your side in the sfo region for this project, not our application.

Evidence:

  • Same instance (7f6ea3ad...) ran continuously the whole window — no crash, no restart, no OOM, no new deployment. Console logs show only the normal startup line (Starting Container → Vern listening on port 8080) once, at deploy time.
  • HTTP logs from 00:56:26Z to 13:02:01Z (today): of 58 requests to /health (a static handler — no DB, no dependencies, res.json({status:'ok'})), 49 returned 499 (client/proxy gave up after ~7.9s) and 5 returned 502 (hard 15s timeout). Only 4 succeeded, all after 13:09:16Z.
  • Zero non-health requests logged in that entire 12-hour window — no inbound webhook traffic reached the app at all (or reached it and never got logged), which is itself notable for a service that normally receives regular webhook calls.
  • Recovery at 13:09:16Z was not caused by anything on our side — we checked our scheduled jobs (GitHub Actions) and confirmed nothing fired in that window; the service just started responding in single-digit milliseconds again with no intervention.

navjotschahal30-ai
HOBBYOP

a month ago

  1. Can you check proxy/edge reachability logs for this deployment/instance in sfo between 2026-07-14T00:54Z and 13:09Z? We'd like to know what changed at 13:09Z that fixed it.
  2. Is this a known recurring issue for the sfo region? If so, would redeploying to a different region avoid it going forward, or is this project-specific?
  3. Given the app was unreachable for ~12 hours despite a healthy, running process, is there anything about this service's health-check or proxy configuration you'd recommend changing on our end?

Welcome!

Sign in to your Railway account to join the conversation.

Loading...