a month ago
Service: magnolia-platform-backend1 (project: ample-simplicity / production environment)
Public domain: magnolia-platform-backend1-production.up.railway.app
Issue: The service is returning "Application failed to respond" on every route, including /health. This started this evening after deploying a few small code changes (adding a new webhook route). The problem persists even after rolling back to a deployment that was confirmed working before tonight's changes (commit 63b619d, "Bump version to force Railway cache-busted rebuild") - that redeploy also shows "Application failed to respond."
What I've checked/ruled out:
- Build logs show successful builds for every deployment tonight (image pushed, no build errors)
- Deploy Logs tab is completely empty for every recent deployment - no startup output, no error output, nothing
- Console tab shows "No running instances" / "Disconnected"
- Metrics tab shows CPU and memory completely flat at 0 for the past hour - no spike, no OOM signature
- No billing issues - payment method valid, no usage limit reached, no account warnings
- No PORT variable conflict - app uses process.env.PORT || 3000 as it always has
- Verified the exact deployed code runs correctly in a clean local Node environment with equivalent env vars (health check, and all new routes, respond correctly with no errors)
Given the code runs fine locally and the build succeeds, but the container never appears to actually start serving traffic (and produces zero deploy logs), this looks like it may be stuck in an unhealthy state at the infrastructure/container level rather than an application code issue.
Recent deploys, most recent first:
342fe92 - Lazy-init Anthropic client so missing API key doesn't crash the whole app
7c8a091 - Add GET handler for JustCall webhook URL validation
1caab1a - Fix JustCall payload parsing to match real webhook shape
8db002f - Add JustCall webhook + AI contact enrichment (Dispatch replacement)
63b619d - Bump version to force Railway cache-busted rebuild (last confirmed working deploy before tonight; also failing now after manual redeploy)
This service is running our production business platform (booking system, staff portal) for multiple physical office locations, so I'd appreciate an expedited look if possible. Happy to provide deployment IDs or any additional diagnostics you need.
Thank you,
James Westbrook
4 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
a month ago
fix
a month ago
When it deploys to the container have you told it to put anything in the container as log connection to database?
adamratty
When it deploys to the container have you told it to put anything in the container as log connection to database?
a month ago
No explicit "connected to database" log - we use the Supabase JS client, which doesn't require an upfront connection step, so there's nothing logged for that specifically.
However, the app does unconditionally log on every successful startup:
- "[Cron] First daily run in X minutes" - printed as soon as the process starts, before the server even begins listening
- A banner: "Magnolia Woods Platform API / Running on port X" - printed once Express successfully binds and starts listening
Neither of these lines - nor anything else - is appearing in the Deploy Logs tab for any of tonight's deployments, including the rolled-back one that was previously working. Since line 1 prints almost immediately on process start (before any network/DB activity), its absence suggests the container may not be reaching the point of executing our code at all, rather than our code hanging or crashing partway through.
Happy to add additional temporary logging (e.g. a console.log as the very first line of the file) if that would help you pinpoint where in the startup sequence things are stalling.
a month ago
Given the pattern you're describing (works fine locally, fails identically after rollback, zero logs, flat metrics), I'd push on two things before anything else: proving whether it's actually your app or something Railway-side, and adding logging that's low enough level to survive even a very early crash.
For isolating it — spin up a throwaway service in the same project, literally just a "hello world" express app that listens and logs on boot. If that also comes back with zero logs and "Application failed to respond," that's basically confirmation this isn't your code, it's something with the project/environment on Railway's end, and worth pushing to their support directly rather than digging further into your app. Also worth trying an actual restart on the instance rather than a redeploy — redeploy can reuse a build that's inheriting a stuck state, restart forces a clean container. And if your service is pinned to a region, try switching regions, just to rule out a regional issue.
On the logging side — since literally nothing is showing up, not even your [Cron] line that should fire almost instantly, I'd go a level below your app code. Stick something in the actual start command itself so it fires before Node even loads:
echo "CONTAINER BOOT $(date)" && node dist/index.js
If that echo doesn't show up in Deploy Logs, that's pretty damning — it means either the container isn't running your start command at all, or Railway's log capture is broken, and either way it's not something you can fix from your side.
Then as the literal first line of your entry file, before anything else loads:
jsprocess.stdout.write(BOOT ${new Date().toISOString()} pid=${process.pid}\n);
process.stdout.write instead of console.log just to rule out any logging lib swallowing output if the process dies fast. Right under that, add handlers for uncaught exceptions and unhandled rejections that also write directly to stdout — this catches a synchronous throw during module load that would otherwise die completely silently, before your normal logger or Express even spins up:
jsprocess.on('uncaughtException', e => { process.stdout.write(FATAL ${e.stack}\n); process.exit(1); });
process.on('unhandledRejection', e => { process.stdout.write(REJECTION ${e}\n); });
If you want to go further, drop a log line after each init step (env load, Supabase client, Express app creation, the .listen() callback) so if it does start but stalls somewhere, you can see exactly where.
If you deploy with all that and still get absolutely nothing in Deploy Logs — no echo, no BOOT line — that's about as clean a proof as you can get that the container isn't executing your start command at all. That's worth taking straight to Railway support (not just this bounty thread) since it's a much stronger case than "logs are empty" — it shows the earliest possible instrumentation point still isn't showing up.