a month ago
Project ID: 5e620e5f-c36b-4cb4-9a2b-0706be89d3ae
Service: reven-dashboard (Service ID: 99742155-da69-4193-a7ca-bc270230fffb)
Environment: staging (Environment ID: 7bc67475-a181-4adb-8472-a771fc05456a)
Domain: reven-dashboard-staging.up.railway.app
Issue: Multiple consecutive deployments reach SUCCESS, but the public domain's HTTP responses do not reflect the code that was verifiably deployed.
Evidence:
We added a hard build-time assertion (a RUN step that fails the build if a freshly-written marker file doesn't match the current build arg). This assertion PASSES on every recent deployment — proving the built image genuinely contains our latest source code.
We added a deployNonce field to our /api/health endpoint, sourced from that same build arg via ENV. After each deploy, we poll this endpoint — uptime shows the container is genuinely freshly booted (tens of seconds old, climbing continuously, single consistent origin per x-hikari-trace), but the deployNonce field is absent, meaning the process running is NOT from the image that just passed our build assertion.
This reproduced on two consecutive, independent deployments: f3059870-1287-41eb-ae95-c5e5ab8533cf (via railway up) and d70aa14c-b448-4bf8-aff2-4776907c0db8 (via railway redeploy).
Ruled out on our side: multiple replicas (dashboard confirms 1), CDN caching (disabled in settings), volume overlap (mount confirmed at /app/data only via df/mount), wrong start command (ps aux / /proc/1/cmdline confirm node backend/src/server.js, matching our Dockerfile CMD), stale build cache (see hard assertion above).
Separately, "Teardown" was disabled for this service until earlier today — we enabled it and confirmed it now works (old deployments correctly transition to REMOVED). This did not resolve the core issue above.
Question: What could cause a deployment marked SUCCESS, serving a freshly-booted container, to not serve the image that was actually built and verified for that deployment? Is there a caching or routing layer between build and public traffic that could explain this?
Happy to share build/deploy logs for the specific deployment IDs above.
1 Replies
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
An absent deployNonce doesn't prove the wrong image is running - it proves process.env.DEPLOY_NONCE is undefined in the running process (JSON.stringify drops undefined fields). Your build assertion can't catch this, because ARG is build-scope only.
Likely cause: multi-stage ARG scoping. ARG must be re-declared in every stage that uses it. If your assertion RUN is in the builder stage (ARG in scope, so the build passes) but the final stage does ENV DEPLOY_NONCE=$DEPLOY_NONCE without its own ARG DEPLOY_NONCE after that stage's FROM, the ENV expands to empty on every deploy.
Test it: railway ssh, then run: env | grep -i nonce. If empty, it's the env chain, not the image.
Fix: add ARG DEPLOY_NONCE in the runtime stage before the ENV line - or skip env entirely: RUN echo "$DEPLOY_NONCE" > /app/nonce.txt at build and have /api/health read the file.
Also: railway redeploy reuses the previous image without rebuilding, so that second repro isn't independent evidence.