a month ago
ubject: Service stuck deploying a stale/cached source snapshot — preDeployCommand doesn't match repo content across ~26 consecutive deploys
Project ID: 42e86d53-8971-48fc-a49e-e1e2c770aa7a Environment: production (23bac9c6-fbd4-4837-a1b0-f0ab631c81a8) Service: lsr-ecos (service ID: be7ebdde-6bd4-46f5-90e0-0b6352115b0a)
Symptom
Since ~2026-07-17 20:39 (America/Sao_Paulo), every deploy of the lsr-ecos service has failed, and the service has been returning HTTP 502 continuously. We've made ~26 deploy attempts across 4 different methods (Railway CLI railway up, railway redeploy --from-source, the in-app Railway AI agent, and switching the Build → Builder setting between Railpack and Nixpacks) — none of them changed the actual runtime behavior.
Evidence that the deployed content is stale/cached, not our current repo
Our repo's railway.json sets deploy.preDeployCommand to a specific shell wrapper script (bash packages/database/scripts/migrate-with-retry.sh), and we've edited/committed that script multiple times today. But the deploy logs for our most recent attempts still show the container running a completely different, older command:
$ drizzle-kit migrate
— not the wrapper script at all. This exact same discrepancy has appeared across multiple deploys with different commits.
Two consecutive build logs (deployment IDs 4aeeb7b3-7b6b-4639-b4b4-9953a99338b4 and 65b1fe34-82c5-4060-b877-46796897a414) fetched different snapshot hashes (sha256:cde47822... vs sha256:cf942035...), but both still ran the stale drizzle-kit migrate command rather than our current repo's configured preDeployCommand — so even when the snapshot hash changes, the effective build/runtime content does not.
We also found — separately from this — that the service's startCommand had a value set directly in the dashboard (sh packages/database/scripts/migrate-with-retry.sh && pnpm start) that was never committed to our repo, which was independently causing a duplicate migration run at container start (set -o pipefail invoked under sh, which doesn't support that flag, causing Illegal option -o pipefail and a crash loop). We've since committed a fix that sets startCommand explicitly in railway.json (to pnpm start), but that fix also does not appear to have taken effect in subsequent deploys, consistent with the stale-snapshot issue in point 1/2.
What we're asking for
Could you manually purge the build/source snapshot cache for this service (lsr-ecos, service ID be7ebdde-6bd4-46f5-90e0-0b6352115b0a, in project 42e86d53-8971-48fc-a49e-e1e2c770aa7a, environment production)? We believe a stuck cache entry is causing every new deploy to reuse stale build output regardless of the actual source uploaded or pulled from GitHub.
Reference deployment IDs for investigation
Last known-good deployment (before the incident): bbf1738c-755a-498c-80df-25775759a7af (2026-07-17 20:27:10 -03:00)
Representative failed deployments showing the stale-content symptom: 4aeeb7b3-7b6b-4639-b4b4-9953a99338b4 (2026-07-17 21:21:04), 65b1fe34-82c5-4060-b877-46796897a414 (2026-07-17 21:35:56)
Happy to provide any additional logs or repo diffs on request. This is currently a full production outage for us, so any expedited help would be greatly appreciated
5 Replies
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
"Already tried that — set NO_CACHE=1 as a service variable a few minutes ago and redeployed. The build log confirmed it worked as intended (no 'fetched snapshot' cache-restore step at all; it installed the OS packages and Node runtime completely from scratch, ~2x longer build time than normal). But the resulting build still shipped the OLD source: the /security/enroll route came out at 1.6 kB (matching the old code) instead of ~10.8 kB (current committed code, which adds a qrcode dependency to that page). So this isn't a build-cache problem — even a fully uncached build is somehow pulling old source content. Something deeper (snapshot/source resolution for this specific service) is stuck, not just the build cache layer."
a month ago
railpack default
a month ago
Have you try Redeploy Latest Commit through the command palette? or it didn't work too?
