a month ago
On 2026-07-11 at 09:22 UTC, a Railway-initiated deployment (meta.reason: "migrate", creator: null) was created for my backend service. Instead of rebuilding the commit that was running at the time, it built a month-old commit from the middle of my repo's history, which contained a since-fixed bug that crashes the container on startup. This replaced my healthy deployment. I discovered it and recovered on 2026-07-13 by redeploying the last good build.
Affected resources:
- Project: SJV Holidays, 96affbac-a5d9-4514-8eb0-02c20b34d660 (hobby plan)
- Service: backend, a49a8622-d10c-4057-bfc7-164009a9b60a
- Environment: production, 7153970a-304f-4e5a-85ed-c5594bde17cc
- Repo: sjvc1111/sjv-holiday-website-backend, branch main
Timeline (all UTC):
- 2026-07-08 05:10: deployment 31915001-b941-4113-a037-7d3d0023e171 (reason: deploy, GitHub push) built main tip b975e45 -> SUCCESS, ran healthy for 3 days.
- 2026-07-11 09:22: deployment 11110d27-9f86-475a-b1dc-5fb625a09460 (reason: migrate, creator: null, skipBuildCache: true) built commit 0c1455a4c7f4ecbf1e142f74971694fd9989caec, an ancestor of main from around June 4, NOT the branch tip. It crash-looped on startup (a migration bug fixed in the very next commit back in June) and replaced the healthy deployment.
- 2026-07-13 07:36: I manually redeployed the last good build (ef894d18-1d40-4bff-af96-13242a2d7276) to recover.
Evidence the deploy was not triggered by us: the GitHub repo has zero events (no pushes, PRs, or branch activity) between 2026-07-06 19:50 UTC and 2026-07-13, and the deployment record has no creator.
Possibly relevant context: on 2026-07-04 we force-pushed a rewritten history to this repo (author reattribution, every commit SHA changed). Commit 0c1455a was never the head of main in either the old or new history, so I would like to understand how the migration resolved to it.
Questions:
- Why did the migration rebuild from source at a stale commit instead of reusing the running image (or the current branch tip)?
- Does the service still hold a stale commit reference that a future migration would rebuild? If so, can you clear it? (I will also trigger a fresh push-based deploy from my side.)
- Is there any way to be notified when a migrate deployment fails or crashes?
3 Replies
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
I was just logging the same finding and so will add it here.
On 11 Jul 2026 at ~09:26 UTC, a Railway-initiated migration (deployment reason migrate, deployment ID 953cdbfb-96cc-4b5a-9e92-7e90623cd1b4, project praxis-be-sg, environment dev, service web) redeployed our service from a commit dated 8 Jun (c7a0091), instead of the deployment that was active at the time (built 10 Jul from commit ff9eaa0, deployment 87314b36-e9a2-43f7-bc3f-ce2efb0b54df). This silently rolled the service back ~91 commits and broke our login flow until we noticed and manually redeployed on 13 Jul.
Could you confirm why the migration selected a stale image, and whether this can recur on our production environment during future migrations?
Status changed to Awaiting Railway Response Railway • about 1 month ago
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
Thanks pk-acrux same incident on my end, just 4 minutes earlier (09:22 UTC, 11 Jul, reason "migrate", null creator, rebuilt from a weeks-stale commit).
Railway team: could you confirm why the migration picked a stale commit, whether the stale reference is cleared for my service, and whether this can recur?
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Hey! We have an open ticket looking into this, when we get any info will update here!
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • about 1 month ago