12 days ago
Hello,
Service ID: 20184990-ec8d-4c98-9d14-4b25851dd608
Project ID: f9158b9c-0d60-4886-8068-501c8c4725e2
Builder: production-builderv3-us-west1-nf4h
Railpack version: 0.23.0
Deployment method: railway up (CLI only, no Git, no GitHub)
THIS IS AN URGENT MATTER FOR US!
The problem
A source change to server.ts is confirmed present in the local file. Railway receives the updated snapshot, runs npm run build, and tsc executes. Yet the deployed server continues to serve the old compiled output.
The evidence in sequence
For many consecutive deployments, the final image digest was identical:
sha256:bd12b175769d7b382023af7bc8553c19f611bcef1b7b0e0937a9f92ca36d3111
After changing the service region from US West to US East, the last deployment produced a genuinely new digest:
sha256:0661ec055af48dbe0d3054cc83ab52865af8d8dce9d3119cd53a931534df0a26
The build log confirmed tsc executed: > lucid-backend@1.0.0 build > tsc. Yet /health still returns the old response, confirming the deployed container does not reflect the compiled source.
The snapshot hash has been consistent across all deploys: sha256:9613258a68750195d0b2498ac682d8ec81d27f628465949357e288cd648c8c88 — confirming Railway receives the correct updated files every time.
Everything tried — all failed
NO_CACHE=1 as a service variable
RAILPACK_NO_CACHE=1 as a service variable
RAILPACK_DISABLE_CACHES=* as a service variable
Version bump in package.json
Echo cache-buster string injected into the build script
Watch Paths set to server.ts in the dashboard
railway.toml added with watchPatterns: ["server.ts", "package.json", "nixpacks.toml"]
Region changed from US West to US East
What we need
Railway is receiving the correct source, building a fresh image, and reporting a new digest — but the deployed container is not running the compiled output of that fresh build. We need Railway to identify what is happening between the fresh build and the container startup that causes the old output to be served.
Thank you!
NOTE: We added a deliberate TypeScript syntax error to server.ts. Railway reported 'no changes detected in watch paths, build will skip' and successfully deployed the unchanged cached image. Railway is not reading server.ts at all.
27 Replies
12 days ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • 12 days ago
12 days ago
Hello,
Can you please link specific deployments where NO_CACHE=1 didn't work and where watchPatterns were not respected.
Status changed to Awaiting User Response Railway • 12 days ago
edgnkv
Hello, Can you please link specific deployments where NO_CACHE=1 didn't work and where `watchPatterns` were not respected.
12 days ago
Thank you for responding. Here are the specific deployment links. etc:
Deployment where NO_CACHE=1 was set and did not work:
Most recent deployment — "No changes to watched files" with Watch Paths field empty and no railway.toml present:
Full chronology of what was tried:
NO_CACHE=1 as a service variable — no effect.
RAILPACK_NO_CACHE=1 as a service variable — no effect.
RAILPACK_DISABLE_CACHES=* as a service variable — no effect.
Version bump in package.json — no effect.
Echo cache-buster string in the build script — no effect.
railway.toml added with watchPatterns: ["server.ts", "package.json", "nixpacks.toml"] — no effect; introduced "no changes detected in watch paths, build will skip" message.
Watch Paths field in dashboard set to server.ts — no effect.
railway.toml deleted, Watch Paths field cleared, all variables removed — build still skips with "No changes to watched files."
Region changed from US West to US East — build still skipped on the new region (us-east4-eqdc4a), confirming the frozen watch path configuration is stored at the service level, not the builder node level.
Deliberate TypeScript syntax error added to server.ts — Railway reported "no changes detected in watch paths, build will skip" and deployed successfully, proving server.ts is never read.
The watch path rules appear permanently frozen in your infrastructure. Nothing from our side has been able to clear them.
Thank you!
Status changed to Awaiting Railway Response Railway • 12 days ago
12 days ago
Thank you — your root cause analysis matches exactly what we found. The snapshot hash has been identical across every deploy despite confirmed file changes, and the syntax error test proved Railway never reads the updated file.
On your workarounds: railway up --clean does not exist in our version of the Railway CLI — it returns an "unexpected argument" error. And switching to Git deployment is not an option for this project.
We have since found one additional data point: after deleting railway.toml and clearing Watch Paths from the dashboard entirely, Railway still reports "No changes to watched files" on every deploy — including on a completely different region (us-east4-eqdc4a). The watch path configuration is frozen at the service level on Railway's infrastructure and cannot be cleared from our side. The Railway employee is now investigating.
Thank you for responding. Here are the specific deployment links. etc: Deployment where NO_CACHE=1 was set and did not work: https://railway.com/project/f9158b9c-0d60-4886-8068-501c8c4725e2/service/20184990-ec8d-4c98-9d14-4b25851dd608?id=faf9001e-cde5-4844-b43b-22e154c9b1bc Most recent deployment — "No changes to watched files" with Watch Paths field empty and no railway.toml present: https://railway.com/project/f9158b9c-0d60-4886-8068-501c8c4725e2/service/20184990-ec8d-4c98-9d14-4b25851dd608?environmentId=63d14ee8-5ef5-49f5-81fe-39e0569716bc&id=6ce9d29a-7e37-476e-bf28-3f192d6040cf#build Full chronology of what was tried: NO_CACHE=1 as a service variable — no effect. RAILPACK_NO_CACHE=1 as a service variable — no effect. RAILPACK_DISABLE_CACHES=* as a service variable — no effect. Version bump in package.json — no effect. Echo cache-buster string in the build script — no effect. railway.toml added with watchPatterns: ["server.ts", "package.json", "nixpacks.toml"] — no effect; introduced "no changes detected in watch paths, build will skip" message. Watch Paths field in dashboard set to server.ts — no effect. railway.toml deleted, Watch Paths field cleared, all variables removed — build still skips with "No changes to watched files." Region changed from US West to US East — build still skipped on the new region (us-east4-eqdc4a), confirming the frozen watch path configuration is stored at the service level, not the builder node level. Deliberate TypeScript syntax error added to server.ts — Railway reported "no changes detected in watch paths, build will skip" and deployed successfully, proving server.ts is never read. The watch path rules appear permanently frozen in your infrastructure. Nothing from our side has been able to clear them. Thank you!
12 days ago
How did you specify NO_CACHE=1 env var? I do not see it specified in the build request. Also, when NO_CACHE=1 is set then you should also see 🚂 Caching Disabled... in build logs.
Separately I am looking into watchPatterns issue.
Status changed to Awaiting User Response Railway • 12 days ago
edgnkv
How did you specify `NO_CACHE=1` env var? I do not see it specified in the build request. Also, when `NO_CACHE=1` is set then you should also see `🚂 Caching Disabled...` in build logs. Separately I am looking into `watchPatterns` issue.
12 days ago
NO_CACHE=1 was set as a service environment variable through the Railway dashboard — Variables tab, added as a key/value pair. It was not set at the CLI level or in any configuration file.
You are correct that we never saw 🚂 Caching Disabled... in the build logs — which confirms the variable was not being picked up by the builder. This is consistent with what another user explained to us: service environment variables are injected at runtime, not at build time, so the builder never sees them. If NO_CACHE=1 needs to be set differently to reach the builder, we were never told the correct method.
To clarify the full sequence: NO_CACHE=1 was added to the Variables tab, had no effect, and was subsequently deleted. The same applies to RAILPACK_NO_CACHE=1 and RAILPACK_DISABLE_CACHES=* — all three were tried and removed. None of them produced the 🚂 Caching Disabled... message in the build logs.
We look forward to your findings on the watchPatterns issue.
Status changed to Awaiting Railway Response Railway • 12 days ago
NO_CACHE=1 was set as a service environment variable through the Railway dashboard — Variables tab, added as a key/value pair. It was not set at the CLI level or in any configuration file. You are correct that we never saw 🚂 Caching Disabled... in the build logs — which confirms the variable was not being picked up by the builder. This is consistent with what another user explained to us: service environment variables are injected at runtime, not at build time, so the builder never sees them. If NO_CACHE=1 needs to be set differently to reach the builder, we were never told the correct method. To clarify the full sequence: NO_CACHE=1 was added to the Variables tab, had no effect, and was subsequently deleted. The same applies to RAILPACK_NO_CACHE=1 and RAILPACK_DISABLE_CACHES=* — all three were tried and removed. None of them produced the 🚂 Caching Disabled... message in the build logs. We look forward to your findings on the watchPatterns issue.
12 days ago
But the deployments that you linked didn't have NO_CACHE variable set. You can verify it by clicking list of variables on "Details" tab of the deployment.
Status changed to Awaiting User Response Railway • 12 days ago
edgnkv
But the deployments that you linked didn't have NO_CACHE variable set. You can verify it by clicking list of variables on "Details" tab of the deployment.
12 days ago
After checking all deployments from today, none of the cache variables — NO_CACHE=1, RAILPACK_NO_CACHE=1, or RAILPACK_DISABLE_CACHES=* — appear in any deployment's variable list. Every build has been skipped due to the frozen watch path state, so no variable change has ever reached the builder. They appeared purple in the dashboard — never white — suggesting they were never applied to any live deployment.
We have now spent an entire working day on this. Every approach has been exhausted. The watch path configuration is frozen in your infrastructure, cannot be cleared from our side, and is blocking all deployments. A production backend is effectively locked.
We are not asking for further diagnosis — we are asking for a direct infrastructure reset of the watch path state and build cache for service 20184990-ec8d-4c98-9d14-4b25851dd608. That is the only action that will unblock this. Can you please do that now?
Status changed to Awaiting Railway Response Railway • 12 days ago
After checking all deployments from today, none of the cache variables — NO_CACHE=1, RAILPACK_NO_CACHE=1, or RAILPACK_DISABLE_CACHES=* — appear in any deployment's variable list. Every build has been skipped due to the frozen watch path state, so no variable change has ever reached the builder. They appeared purple in the dashboard — never white — suggesting they were never applied to any live deployment. We have now spent an entire working day on this. Every approach has been exhausted. The watch path configuration is frozen in your infrastructure, cannot be cleared from our side, and is blocking all deployments. A production backend is effectively locked. We are not asking for further diagnosis — we are asking for a direct infrastructure reset of the watch path state and build cache for service 20184990-ec8d-4c98-9d14-4b25851dd608. That is the only action that will unblock this. Can you please do that now?
12 days ago
The latest redeployment (5 hours ago) is successful (there is UI bug that says No changes to watched files because redeploy was triggered from skipped build). I see that you have deleted watchPatterns, thus new deployments should work.
Status changed to Awaiting User Response Railway • 12 days ago
edgnkv
The latest redeployment (5 hours ago) is successful (there is UI bug that says `No changes to watched files` because redeploy was triggered from skipped build). I see that you have deleted `watchPatterns`, thus new deployments should work.
12 days ago
Thank you for your help so far. The watch path issue appears partially resolved — we are no longer seeing "no changes detected in watch paths, build will skip." However, the build cache is still serving a frozen image.
Here is the evidence from the latest deploy:
Snapshot hash: sha256:7e072a7d93a60886813a8d71421c13f29b09ebf01c696df49edf4c676e0a9411 — this is different from all previous deploys, confirming Railway received the updated source files.
Final image digest: sha256:fd4539612452f09c13af32194341ab3eeb5094861b5146fd55b36d4ae1055411 — identical to the previous deploy.
Config digest: sha256:afaef30b3e6db39e4731c1a0be85471efb4972b54b3e62eb663e4be5b67f0d33 — also identical to the previous deploy.
Railway is receiving the new files but producing the same compiled image. The compiled output layer is still being pulled from cache rather than built from the uploaded source. The /health endpoint continues to return the old response, confirming the deployed server does not reflect the source changes.
Deployment link:
Status changed to Awaiting Railway Response Railway • 12 days ago
Thank you for your help so far. The watch path issue appears partially resolved — we are no longer seeing "no changes detected in watch paths, build will skip." However, the build cache is still serving a frozen image. Here is the evidence from the latest deploy: Snapshot hash: sha256:7e072a7d93a60886813a8d71421c13f29b09ebf01c696df49edf4c676e0a9411 — this is different from all previous deploys, confirming Railway received the updated source files. Final image digest: sha256:fd4539612452f09c13af32194341ab3eeb5094861b5146fd55b36d4ae1055411 — identical to the previous deploy. Config digest: sha256:afaef30b3e6db39e4731c1a0be85471efb4972b54b3e62eb663e4be5b67f0d33 — also identical to the previous deploy. Railway is receiving the new files but producing the same compiled image. The compiled output layer is still being pulled from cache rather than built from the uploaded source. The /health endpoint continues to return the old response, confirming the deployed server does not reflect the source changes. Deployment link: https://railway.com/project/f9158b9c-0d60-4886-8068-501c8c4725e2/service/20184990-ec8d-4c98-9d14-4b25851dd608?id=068ddc7a-e4f0-447f-9a6c-ee5453565c50
12 days ago
I suggest you to set NO_CACHE=1 env variable. As I see there were un-cached builds (e.g. 6ce9d29a-7e37-476e-bf28-3f192d6040cf), but next builds were cached.
Status changed to Awaiting User Response Railway • 12 days ago
edgnkv
I suggest you to set `NO_CACHE=1` env variable. As I see there were un-cached builds (e.g. `6ce9d29a-7e37-476e-bf28-3f192d6040cf`), but next builds were cached.
12 days ago
Thank you for the NO_CACHE=1 suggestion — it produced a fresh config digest (sha256:331f369cf2654629db4b6e5c0b02274ada1d71aa5c38f088e63c6905a4cd6478), which is different from all previous deploys, and tsc executed during the build. However the deployed server still does not reflect the source changes.
We have a specific change in our local server.ts — a build_marker field added to the /health endpoint. This change has been confirmed present in the local file multiple times. After every build where tsc ran, the /health endpoint continues to return the old response without build_marker.
This means Railway is running tsc but not compiling from the server.ts we are uploading. The source file Railway is actually reading during compilation appears to be something other than what we are providing.
Deployment link for the latest build:
Can you check what source file tsc is actually reading during the build on your infrastructure side?
Status changed to Awaiting Railway Response Railway • 12 days ago
edgnkv
I suggest you to set `NO_CACHE=1` env variable. As I see there were un-cached builds (e.g. `6ce9d29a-7e37-476e-bf28-3f192d6040cf`), but next builds were cached.
12 days ago
Following up — the issue remains unresolved and the service is unable to deploy updated code. Any update on your investigation would be appreciated.
edgnkv
I suggest you to set `NO_CACHE=1` env variable. As I see there were un-cached builds (e.g. `6ce9d29a-7e37-476e-bf28-3f192d6040cf`), but next builds were cached.
11 days ago
Following up with a significant new finding. We have now been completely blocked from deploying updated code for two full days.
The latest build log still shows copy backup/package.json referencing a folder that no longer exists in the project. npm run build and tsc did not appear in this build at all — the compilation step was skipped entirely and the cached image was served again.
This confirms the build plan itself is cached on Railway's infrastructure and does not reflect the actual current state of the project. No local change — file additions, deletions, or modifications — is reaching the builder. Our service is effectively frozen in a state we cannot update.
Latest deployment: https://railway.com/project/f9158b9c-0d60-4886-8068-501c8c4725e2/service/20184990-ec8d-4c98-9d14-4b25851dd608?id=45b7822b-aa69-4a08-91f8-4f91d0264211
We also note that yesterday's major Railway outage coincided exactly with when this issue began. The build plan cache was very likely corrupted during that event.
We are on the Pro plan. Two days of being unable to deploy is a serious production impact.
Can you please clear the cached build plan for this service from the infrastructure side?
Following up with a significant new finding. We have now been completely blocked from deploying updated code for two full days. The latest build log still shows copy backup/package.json referencing a folder that no longer exists in the project. npm run build and tsc did not appear in this build at all — the compilation step was skipped entirely and the cached image was served again. This confirms the build plan itself is cached on Railway's infrastructure and does not reflect the actual current state of the project. No local change — file additions, deletions, or modifications — is reaching the builder. Our service is effectively frozen in a state we cannot update. Latest deployment: https://railway.com/project/f9158b9c-0d60-4886-8068-501c8c4725e2/service/20184990-ec8d-4c98-9d14-4b25851dd608?id=45b7822b-aa69-4a08-91f8-4f91d0264211 We also note that yesterday's major Railway outage coincided exactly with when this issue began. The build plan cache was very likely corrupted during that event. We are on the Pro plan. Two days of being unable to deploy is a serious production impact. Can you please clear the cached build plan for this service from the infrastructure side?
11 days ago
I multiple times cycled builders, you had multiple deployments being uncached - e.g. https://railway.com/project/f9158b9c-0d60-4886-8068-501c8c4725e2/service/20184990-ec8d-4c98-9d14-4b25851dd608?environmentId=63d14ee8-5ef5-49f5-81fe-39e0569716bc&id=3fb2fcd5-0839-4ced-a6d5-76c290c885ec#build
The next deployment after this one was cached.
edgnkv
I multiple times cycled builders, you had multiple deployments being uncached - e.g. https://railway.com/project/f9158b9c-0d60-4886-8068-501c8c4725e2/service/20184990-ec8d-4c98-9d14-4b25851dd608?environmentId=63d14ee8-5ef5-49f5-81fe-39e0569716bc&id=3fb2fcd5-0839-4ced-a6d5-76c290c885ec#build The next deployment after this one was cached.
11 days ago
Thank you for cycling the builders. The uncached build at 3fb2fcd5 ran successfully, but every subsequent deploy has reverted to the cached image digest sha256:b52be351ed7a588990686acf13385cfa29aca6da672b32723d44beefb09c8c93.
The build log continues to show copy backup/package.json even though the backup folder has been completely removed from the project. The cached build plan on your infrastructure still references the old project structure and is not reflecting the current filesystem state.
Can you please cycle the builders again so we get another uncached build? And is there a permanent fix that prevents the cache from reverting after each uncached build?
Thank you for cycling the builders. The uncached build at 3fb2fcd5 ran successfully, but every subsequent deploy has reverted to the cached image digest sha256:b52be351ed7a588990686acf13385cfa29aca6da672b32723d44beefb09c8c93. The build log continues to show copy backup/package.json even though the backup folder has been completely removed from the project. The cached build plan on your infrastructure still references the old project structure and is not reflecting the current filesystem state. Can you please cycle the builders again so we get another uncached build? And is there a permanent fix that prevents the cache from reverting after each uncached build?
11 days ago
As I suggested above the way here is to set NO_CACHE=1 env variable. as I see you still didn't set it.
edgnkv
As I suggested above the way here is to set `NO_CACHE=1` env variable. as I see you still didn't set it.
11 days ago
NO_CACHE=1 is already set as a service variable and has been set for the duration of this issue. Yet the build log still shows the cached image digest sha256:b52be351... and copy backup/package.json referencing a folder that no longer exists. The variable is not preventing caching. What else can be done?
NO_CACHE=1 is already set as a service variable and has been set for the duration of this issue. Yet the build log still shows the cached image digest sha256:b52be351... and copy backup/package.json referencing a folder that no longer exists. The variable is not preventing caching. What else can be done?
11 days ago
No, it is not set. This change is pending, you need to click on Deploy on the popup window.
edgnkv
No, it is not set. This change is pending, you need to click on `Deploy` on the popup window.
11 days ago
NO_CACHE=1 is now confirmed working — we see 🚂 Caching Disabled in the build log and got a fresh image digest. However, the build log still shows copy backup/package.json referencing a folder that no longer exists in the project. A specific change we made to server.ts is confirmed present in the local file but is not appearing in the deployed server's output, suggesting the build plan is still compiling from an outdated file reference. Can you please regenerate the build plan for this service?
NO_CACHE=1 is now confirmed working — we see 🚂 Caching Disabled in the build log and got a fresh image digest. However, the build log still shows copy backup/package.json referencing a folder that no longer exists in the project. A specific change we made to server.ts is confirmed present in the local file but is not appearing in the deployed server's output, suggesting the build plan is still compiling from an outdated file reference. Can you please regenerate the build plan for this service?
11 days ago
There is not such option as re-generate build plan. Perhaps NO_CACHE=1 will not cache new stuff. I've cordoned the builder - can you try again to redeploy?
edgnkv
There is not such option as re-generate build plan. Perhaps NO_CACHE=1 will not cache new stuff. I've cordoned the builder - can you try again to redeploy?
11 days ago
Still no change in the deployed output despite caching disabled and a new builder. The critical observation: the snapshot hash has been sha256:7e072a7d93a60886813a8d71421c13f29b09ebf01c696df49edf4c676e0a9411 on every single deploy throughout this entire two-day issue — including after files were modified and the backup folder was deleted. The Railway CLI is not uploading the current filesystem state. It is sending the same stale cached snapshot on every railway up.
How do we clear the CLI's local snapshot cache on Windows?
Still no change in the deployed output despite caching disabled and a new builder. The critical observation: the snapshot hash has been sha256:7e072a7d93a60886813a8d71421c13f29b09ebf01c696df49edf4c676e0a9411 on every single deploy throughout this entire two-day issue — including after files were modified and the backup folder was deleted. The Railway CLI is not uploading the current filesystem state. It is sending the same stale cached snapshot on every railway up. How do we clear the CLI's local snapshot cache on Windows?
11 days ago
Afaik, there is not CLI-side caching. What you might want to check is .gitignore/.railwayingnore files is there anything ignored that should not be.
edgnkv
Afaik, there is not CLI-side caching. What you might want to check is .gitignore/.railwayingnore files is there anything ignored that should not be.
11 days ago
The latest build log shows no fetched snapshot line at all — Railway did not upload or read from the local filesystem. The deployed image digest reverted to a previous one despite NO_CACHE=1 being active.
The snapshot hash has been sha256:7e072a7d93a60886813a8d71421c13f29b09ebf01c696df49edf4c676e0a9411 on every single deploy across two days, despite confirmed file changes including deletion of an entire folder. There is no .railwayignore file. The project folder contains exactly 4 files, 0 folders, 74.1 KB total: nixpacks.toml, package.json, server.ts, tsconfig.json. The CLI is version 4.59.0 and reports "Indexed", "Compressed", "Uploaded" on every run, but the snapshot hash never changes.
Most telling: every build log continues to show copy backup/package.json referencing a folder that was deleted from the project two days ago. Railway is reading from a source that does not match the current local filesystem.
The local server.ts contains the expected change, confirmed visually. Where is Railway pulling the build source from, and what could cause the CLI on Windows to produce an identical snapshot hash despite filesystem changes?
edgnkv
Afaik, there is not CLI-side caching. What you might want to check is .gitignore/.railwayingnore files is there anything ignored that should not be.
11 days ago
Following up — it has been five hours since our last exchange, and the issue remains unresolved.
Our service cannot deploy updated code.
We are on the Pro plan and this is blocking active development.
Any update on your investigation would be appreciated.
edgnkv
Afaik, there is not CLI-side caching. What you might want to check is .gitignore/.railwayingnore files is there anything ignored that should not be.
11 days ago
Seven hours without a response on a Pro plan with a blocked production deployment.
We need escalation.
Can this be assigned to someone who can action it today?
edgnkv
Afaik, there is not CLI-side caching. What you might want to check is .gitignore/.railwayingnore files is there anything ignored that should not be.
11 days ago
Issue resolved. The root cause was entirely on our end — we had two project folders (LUCID and LUCID2) and the terminal was running railway up from the old LUCID folder while all file edits were being made in the LUCID2 folder. Railway was correctly deploying the old unchanged files the entire time. Once we ran railway up from the correct directory, the deployment succeeded immediately with a fresh image digest and the expected output.
Thank you for your patience and assistance throughout this investigation.
Status changed to Solved chandrika • 10 days ago