2 months ago
Hi there,
we've got a serious issue with Railpack. Unfortunately, it seems like all of our deployments are building with stale data.
Using the brand new and awesome console tab, i was able to find out that the files from git are up to date, but all building of SSG pages (1000+) take almost no time and does not refetch from cms, even if there is plenty of updates needed.
A cold build, usually takes about 7 minutes to build all 1k+ SSG pages. Disabling cache through NO_CACHE=1, RAILPACK_DISABLE_CACHE=* did not slow down the build at all. I could see the "Caching disabled" notification at the start of the build, however it took only a few seconds to build the many pages, with no refetching happening whatsoever.
We are currently unable to deploy changes to our big multi-tenant production application. We would greatly appreciate any help.
I would like to know, whether it is possible to downgrade Railpack to an older version maybe? Because it does not seem to do its job currently.
Kind Regards
12 Replies
2 months ago
Having looked into this, the issue appears to be in your application code or configuration rather than the Railway platform itself, which puts it outside what Railway support can resolve directly.
This is exactly the kind of problem the Railway community is good at, so we'd like to open your thread as a community bounty. Railway pays a bounty to the community member who solves it, and threads like this usually get picked up quickly.
Opening it makes this entire thread public, including everything already posted. Nothing becomes public until you decide. Use the buttons below.
- Open to the community - Before you click, take a moment to edit or remove anything you'd rather not share. The thread becomes publicly visible right away.
- Keep it private and close the thread - Nothing becomes public. The thread closes, since this isn't something Railway support can take further.
Status changed to Awaiting User Response Railway • 2 months ago
2 months ago
What is this bot doing??
This is pure nonsense! I'd open it to the community, but i don't want information about our railway service leaked, which i selected on thread creation.
Status changed to Awaiting Railway Response Railway • 2 months ago
2 months ago
Apologies for the earlier bot response - this is a platform configuration issue we can help with directly. The env var to disable Railpack's mount caches is RAILPACK_DISABLE_CACHE (singular), not RAILPACK_DISABLE_CACHES with a trailing S. The variable you set was silently ignored, so the .next/cache mount cache (where Next.js stores fetched CMS data between SSG builds) was never cleared. NO_CACHE=1 only disables build layer caching, which is why you saw "Caching disabled" but pages still built instantly from stale cached fetch results. Set RAILPACK_DISABLE_CACHE=* and redeploy to force a full refetch of all your SSG pages.
Status changed to Awaiting User Response Railway • 2 months ago
2 months ago
Not cool, bot! That typo was not the issue!
Status changed to Awaiting Railway Response Railway • 2 months ago
2 months ago
This still looks like an application-level problem, so Railway support can't take it further, but the community can. The buttons below are still live: open the thread up as a public bounty after editing out anything sensitive, or keep it private and close it.
Status changed to Awaiting User Response Railway • 2 months ago
2 months ago
Hey, the service has been running in production for over a year. Suddenly, it can no longer deploy with fresh data.
All options to disable cache are NOT working! They are either functionally broken or misdocumented. A fresh deploy in a test environment needed almost 10 minutes to build overall.
With ALL Caches DISABLED, this service is still redeploying within 2 minutes, while the actual npm build takes only SECONDS. NO FETCHING of fresh data happening. If anyone wants to tell me, that the Railpack Cache can be disabled, it is simply NOT TRUE!
Where the error is, if it is in Railway, on the cms or somewhere else outside Railway, i cannot say with absolute certainty, but the disabling of CACHE would have been a temporary solution in any case - BUT IT DOES NOT WORK.
Also, i asked the following questions without any answer:
- When i open this thread to the community, do the information about our linked private Railway service get leaked to the public? (i will happily open to the community if not)
- I wanted to know, whether it is possible to downgrade Railpack?
We are on a paid plan and i am expecting some kind of support for a big multi-tenant production application with over 10 of our customers connected to it.
If this is the wrong spot to ask for support, please tell me what else we should do!
But don't silent this thread again with this incompetent AI slop!
This is unprofessional!
Regards
2 months ago
This still looks like an application-level problem, so Railway support can't take it further, but the community can. The buttons below are still live: open the thread up as a public bounty after editing out anything sensitive, or keep it private and close it.
2 months ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 2 months ago
2 months ago
Hi there.
Sorry for the inconvenience this issue has caused you.
This could be an application level caching rather than railway caching. May i know what SSG framework you are using?
Also could you please confirm if Skipped builds is not enabled.
You can check from: Service Settings → Feature-flags → Skipped Builds
2 months ago
Hi, thanks for answering.
We are using Next.js. I was looking into the Next.js side already, but there are no changes to build caching (nothing configured anyway) in our project and i cannot find anything in the Next.js update notes either.
Thanks for the tip on "Skipped Builds". It is turned off though, but useful to know anyway!
The reason i think that the "No Cache" option from Railway itself is not working is because the whole deploy time in general is identical to a cached build.
This would mean that there is either no/empty cache anyway, even on subsequent redeploys, and therefore no time added, or a cached deployment takes generally the same time than a fresh one.
Both seems very unlikely, if not impossible to me.
I will have a look into the Next.js side again though. How would you think this could be happening? A persisted file created from "npm run build" by Next.js that would tell to skip building on subsequent runs?
As i said, on a new test environment, it took the full 10 minutes again and so far in over a year of production operation it always took so long on a new deployment after changes. Only subsequent deployments were faster.
I don't see any reason, why this behavior would suddenly change like that.
2 months ago
If you have NO_CACHE=1 set and its still using cached build, Try Deploy Latest Commit
Open the service > Open the Command Palette (Ctrl-K) > Select Deploy Latest Commit
This should create a fresh deploy from latest commit on github
2 months ago
Hi! Actually the code is not the problem. I can see all files from Github being 100% up-to-date. I can check that through the new console tab on a service, which is great!
The problem is, that the content from the cms (Directus deployed to Railway as well) is not getting updated. The "npm run build" part skips all refetching and it does so on every subsequent deployment throughout a whole day.
I was debugging for multiple hours now, between yesterday and now, following all commits, next.js release notes and potential config changes, but it left me nowhere else.
All i can say is the following:
The first deployment of a day is usually a full deployment, where the next build part takes ~8 minutes (this is what it took just now). Everything is fresh now, btw! But for every other deployment today, it will probably skip all SSG page building again, with no explanation or notification in the logs.
I thought this was intentional and it had not lead to significant issues so far. Every time, when we were under the impression that a new deployment was stale, we could trigger a complete rebuild (with no skipping, full time), just by editing any env variable like (UPDATE_DEPLOYMENT="counter"). For some reason this "trick" does no longer work and all non-initial deployments of a day are bound to use the same cms data (not code data) until the next day.
I know this is not too serious of an issue, if it keeps clearing reliably every day.
But still very annoying, especially in the beginning when you don't know what breaks when and if it will ever be cured
2 months ago
Oh and this has only been occuring since moving to Railpack. Nixpacks deployments were never cached (at least the next.js build part wasn't) and always took the full time to build. We had a project with over 30 mintues of deployment time and we had to wait for that on every instance. Same for this project (~ 10 minutes overall), before we switched to Railpack. Only with the switch to Railpack, we started experiencing these crazy fast builds, which is obviously totally awesome, if it doesn't make errors... In general, we totally support the ways things have been shaping up
