a month ago
Hi guys!
I have facing with a very annoing problem. On the same commit sometimes I can see successful building and deploying and sometimes not – it's like a 50% chance that all will be working correctly on the each new commit...
And the most annoying part that I can't see anything usefull from the build log... Just attached the today's example
Attachments
5 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
a month ago
This does not look like a normal frontend, pnpm, or Dockerfile build failure.
Since the exact same commit sometimes builds successfully, this is unlikely to be a deterministic TypeScript, Vite, pnpm-lockfile, or Dockerfile problem. A deterministic code failure should normally fail every build of the same source and environment in the same way.
Railway had similar cases where a specific Metal builder or the build queue accepted a job but failed to execute it or return the actual logs.
I would recommend:
- Cancel any currently queued or hanging deployments. Do not repeatedly enqueue more builds, since Railway has previously warned that this can add queue pressure.
- Trigger one fresh deployment.
- Compare the builder ID in a successful build with the builder ID in the failed builds.
- Provide Railway (so them can look) with:
- one successful deployment ID;
- two or more failed deployment IDs for the same commit;
- their timestamps;
- service region;
- builder IDs from each build.
Also recommend to try build it locally using docker repeatedly.
If that succeeds consistently, and Railway also sometimes succeeds with the exact same commit, changing application code or removing --frozen-lockfile would only hide the infrastructure problem rather than fix it.
ve-jo
This does not look like a normal frontend, pnpm, or Dockerfile build failure. Since the exact same commit sometimes builds successfully, this is unlikely to be a deterministic TypeScript, Vite, pnpm-lockfile, or Dockerfile problem. A deterministic code failure should normally fail every build of the same source and environment in the same way. Railway had similar cases where a specific Metal builder or the build queue accepted a job but failed to execute it or return the actual logs. I would recommend: 1. Cancel any currently queued or hanging deployments. Do not repeatedly enqueue more builds, since Railway has previously warned that this can add queue pressure. 2. Trigger one fresh deployment. 3. Compare the builder ID in a successful build with the builder ID in the failed builds. 4. Provide Railway (so them can look) with: - one successful deployment ID; - two or more failed deployment IDs for the same commit; - their timestamps; - service region; - builder IDs from each build. Also recommend to try build it locally using docker repeatedly. If that succeeds consistently, and Railway also sometimes succeeds with the exact same commit, changing application code or removing --frozen-lockfile would only hide the infrastructure problem rather than fix it.
a month ago
Thank you for your recommendations!
Cancel any currently queued or hanging deployments. Do not repeatedly enqueue more builds, since Railway has previously warned that this can add queue pressure.
Trigger one fresh deployment.
Usually I try exactly this approach, but it doesn't help too much. Sometimes build is failed for the commit in the evening and successfully will work next morning.
I have seen some strange moment. I see that the job is working too long and abort the build, but when I click "Deploy latest commit" in the Command + K panel - the new build job looks like continue the previous too long command execution – maybe that's why just to restart building doesn't work...
Compare the builder ID in a successful build with the builder ID in the failed builds
Thank you! I will do that, I suspect, that the problem can be in the worker or some limitations of the platform
Also recommend to try build it locally using docker repeatedly.>
Locally both pnpm --filter frontend build and pnpm --filter bravolisa-backend build works very well - it's done in several seconds successfully all the times
Attachments
a month ago
This is the very fresh example
The same commit but successful build – https://railway.com/project/a87471e5-a07b-4bc6-a436-ebf40a33046a/service/db9e8d8e-7483-4b5e-a71e-af9773b0e410?environmentId=56f15fb3-19bd-4f37-a004-076ba28d15e1&id=45d00a46-59d7-481a-8cc5-e63c57eccec1#details
Looks like the run was on the same machine – "scheduling build on Metal builder "builder-ebmbnt""
mayori
Try disable the build cache, add `NO_CACHE=1` to your environment variable
a month ago
Thank you, will experiment with this variable
Status changed to Open brody • 21 days ago
