Successful project building only in half of the cases
dimau
HOBBYOP

a month ago

Hi guys!

I have facing with a very annoing problem. On the same commit sometimes I can see successful building and deploying and sometimes not – it's like a 50% chance that all will be working correctly on the each new commit...

And the most annoying part that I can't see anything usefull from the build log... Just attached the today's example

Attachments

$10 Bounty

5 Replies

Railway
BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway about 1 month ago


ve-jo
HOBBY

a month ago

This does not look like a normal frontend, pnpm, or Dockerfile build failure.

Since the exact same commit sometimes builds successfully, this is unlikely to be a deterministic TypeScript, Vite, pnpm-lockfile, or Dockerfile problem. A deterministic code failure should normally fail every build of the same source and environment in the same way.

Railway had similar cases where a specific Metal builder or the build queue accepted a job but failed to execute it or return the actual logs.

I would recommend:

  1. Cancel any currently queued or hanging deployments. Do not repeatedly enqueue more builds, since Railway has previously warned that this can add queue pressure.
  2. Trigger one fresh deployment.
  3. Compare the builder ID in a successful build with the builder ID in the failed builds.
  4. Provide Railway (so them can look) with:
  • one successful deployment ID;
  • two or more failed deployment IDs for the same commit;
  • their timestamps;
  • service region;
  • builder IDs from each build.

Also recommend to try build it locally using docker repeatedly.

If that succeeds consistently, and Railway also sometimes succeeds with the exact same commit, changing application code or removing --frozen-lockfile would only hide the infrastructure problem rather than fix it.


ve-jo

This does not look like a normal frontend, pnpm, or Dockerfile build failure. Since the exact same commit sometimes builds successfully, this is unlikely to be a deterministic TypeScript, Vite, pnpm-lockfile, or Dockerfile problem. A deterministic code failure should normally fail every build of the same source and environment in the same way. Railway had similar cases where a specific Metal builder or the build queue accepted a job but failed to execute it or return the actual logs. I would recommend: 1. Cancel any currently queued or hanging deployments. Do not repeatedly enqueue more builds, since Railway has previously warned that this can add queue pressure. 2. Trigger one fresh deployment. 3. Compare the builder ID in a successful build with the builder ID in the failed builds. 4. Provide Railway (so them can look) with: - one successful deployment ID; - two or more failed deployment IDs for the same commit; - their timestamps; - service region; - builder IDs from each build. Also recommend to try build it locally using docker repeatedly. If that succeeds consistently, and Railway also sometimes succeeds with the exact same commit, changing application code or removing --frozen-lockfile would only hide the infrastructure problem rather than fix it.

dimau
HOBBYOP

a month ago

Thank you for your recommendations!

Cancel any currently queued or hanging deployments. Do not repeatedly enqueue more builds, since Railway has previously warned that this can add queue pressure.

Trigger one fresh deployment.

Usually I try exactly this approach, but it doesn't help too much. Sometimes build is failed for the commit in the evening and successfully will work next morning.

I have seen some strange moment. I see that the job is working too long and abort the build, but when I click "Deploy latest commit" in the Command + K panel - the new build job looks like continue the previous too long command execution – maybe that's why just to restart building doesn't work...

image.png

Compare the builder ID in a successful build with the builder ID in the failed builds

Thank you! I will do that, I suspect, that the problem can be in the worker or some limitations of the platform

Also recommend to try build it locally using docker repeatedly.>

Locally both pnpm --filter frontend build and pnpm --filter bravolisa-backend build works very well - it's done in several seconds successfully all the times

Attachments


a month ago

Try disable the build cache, add NO_CACHE=1 to your environment variable



mayori

Try disable the build cache, add `NO_CACHE=1` to your environment variable

dimau
HOBBYOP

a month ago

Thank you, will experiment with this variable


Status changed to Open brody 21 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...