2 months ago
I have a service that automatically shuts down (gracefully, with an exit code 0) when memory usage reaches a certain threshold. I have Railway set to always restart. It works most of the time, but sometimes Railway will not pick it up and the service will go down silently. It has happened twice now, in different weeks. I have to manually restart the service to bring it back up.
When the issue happens,the last line in the logs is always: "Mounting volume on: ..." then it just sits there.
I cannot use Serverless because my server runs background jobs. Serverless is also not working correctly with the constant 502s, but as that's expected from the feature it's just not suitable for my case.
Any suggestions on how to fix this? Is the number of restarts (limited to 10 when restarting on crashes) also applying to normal restarts? My app is not crashing at all so I'd think that limit would not apply.
7 Replies
2 months ago
Your service's restart policy is currently set to "On Failure" (not "Always"), which only restarts on non-zero exit codes. A graceful exit with code 0 will not trigger a restart under that policy, which explains why the service stays down after your memory-threshold shutdown. Change the restart policy to "Always" in Service Settings > Deploy, and note that your config-as-code file at /api/config/railway.toml can override dashboard settings, so check there as well. The 10-retry limit applies only to "On Failure" restarts and does not cap "Always" restarts on paid plans.
Status changed to Awaiting User Response Railway • about 2 months ago
Railway
Your service's restart policy is currently set to "On Failure" (not "Always"), which only restarts on non-zero exit codes. A graceful exit with code 0 will not trigger a restart under that policy, which explains why the service stays down after your memory-threshold shutdown. Change the restart policy to "Always" in Service Settings > Deploy, and note that your config-as-code file at `/api/config/railway.toml` can override dashboard settings, so check there as well. The 10-retry limit applies only to "On Failure" restarts and does not cap "Always" restarts on paid plans.
2 months ago
My confilg file has this line:
restartPolicyType = "ALWAYS"
Isn't that the correct syntax?
Status changed to Awaiting Railway Response Railway • about 2 months ago
2 months ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 2 months ago
2 months ago
Can you clarify "restart the worker process inside the container"?
The app is a Bun webserver and all I do is close the server gracefully and then process.exit(0), then Railway picks it back up.
2 months ago
Update: in one of the restarts, the "Mounting volume" log didn't even show up. The app just went down with no visible restart attempt from Railway.
I've implemented a shell wrapper as a workaround, but this is the second feature of Railway that I can no longer rely on. Serverless (formerly App Sleep) was the first one with the "occasional 502s on boot" that were just accepted as WAI (it's even mentioned in the docs) and from a customer standpoint I find this completely ruins the feature. I hope Railway finds ways to fix the auto-restart instead of just accepting "this is expected to happen".
2 months ago
How are you running you app? iirc there is an issue with wrapper processes not returning exit codes correctly, i.e. npm run vs node index.js (not sure if there's a Bun equivalent, but something worth checking).
Could you also make sure that restartPolicyType is correctly under the "deploy" key/section of your config file, and that it's correctly registered against the service? iirc if it's not called railway.toml/json (and called something else), it won't be automatically detected, so you have to set the path in the dashboard.
thaumanovic
How are you running you app? iirc there is an issue with wrapper processes not returning exit codes correctly, i.e. `npm run` vs `node index.js` (not sure if there's a Bun equivalent, but something worth checking). Could you also make sure that `restartPolicyType` is correctly under the "deploy" key/section of your config file, and that it's correctly registered against the service? iirc if it's not called railway.toml/json (and called something else), it won't be automatically detected, so you have to set the path in the dashboard.
2 months ago
The setting is correct. As stated in the OP, the issue only happens occasionally, meaning every time it worked the policy was applied.
The wrapper is a .sh file that runs a Bun compiled executable (it doesn't use bun to run it). The exit codes work correcly, again proved by the most times it works. If the exit code was different than zero during one of those failed restarts, Railway would show the deploy as crashed, which is not the case. It just sits there, offline (but Railway shows it as online), until I have to restart it manually.
I believe this is an issue on Railway's side.
2 months ago
Update: the wrapper approach failed. Bun is not the target of the memory cleanse, but the entire container -- the memory that I need to free is from system's page cache, which Railway charges for.
New workaround: I deployed a separate, tiny, always-on Rust api that is notified when the main app goes down gracefully (no notify when the app goes down after a signal, e.g., after a new deploy or when Railway needs to run maintenance), then try to ping it after a grace period. if it's down, it means Railway failed to restart it. In that case, the tiny app calls the Railway API to restart the container manually and bring it back up.