RAILWAY_DEPLOYMENT_DRAINING_SECONDS and debugging graceful shutdowns
timbo-tj
PROOP

2 years ago

Hi there, I saw that I could set RAILWAY_DEPLOYMENT_DRAINING_SECONDS to give my change an app to shut down gracefully.

I have some logic in there that works fine on Heroku - I can see my dynos cleaning up and shutting down when I scale down my dynos. I see all my clean up logs and messages.

When I try the same on Railway, I don't see any logs. I am not sure if my app is gracefully shutting down. I am not sure how to debug this.

61 Replies

timbo-tj
PROOP

2 years ago

N/A



brody
EMPLOYEE

2 years ago

are you catching sigterm?


timbo-tj
PROOP

2 years ago

yes, should be, i use enableShutdownHooks in my nest app which by default catches…

SIGHUP = 'SIGHUP', SIGINT = 'SIGINT', SIGQUIT = 'SIGQUIT', SIGILL = 'SIGILL', SIGTRAP = 'SIGTRAP', SIGABRT = 'SIGABRT', SIGBUS = 'SIGBUS', SIGFPE = 'SIGFPE', SIGSEGV = 'SIGSEGV', SIGUSR2 = 'SIGUSR2', SIGTERM = 'SIGTERM',


brody
EMPLOYEE

2 years ago

do you see logs during regular app activity?


timbo-tj
PROOP

2 years ago

yeah I see my app working normally, and when I restart or redeploy my service it just 'cuts out' where it is and then the new one boots up


brody
EMPLOYEE

2 years ago

okay thanks for the info, ill try to get a proof of concept working


timbo-tj
PROOP

2 years ago

ok that would be fantastic, thanks. I thought maybe it is working in the background but the logs aren't being captured anymore? But i have no way of verifying it without seeing the logs


brody
EMPLOYEE

2 years ago

one idea popped into my head, what logger are you using?


timbo-tj
PROOP

2 years ago

just the default one what nest comes with


brody
EMPLOYEE

2 years ago

have you disabled log buffering?


timbo-tj
PROOP

2 years ago

just checking, sorry


timbo-tj
PROOP

2 years ago

i dont specify, and I can see that the default for bufferLogs is false


brody
EMPLOYEE

2 years ago

sounds good, i shall get back to my tests


timbo-tj
PROOP

2 years ago

<:salute:1137099685417451530>


timbo-tj
PROOP

2 years ago

i will let you know if i discover anything, i am poking around too


timbo-tj
PROOP

2 years ago

i wonder if it is because i am using newrelic and starting my app that way, ie node -r newrelic dist/apps/ssr-game-server/main


timbo-tj
PROOP

2 years ago

maybe the sigterm is not being propogated properly


brody
EMPLOYEE

2 years ago

that is very possible


brody
EMPLOYEE

2 years ago

i wouldn't know if newrelic forwards the signals along, are you doing the same on heroku?


timbo-tj
PROOP

2 years ago

no, so it very well might be the culprit


brody
EMPLOYEE

2 years ago

try without it


timbo-tj
PROOP

2 years ago

roger


brody
EMPLOYEE

2 years ago

it does work, i set it to 15, the default being 3

1216991492300603392


brody
EMPLOYEE

2 years ago

any luck taking newrelic out of the equation?


timbo-tj
PROOP

2 years ago

no luck here, hmm


brody
EMPLOYEE

2 years ago

nixpacks or dockerfile?


timbo-tj
PROOP

2 years ago

nixpacks


brody
EMPLOYEE

2 years ago

iirc nixpacks wraps the start command in a shell, that also might not be passing down the signals


timbo-tj
PROOP

2 years ago

hmm…


brody
EMPLOYEE

2 years ago

how hard would it be to write a dockerfile for your app? it would give you much better control and allow you to ensure the node process is started in exec mode


timbo-tj
PROOP

2 years ago

I have no experience in it, so honestly I have no idea! I am transitioning from Heroku to Railway, solo indie game dev wearing so many hats, haha. My experience is limited. Though I am not doing anything fancy. It's just a boring old nodejs/nest app


timbo-tj
PROOP

2 years ago

end of day now.. tomorrow problem…


timbo-tj
PROOP

2 years ago

thanks for all the help thus far


brody
EMPLOYEE

2 years ago

no problem, ill see if i can replicate this issue when starting my app wrapped in a shell like how nixpacks starts node apps


brody
EMPLOYEE

2 years ago

there is an issue on nixpacks about this
but neither me or dush can reproduce, and i've tried wrapping my app in a bash and sh shell.
i think it might still be worth a try to write a dockerfile, i can try to help you with that if you'd like?


timbo-tj
PROOP

2 years ago

ok, back in the office…!


timbo-tj
PROOP

2 years ago

i'd love a hand with that, sounds like a good test considering the open issue


brody
EMPLOYEE

2 years ago

for me to have a good chance at writing a working dockerfile for you, please share your most recent build log, anything you have changed nixpacks or service settings wise, and your repo


timbo-tj
PROOP

2 years ago

Hey thanks for offering, i did some more digging and I belive i've found the root cause


brody
EMPLOYEE

2 years ago

oh yeah? do tell!


timbo-tj
PROOP

2 years ago

but having a hard time verifying as there seems to be some issues with logs right now..? I have a scattering of logs missing from obervability


brody
EMPLOYEE

2 years ago

hmmm I haven't seen any other reports of that, what kind of logs are you missing?


timbo-tj
PROOP

2 years ago

my deploy logs for a fresh deploy look like this:

  • nothing

  • business as usual

It's missing the typical 'Nest app started' etc. Its not even visible in the Observability panel when i narrow down to the specific service …

1217300620575248400


brody
EMPLOYEE

2 years ago

incorrect log level? are we sure next doesn't buffer logs by default?


timbo-tj
PROOP

2 years ago

my typical logs look like for the same service

1217300946745561000


timbo-tj
PROOP

2 years ago

mmm pretty certain, and I haven't touched that


timbo-tj
PROOP

2 years ago

anyway, lets ignore that for now, I did observe a log that shows my shutdown signals are working


brody
EMPLOYEE

2 years ago

what was the fix?


timbo-tj
PROOP

2 years ago

1217301291932455000


timbo-tj
PROOP

2 years ago

though, i mentioned the log thing because this is just a few of my graceful shutdown logs haha (I have some logs missing within the same function that those 2 are from hah)


timbo-tj
PROOP

2 years ago

the fix was not to use npm run ... and use node ... instead



timbo-tj
PROOP

2 years ago

so for me it was a matter of changing my custom start command from

npm run start-game-server:newrelic

to

node -r newrelic dist/apps/ssr-game-server/main


timbo-tj
PROOP

2 years ago

which is just what the start-game-server:newrelic command is, anyway


brody
EMPLOYEE

2 years ago

I thought you where using node -r already?


timbo-tj
PROOP

2 years ago

no, i had a custom start command as i have a mono repo and had to specify which to run


brody
EMPLOYEE

2 years ago

ah I see


timbo-tj
PROOP

2 years ago

anyway thanks for helping me trouble shoot 🙂


brody
EMPLOYEE

2 years ago

happy to help where I can


brody
EMPLOYEE

2 years ago

if you are still having issues with logs, please open another thread! 🙂


Loading...