previous deployment crash

2 years ago

When I deploy the API (express, nodejs, socketio) everything goes well,
but right before the new deployment will replace the old one,
I see on the dashboard and I get an email notification
**Deployment crashed for [service name] in [project name]! **
why this happens? is there something I can do?
project id: b799250d-dd7b-47bb-be65-0ae878afd304

36 Replies

2 years ago

did the deployment actually crash or is it a false report? I don't think you specified this information


2 years ago

I think it's a false report. however it's slightly annoying.
The new deployment is completely fine, deployed, works, and never crashes.
But the previous one, when it's killed by railway, marked as CRASHED in the control panel and I also get an email about this.
with the next deployment it happens again.
unfortunately, there are no logs for previous deployments or explanation the source of the crash.


2 years ago

is your app handing sigterm properly?


2 years ago

I guess, it did not. maybe this is what I am looking for


2 years ago

its not like you are using nodemon or pm2 right?


2 years ago

no, I do not use nothing external. pure railway



2 years ago

what do the logs say of that crashed service?


2 years ago

nothing in logs,
yarn run v1.22.22
$ node index.js
Connection has been established successfully.
server started at 3040


2 years ago

those are logs from the service that was marked as crashed?


2 years ago

yes


2 years ago

found something like How do I shut down my Express server gracefully…
trying this. thanks for the tip


2 years ago

let me know how that goes!


2 years ago

but, are you on the legacy or v2 runtime? check your service settings


2 years ago

the service/project was created last month. I did not choose the runtime, and I hope that by default it's latest


2 years ago

well could you please check


2 years ago

I do not know where yo see it


2 years ago

the service settings


2 years ago

aha.. legacy


2 years ago

shall I change it?


2 years ago

you can try it


2 years ago

```process.on('SIGTERM', shutDown);
process.on('SIGINT', shutDown);

function shutDown() {
console.log('Received kill signal, shutting down gracefully');
server.close(() => {
console.log('Closed out remaining connections');
process.exit(0);
});

setTimeout(() => {
    console.error('Could not close connections in time, forcefully shutting down');
    process.exit(1);
}, 10000);

}```

neither that code, nor v2 did not help.
okay, I will try to make some experiments and come back later


2 years ago

do you see the Received kill signal, shutting down gracefully logs on the crashed service?


2 years ago

no 😦


2 years ago

back to the drawing board


2 years ago

okay, locally it catches sigterm and sigint.
if I run the api locally and kill it somehow, I get in the console
yarn start yarn run v1.22.21 $ node index.js Connection has been established successfully. server started at 3040 Shutting down gracefully...

on Railway this message does not appear.

https://expressjs.com/en/advanced/healthcheck-graceful-shutdown.html


2 years ago

what is your start command on railway?


2 years ago

just yarn start


2 years ago

thanks, what is your start script?


2 years ago

{
    "$schema": "https://railway.app/railway.schema.json",
    "build": {
        "builder": "NIXPACKS",
        "watchPatterns": ["/apps/ai-api"]
    },
    "deploy": {
        "numReplicas": 1,
        "startCommand": "yarn --cwd apps/ai-api start",
        "healthcheckPath": "/",
        "sleepApplication": false,
        "restartPolicyType": "ON_FAILURE",
        "restartPolicyMaxRetries": 10
    }
}

2 years ago

what is the start script for apps/ai-api?


2 years ago

"scripts": {
"dev": "node --env-file=.env.local index.js",
"start": "node index.js"
},


2 years ago

I don't think yarn is passing signals down, can you have railway start index.js directly with node?


2 years ago

ok


2 years ago

@Brody I owe you a beer. Resolved. Thanks a lot!


2 years ago

no problem!


Loading...