Hi there,

Thank you for the feedback, which we are taking into consideration as we expand Railway's monitoring capabilities.

For clarity, do you have "Crashed Deploys" enabled in your Notification?

Regards,

Christian

I mean yup I have that enabled but that only refers to when a deployment fails, not when a server or deployment actually crashes (after a successful deploy). So it's actually completely different to what's being discussed in the thread (and I think "crashed deploy" is therefore a misleading title to use there in the settings)

Hey Dan,

Can you expand more then on what you mean when you mention that it only applies to your application when the deployment fails? We do send an email for every non-zero exit code that incurs a crash. Can you tell me more about how your application goes into a ungraceful state and how you would like to be notified of it?

- Angelo

Hi - Railway does not appear to send emails for non-zero exit codes. I just tested by manually causing an uncaught exception which caused an exit code of 1 and did not receive an email. Additionally, our servers have crashed many times and we have never received any such email or alert from Railway - hence the reason for this thread.

Can you check again - and send a screenshot or a link to the set-up for this, to demonstrate?

Wait... I have a question, do you know what your email preferences are set to? I don't think this is not working because I just got spammed by my own service with a non-zero exit code.

Hey Angelo - yup, I've reviewed my settings as discussed in the thread. The only relevant-looking setting is "Crashed Deploys", but this only sends an alert when the process of deployment fails (not when a server is successfully deployed, and later crashes), as mentioned. This is why I asked for a screenshot or link to the setting in question, maybe a screenshot to the email you're referring to!

Wait... last piece of debugging. Are you using pm2?

Nope, just a standard Node server. In another case, a next.js project with a custom Express server. (But neither of them alert about crashes)

I noticed that when our server gets restarted because of a new deploy, an error always appears in the logs of an "ungraceful shutdown". I tried to capture that in Node code (by listening for SIGTERM/SIGINT) and shut down gracefully but Node never captures it. Maybe that's related. Something to do with the infrastructure/containers Railway sets up.

Do you have a repo/Dockerfile I can reference so I can try to re-pro, finally have time to look at this further. (And end the feet dragging)

Thanks for looking into it! I'll attach a simple index.js and package.json for a Node/Express server to reproduce this. Deploy this to Railway, visit the URL it's hosted on or send any kind of HTTP request - and it will count to 100 and then crash. (I also used this to demonstrate bugs with logging but it's still a super minimal repo).

I also currently have this deployed at https://expressjs-log-debugging-demo.up.railway.app/ if that helps.

package.json:

{
  "name": "railway-debugging",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "start": "node --experimental-specifier-resolution=node ./index",
    "server": "nodemon --experimental-specifier-resolution=node --watch . --watch ./index"
  },
  "engines": {
    "node": "18.19.0"
  },
  "type": "module",
  "dependencies": {
    "body-parser": "^1.20.2",
    "compression": "^1.8.0",
    "cors": "^2.8.5",
    "dotenv": "^16.3.1",
    "express": "^4.18.2",
    "jsonwebtoken": "^9.0.2",
    "mongoose": "^7.5.1",
    "morgan": "^1.10.0"
  },
  "devDependencies": {
    "eslint": "^8.28.0",
    "eslint-config-next": "13.1.6",
    "eslint-config-react-app": "^7.0.1",
    "eslint-import-resolver-alias": "^1.1.2",
    "eslint-plugin-import": "^2.28.1",
    "nodemon": "^2.0.22"
  }
}

index.js

import 'dotenv/config'
import express from 'express'
import logger from 'morgan'
import bodyParser from 'body-parser'
import cors from 'cors'

const port = process.env.PORT || 3005;
const app = express();

app.use(cors());
app.use(logger('dev'));
app.use(bodyParser.json({ limit: '10mb' }));
app.use(bodyParser.urlencoded({ limit: '10mb', extended: false }));

function wait(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
}

app.use('*', async (req, res) => {
    const { objectToLog, countMax = 100 } = req.body
    
    res.send({ status: 'processing' })
    
    console.log(objectToLog)
    
    for(let i=1; i<= countMax; i++) {
        console.log(i + '/' + countMax)
    }
    
    throw new Error('💥 Test crash triggered')
})

app.listen(port, async () => {
    console.log('server is running on', port);
})

Wait uh... are you deploying nodemon to production?

(You likely aren't but checking.)

Did a pass, no nodemon.