hey,

one thing that's easy to miss ; the port tied to your custom domain in railway might not match the one your app is actually running on. go to your service settings, remove the custom domain completely, then re add it and make sure you select the correct port when doing so. also double check that the port under your custom domain matches the one under your railway-generated domain

Ok, attempting to reduce the complexity of the situation. I first tried your suggestion but same result and issue.

So I removed the Custom Domain just to eliminate any potential issues with that or Cloudflare. Still same issue and results.

Ok, attempting to reduce the complexity of the situation. I first tried your suggestion but same result and issue.

okay , can you share your deploy logs? ,i think theres usually a clear error in there

Not much there:

so the container starts and kicks off node dist/src/main but there's no "listening on port" message after that , which means your app is either crashing silently on startup or never actually binding to the port. can you check your main.ts file and confirm that when you call app.listen() you are passing process.env.PORT and 0.0.0.0 explicitly? something like app.listen(process.env.PORT, '0.0.0.0'). if the port is hardcoded or the host is missing that would explain everything

One thing that could possibly be causing this issue is if you mapped a root domain (e.g., yourdomain.com) instead of a subdomain (e.g. www.yourdomain.com or api.yourdomain.com), Cloudflare automatically flattens the CNAME into an A record. If you are trying to use the root domain, switch your setup to a subdomain (like www or api), then add the subdomain in Railway, and create the corresponding CNAME in Cloudflare.

I have dropped all of Custom Domain stuff from the configuration and I have confirmed that locally I am seeing logs like this:

15:10 $ node dist/src/main

2026-03-23T22:10:41.607Z info: [NestFactory] Starting Nest application...

2026-03-23T22:10:41.607Z info: [InstanceLoader] DatabaseModule dependencies initialized

2026-03-23T22:10:41.607Z info: [InstanceLoader] QueueModule dependencies initialized

2026-03-23T22:10:41.608Z info: [InstanceLoader] BullBoardModule dependencies initialized

But I am still getting 502 errors from Railway. Completely baffled now...

Also, this is in my main.ts:

    const port = process.env.PORT ? parseInt(process.env.PORT) : 4000;
    const host = `0.0.0.0`;
    await app.listen(port, host);

your main.ts is fine ; but look at the difference , locally you see all the nestjs module initialization logs (Databasemodule, QueueModule etc.), but on railway the logs stop completely right after node dist/src/main with none of that. your app is crashing on railway before it even reaches the initialization phase. can you check your railway service variables? are all the env vars your app needs actually set there (db connection, queue config, etc.)? that silent crash pattern is usually a missing or misconfigured env var on railway

BEFORE locally, it ran but the logs were shortened as you pointed out. Working through some configuration changes that were hijacking the logs and committed them.

NOW the same command works great and dumps the logs locally (both in the Console and in a rotating log file). I have now removed the rotating log file and only maintain the console output yet it still will not show anything of value in the Deploy Logs.

I am so confused because the ENV Vars are all there, the values are populated correctly, and locally it is very chatty so I am stuck on the next steps to debug.

I have completely stripped out the Custom Domain at this point to just get it working but I will come back to this when I am ready to add it back in. Appreciate the thought

quick question , after you committed those changes, did you trigger a new deploy on railway? railway won't pick up new code automatically unless it's connected to your repo and you pushed, or you manually triggered a redeploy. if railway is still running the old build it would explain why the logs are still silent there even though everything works locally now

Good thought, I am connected to my GitHub repo so I see the commit message for the changes in the Railway so I know it is pulling the right hash. Though considering the depths of the search, I am going to delete all of the deployments and have it just repull and deploy fresh right now.

Unfortunately same thing with this reference ID:

Request ID:

xwxuxhnGSIaY-prmnpoFkQ

What I cant figure out is that it was running without an issue last week. I locally updated some packages including NestJS framework and then committed them and this happened (also tried the Custom Domain as well). Very little was done other than that. I had another issue around the watcher package within Railway after the updates as well so I am thinking its the updates but I am not seeing errors showing that.

okay that's a key detail , it broke right after the nestjs package updates. since your deploy logs are still silent on railway (no initialization logs) but work fine locally, the update likely introduced something that fails specifically in railway's environment.

two things to check now:

first, go to your http logs tab on railway, find a 502 entry and share what's in the responseDetails and upstreamAddress fields , that will show us exactly why railway's proxy can't reach your app

second, check if your package-lock.json was committed alongside the update , if not, railway might be resolving different package versions than what you have locally and that could explain why it works locally but not on railway

Ok, attached are the logs of one of the 502 errors:

{ "requestId": "JdZoG7EpR4mlHgvbLPU1MQ", "timestamp": "2026-03-23T23:23:12.096621969Z", "method": "GET", "path": "/docs", "host": "mentha-backend-staging.up.railway.app", "httpStatus": 502, "upstreamProto": "", "downstreamProto": "HTTP/2.0", "responseDetails": "Retried single replica", "totalDuration": 8, "upstreamAddress": "", "clientUa": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36", "upstreamRqDuration": 6, "txBytes": 4682, "rxBytes": 782, "srcIp": "45.26.36.200", "edgeRegion": "us-west2", "upstreamErrors": "[{\"deploymentInstanceID\":\"8e39ff99-2e7e-434f-b05c-1dc8099b695d\",\"error\":\"connection refused\",\"duration\":1},{\"deploymentInstanceID\":\"8e39ff99-2e7e-434f-b05c-1dc8099b695d\",\"error\":\"connection refused\",\"duration\":3},{\"deploymentInstanceID\":\"8e39ff99-2e7e-434f-b05c-1dc8099b695d\",\"error\":\"connection refused\",\"duration\":2}]" }

I am using pnpm and I have deleted and removed the pnpm-lock.yaml file and had it regenerated as part of all of this debugging. Just deleted it and regenerated it and it is the exact same as whats in the repo.

upstreamAddress is empty and you're getting connection refused on every retry so your app is not binding to any port on railway at all, meaning it's crashing before it even reaches app.listen(),this matches your deploy logs which show zero nestjs initialization

since this started after the nestjs update and works fine locally, something in the update is causing a silent crash specifically on railway. can you check your railway deploy logs scroll all the way, is there absolutely nothing after node dist/src/main, not even an error?

second thing to do is to run node dist/src/main locally with your railway env vars set (not your local ones) , does it still start up fine?, this test will likely reproduce the crash and show the actual error

Screenshot of the Deploy Logs

okay so the app crashes immediately and silently right after node dist/src/main with zero output. no error, nothing

given you updated nestjs packages: your dist/ folder might not have been rebuilt after the updates. if you committed the old compiled dist/ to your repo, railway is running stale compiled code that doesn't match your updated packages , make sure your railway build step is actually running pnpm build before pnpm start:prod check your package.json start:prod script and your railway build command , if railway is skipping the build step it would explain this

I followed all of the commands and copied over config for the ENV file to my local machine and the system builds and runs without issue. I am able to connect to the Railway database and the Redis instance. Nothing is making sense at this point. There are no errors on startup and I am getting the log outputs without issue.

Also, the package.json is simply running "node dist/src/main" (without anything else) so I cant imagine it is stale builds

if your start command is just node dist/src/main with no build step, railway is running your committed dist/ folder as-is. is your dist/ folder committed to your repo? if yes, was it rebuilt locally after the nestjs package updates before you committed? if the compiled dist/ was built with the old packages and never rebuilt after the update, railway would be running mismatched compiled code which could cause exactly this silent crash

is your dist/ folder committed to your github repo (i m jst asking if it is in your .gitignore or not)?

the dist/ is not committed to the repo

also what does your railway build command look like in your service settings (under settings > deploy)?

Attached is the build logs

since dist/ is not in your repo, railway has to build it before starting. can you go to your railway service settings under the deploy section and tell us what your build command is set to? if there's no build command configured, railway would try to run node dist/src/main but dist/ wouldn't exist yet, which would cause exactly this instant silent crash with no error output

Attached is the screenshot of the Build Settings

okay do this if it works i will explain you what happend hahaha , add NO_CACHE=1 as an environment variable in your service variables tab, then redeploy. that tells railway to skip all cached layers including the pnpm install step and build everything from scratch. once it deploys successfully you can remove that variable again

Trying it now

okay , can you return to me faster bc i need to go to sleep :)

Absolutely... it is deploying now ... will respond as soon as the system does

Definitely taking longer on the build portion of the deploy (2x)

Same thing on the result. The build logs look the same, all calls are getting 502s, and no more logs than before

ok so the cache was not the issue , the one thing we haven't seen yet that could explain everything is the railway.json , can you share the contents of that file?

JSON below:

{
    "$schema": "https://railway.com/railway.schema.json",
    "build": {
        "builder": "RAILPACK",
        "buildCommand": "pnpm build"
    },
    "deploy": {
        "runtime": "V2",
        "numReplicas": 1,
        "startCommand": "pnpm start:prod",
        "sleepApplication": false,
        "useLegacyStacker": false,
        "ipv6EgressEnabled": false,
        "multiRegionConfig": {
            "us-west2": {
                "numReplicas": 1
            }
        },
        "restartPolicyType": "ON_FAILURE",
        "restartPolicyMaxRetries": 10
    }
}

did anything change in this railway.json file at the same time? worth checking your git history on that file specifically

It has only been added at the beginning. No other changes to the file.

okay so we've exhausted everything external the build is fine, env vars are fine, railway.json is fine the only thing we haven't been able to see is the actual runtime crash reason because the process dies silently with zero output

at this point the crash is happening so early and so silently that we need to force node to surface the error. can you temporarily wrap your bootstrap in main.ts like this:

async function bootstrap() {
  try {
    // your existing code
  } catch (err) {
    console.error('BOOTSTRAP ERROR:', err);
    process.exit(1);
  }
}
bootstrap();

commit and redeploy, if something is throwing during startup this will force it to print the actual error in your deploy logs

Trying that now

Since you last saw the logs, I added the DB migrations to ensure I wasnt failing at the DB connection on deploy so that is addition logs you see here, but as you can see nothing else:

?

I added the Try / Catch loop that you requested and just sent the Deploy Logs in another message

this is actually really useful, your migrations run fine on the first replica, which means your database connection is working. but the second replica that runs the actual app still crashes silently after "Starting Container".

did this deploy include the try/catch change in main.ts? because there's still no error output at all from that replica, which means either the try/catch wasn't in this build yet, or the crash is happening before node even executes your code , which would point to a missing or corrupted file in the built dist/

Yes, it included the Try/Catch

greaat the try/catch was there but printed nothing, which means node is crashing before it even reaches your bootstrap function. the crash is happening at the import stage, one of your imports at the top of main.ts or a module it loads is throwing or failing to resolve on railway specifically

can you add this at the very top of main.ts, above all your imports:

process.on('uncaughtException', (err) => {
  console.error('UNCAUGHT EXCEPTION:', err);
  process.exit(1);
});

commit and redeploy , this catches errors that happen before bootstrap runs and will finaaaaaaaaaaaally show us the actual error message

Doing the change and deploying now

Unfortunately, same thing ...

And the code was in there:

omg your first import is ./instrument , in the compiled js, that runs before your uncaughtException handler even gets registered, so if it throws, nothing prints. that's why we're getting zero output

can you check what's in src/instrument.ts? if it's sentry or opentelemetry, a package version change could be causing it to throw silently on railway . try temporarily commenting out import './instrument' and redeploying

if the app starts up, that's the culprit

Got it. It is Sentry ... trying the comment now

nice , let me know what happens

Same error but more logs in the Deploy Logs

okay so commenting out sentry made the app get much further, we can finally see logs , the punycode warning is harmless can you scroll down and share what comes after those last lines? the app might have fully started or crashed further down with an actual error message this time

There is nothing more. That is the full log

okayy , sentry was definitely part of the problem , , the punycode warning is harmless but the app is still not reaching nestjs initialization. since commenting out the sentry import helped but didn't fully fix it, is it possiblz to share the full src/instrument.ts file and the full src/main.ts file? there might be another import or something sentry-related being imported elsewhere that's still failing

Nothing in the files of any real relevance since it is still in the development configuration (hardened for prod coming later) so here they are:

Sorry had to change the extensions because it would not allow me to upload JS and TS files

since both your uncaughtException and bootstrap try/catch produce zero output, the crash is likely an unhandled promise rejection which is a different event entirely. can you add this right below your uncaughtException handler:

process.on('unhandledRejection', (reason) => {
    console.error('UNHANDLED REJECTION:', reason);
    process.exit(1);
});

commit and redeploy, i hooope this finally catch and print whatever is killing your app silently

Doing so now.

Sidenote: have you seen this level of a problem before?

Nothing new in the logs:

hahaha yes i’m a software engineer since 2019 and i’m getting used to this kind of debugging

I meant within Railway. Very strange that none of this is happening locally for me.

is there anything below that? can you scroll down or not?

Nothing below that

i can say confidently that your app isn't crashing it's hanging. that's why none of your error handlers print anything, there's no error to catch. the process just freezes after the punycode warning and it is killed externally after a timeout

something during module initialization is waiting forever and never resolving. this is usually a network connection that hangs , redis or the database trying to connect eagerly during module load rather than lazily

can you check your redis and queue module configs , specifically whether they try to establish a connection at startup? also can you confirm your redis env vars (REDIS_HOST, REDIS_PORT, REDIS_PASSWORD) are all set correctly in railway? a hanging redis connection during bullmq/queue module initialization would cause this

That was my suspicion too. I am using the same config locally to connect to the Redis instance and Postgres instance on Railway. I will experiment with what other tooling I have that may be a problem. I appreciate the help and want to let you go to bed. I will update the issue with what I find.

bullmq uses ioredis under the hood, and by default ioredis only does ipv4 lookups. railway's private network uses ipv6, so bullmq silently hangs trying to connect and never resolves, which is exactly what you're seeing

the fix is to add family: 0 to your bullmq connection config to enable dual stack lookup. in your nestjs bullmq module config it should look like this:

BullModule.forRoot({
  connection: {
    host: process.env.REDIS_HOST,
    port: parseInt(process.env.REDIS_PORT),
    username: process.env.REDIS_USERNAME,
    password: process.env.REDIS_PASSWORD,
    family: 0, // this is the fix
  }
})

I will give it a shot and check the Redis vars

okay see you tomorrow, good night

UNBELIEVEABLE! The entire issue: REDIS_PASSWORD="${{Redis.REDISPASSWORD}}}}" which render the password with a trailing '}}' and nothing through an error or notice at all when it tried to connect. I only noticed it when I deliberately looked at the value filled in on the Variables tab.

I am so sorry for basically wasting your time.