10 days ago
Hi Railway team,
I'm running an n8n Pro Stack (Primary + Worker + Postgres + Redis) in production and ran into two related issues today I'd appreciate help diagnosing.
Issue 1: Primary crashed from OOM at ~950MB–1GB usage, despite a 24GB allocation
My Scale settings show Primary provisioned at 24 vCPU / 24GB memory (matching my plan limit). Today Primary crashed with a "Deploy Ran Out of Memory" alert. Checking the memory graph, usage spiked to roughly 950MB–1GB right before the crash — nowhere near the 24GB shown as allocated.
I also have NODE_OPTIONS=--max-old-space-size=8192 already set on this service (8GB heap allowance), so Node's own default heap ceiling (~1.5-2GB) shouldn't be the limiting factor either — the crash happened well under even that.
Given both the container allocation (24GB) and the Node heap flag (8GB) are far above where the actual crash occurred (~1GB), I'm not clear what's actually imposing that lower ceiling. Is there a separate resource limit on this service (or plan tier) that isn't reflected in the Scale panel?
Issue 2: Queued/webhook-triggered executions appear to be permanently lost after a crash, not resumed
Separately, earlier today I also hit a disk-full crash on my Postgres volume (resolved by resizing 500MB → 10GB). After recovering from both crashes and restarting the affected services, I have several hundred workflow executions that were queued (webhook-triggered, not manual) before the crashes. They're not resuming — Worker is healthy and processing new work, but these specific queued executions never picked back up, and n8n's own "queue recovery" process appears to have marked some as crashed/dangling during the cleanup.
Is there a way to recover or requeue executions that were in-flight/queued at the time of a crash, or is this expected behavior in queue mode? I want to make sure I'm not silently losing work on every crash going forward.
Service: Primary (primary-production-4d9c.up.railway.app)
Project: n8n Pro Stack
Today's timeline: Postgres volume full crash ~13:42, resolved via volume resize ~14:50. Primary OOM crash ~17:00, recurring afterward with executions stuck in "queued" state despite Worker being healthy.
Appreciate any insight into the actual memory ceiling being hit and whether queued execution loss on crash is expected or something I can prevent.
Thanks,
Elliott
3 Replies
10 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 10 days ago
9 days ago
Hey, about your first issue... Do you got any logs that you can share from that service? Have you tried redeploying instead of restarting your services?
About your second issue, as far as I know that is expected from queue mode, but that would be better answered in a n8n specific support channel.
9 days ago
If the Primary logs show a "JavaScript heap out of memory" error with a V8 stack trace, that's Node's own heap hitting its limit. The NODE_OPTIONS=--max-old-space-size=8192 setting only caps that one part of memory — it doesn't cover everything else the process uses. Native modules, libuv, Postgres clients, task runners — they all live outside that heap, so the process can still eat way more than 8GB total. And honestly, it's worth double-checking that the setting is even actually live in the running process, because if something's wrapping the launch command, NODE_OPTIONS can get stripped out entirely without you ever knowing.
But if you're seeing "Killed", exit code 137, or Railway telling you the deploy ran out of memory, that's a different beast — that's the container being killed at the system level. And honestly, given the numbers you're describing (crashing at ~1GB with 24GB allocated), that's the interesting case.
Once you've narrowed it down, the next step is to check the actual limit — not what the Scale panel says, but what the system is really enforcing. Railway gives you a terminal for a reason, so hop into the Primary and run:
bash
cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.high
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.events # oom_kill counters
cat /sys/fs/cgroup/memory.stat # anon vs file (page cache)
That'll land you in one of three situations:
memory.max isn't 24GB. The real ceiling is lower than what the panel shows. And while you're at it, go back and make sure the Scale panel actually saved your 24 vCPU/24GB config — template services have a nasty habit of silently staying at their default settings.
memory.max says 24GB, but memory.events shows an oom_kill even though actual usage (anon) is low. That usually means page cache is the culprit — file writes, logs, binary data, stuff in /tmp — all of it counts against the container's memory, even though it doesn't look like "real" usage. Or it could be host-level pressure: Railway runs on shared machines, and when the host itself is running low, it kills the biggest consumer, which is you.
The Railway memory graph looks fine right up until the crash. This is the sneaky one and honestly the most common n8n OOM pattern I see. The spike was real but brief — a burst of webhook executions, binary data being written, a sudden heap jump — and the graph samples too infrequently to catch a peak that only lasted a few seconds. So the crash wasn't a slow climb over hours; it was one fast spike that blew through everything at once.
molochdagod
If the Primary logs show a "JavaScript heap out of memory" error with a V8 stack trace, that's Node's own heap hitting its limit. The NODE_OPTIONS=--max-old-space-size=8192 setting only caps that one part of memory — it doesn't cover everything else the process uses. Native modules, libuv, Postgres clients, task runners — they all live outside that heap, so the process can still eat way more than 8GB total. And honestly, it's worth double-checking that the setting is even actually live in the running process, because if something's wrapping the launch command, NODE_OPTIONS can get stripped out entirely without you ever knowing. But if you're seeing "Killed", exit code 137, or Railway telling you the deploy ran out of memory, that's a different beast — that's the container being killed at the system level. And honestly, given the numbers you're describing (crashing at ~1GB with 24GB allocated), that's the interesting case. Once you've narrowed it down, the next step is to check the actual limit — not what the Scale panel says, but what the system is really enforcing. Railway gives you a terminal for a reason, so hop into the Primary and run: bash cat /sys/fs/cgroup/memory.max cat /sys/fs/cgroup/memory.high cat /sys/fs/cgroup/memory.current cat /sys/fs/cgroup/memory.events # oom_kill counters cat /sys/fs/cgroup/memory.stat # anon vs file (page cache) That'll land you in one of three situations: memory.max isn't 24GB. The real ceiling is lower than what the panel shows. And while you're at it, go back and make sure the Scale panel actually saved your 24 vCPU/24GB config — template services have a nasty habit of silently staying at their default settings. memory.max says 24GB, but memory.events shows an oom_kill even though actual usage (anon) is low. That usually means page cache is the culprit — file writes, logs, binary data, stuff in /tmp — all of it counts against the container's memory, even though it doesn't look like "real" usage. Or it could be host-level pressure: Railway runs on shared machines, and when the host itself is running low, it kills the biggest consumer, which is you. The Railway memory graph looks fine right up until the crash. This is the sneaky one and honestly the most common n8n OOM pattern I see. The spike was real but brief — a burst of webhook executions, binary data being written, a sudden heap jump — and the graph samples too infrequently to catch a peak that only lasted a few seconds. So the crash wasn't a slow climb over hours; it was one fast spike that blew through everything at once.
9 days ago
This was extremely helpful!
Ran your commands, here's what came back:
memory.max: max (unbounded)
memory.events: oom_kill 0, oom_group_kill 0
No cgroup limit enforced, and zero OOM kills recorded at the container level. Despite Railway's dashboard reporting "Deploy Ran Out of Memory" and the service actually crashing.
Sounds like your third scenario: a host-level kill rather than something my container hit internally. In your opinion, is this a Railway-support-only question?
Context: happened twice today, both during heavy batch processing (500-1000+ workflow executions in a short window). Graph looked flat right up until each crash, consistent with a brief spike between samples.
Status changed to Open medim • 9 days ago
