a month ago
Hi Railway team,
We’re seeing an intermittent issue affecting one of our production worker services.
The worker generates PDFs using Google Chrome (Pyppeteer), but after a period of normal operation the runtime starts failing to create any new process.
The failure is not limited to our application.
Even Railway SSH fails with:
crun: fork: Resource temporarily unavailable
fail startup: OCI runtime error
At the same time, our application receives:
BlockingIOError: [Errno 11] Resource temporarily unavailable
originating from:
subprocess.Popen()
while trying to launch Chrome.
Environment
Runtime:
Runtime V2
Worker service:
worker-heavy-pdf
Deployment:
70f09e9c-728c-4156-9801-e9ae26a619b1
Resource usage
At the time of the incident:
CPU:~0.1%
Memory:~4.1 GB / 6 GB
No CPU saturation.
No memory saturation.
No queue backlog.
Chrome
Chrome is correctly installed during build:
google-chrome-stable
149.0.7827.196
The executable exists.
The application is configured with: BROWSER_PDF_EXECUTABLE=/usr/bin/google-chrome-stable
After restarting only this worker Immediately after restart:
Container status:
pids.max = 1000
pids.current = 4
pids.peak = 7
pids.events.max = 0
Chrome processes:0
Threads:6
Processes:5
The issue completely disappears until it happens again.
⸻
Question
Could this be related to Runtime V2, crun, or container process management?
Specifically:
- OCI runtime process creation
- fork()/clone() failures
- hidden PID limits
- runtime-level resource exhaustion
- known Runtime V2 issues
The fact that Railway SSH itself fails with: crun: fork: Resource temporarily unavailable
suggests the problem may exist below the application layer.
If there are additional runtime metrics (process count, PID exhaustion, kernel logs, runtime diagnostics) available for this deployment, we’d appreciate any guidance.
Thank you.
1 Replies
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
Monitor pids.current while the worker is running, not just after a restart. If it keeps increasing until fork() starts failing, that's a strong sign that processes or threads aren't being released.
Also check for orphaned Chrome processes after each PDF generation:
ps -ef | grep chrome
or
ps -eLf | grep chrome
Make sure every Chrome instance exits cleanly after the job completes.
Since subprocess.Popen() and Railway SSH both fail with Resource temporarily unavailable, collect pids.current, pids.peak, and the output of ps at the moment the issue occurs. Comparing those values with the healthy state should make it much easier to determine whether this is PID exhaustion or another runtime-level issue.