2 months ago
Problem and desired outcome
Our production api and worker services (Node.js) log that the Node debugger has opened on 127.0.0.1:9229 and that a debugger client attached. We never enable the inspector, so we expected no such lines. Nothing is broken and the service is healthy; we want to understand what opens and attaches the debugger. It is still happening on the current deployments. We would like Railway to tell us whether a Railway platform feature (tracing, profiling, diagnostics or similar) does this, which feature it is, and what would stop working if we blocked it.
Resources and timing
- Project: ajoquest (38d7f7c6-7278-4409-afae-f7c6bcc7f483)
- Environment: production (7db59fba-6b53-4a6f-bdad-54682b72ff07)
- Services: api and worker (web is a Next.js service and was not checked)
- Builder: Railpack. api start command:
pnpm start:api, which runsnode -r ./apps/api/dist/tracing.js apps/api/dist/index.js
Occurrences we observed (all times UTC, from railway logs <deploymentId> --json):
- api deployment 932fa820-586b-489c-ba4f-49157514e629: started 2026-09-25T08:00:49Z; debugger lines at 2026-09-26T10:43:48Z (about a day after start)
- api deployment 1f74dd49-fef9-4baa-b01f-4191e6ec188a (2026-09-28): one occurrence (timestamp not pulled)
- api deployment 8ab65e75-d8d0-411c-addb-e6e314573605 (currently running): started 2026-10-01T05:28:14Z; debugger lines at 2026-10-01T05:28:44Z (30 seconds after start)
- worker deployment 1dc4a860-6f9c-481e-891f-32863f04facd (currently running): same three lines seen
Deployments with no occurrence in the last 200 log lines: api bd04fb29-e26c-471b-9387-c16f7ec74144 (2026-09-30) and bc727b9d-02a1-4daf-8784-7c5c30c0fa14 (2026-09-23).
Note: a newer api/worker deployment from commit 933a7b9 (pushed 2026-10-01 around 08:33 UTC) may now be the running one; its logs have not been checked.
Evidence and attempts
Log excerpt, api deployment 8ab65e75-d8d0-411c-addb-e6e314573605:
2026-10-01T05:28:14.866Z AjoQuest API listening on :8080
2026-10-01T05:28:44.854Z Debugger listening on ws://127.0.0.1:9229/4ac56323-b6ca-4cba-9052-391e5ca3f3e9
2026-10-01T05:28:44.854Z For help, see: https://nodejs.org/learn/getting-started/debugging
2026-10-01T05:28:44.854Z Debugger attached.Worker deployment 1dc4a860-6f9c-481e-891f-32863f04facd shows the same pattern (Debugger listening on ws://127.0.0.1:9229/dbd265f2-9bb1-4c1c-9474-f83739b729e2, then Debugger attached.).
Checks performed:
- Our repository contains no
--inspectflag. The api service variables contain noNODE_OPTIONSor inspect-related variable. - Using
railway sshinto the api container:NODE_OPTIONSis empty, and the process list shows onlypnpm start:api,sh -c node -r ./apps/api/dist/tracing.js apps/api/dist/index.jsandnode ...(plus our ssh shell). No--inspecton the command line. The container has OTEL_EXPORTER_OTLP_* and OTEL_SERVICE_* variables, which we did not set ourselves. - We searched all installed npm packages for code that calls
inspector.open. The only matches used by the api are in@sentry/node11.1.0 (LocalVariables and ANR integrations). Neither is active: LocalVariables only starts whenincludeLocalVariables: true(we don't set it), and we don't use ANR. The lines also appear on 2026-09-25/26, before we added Sentry on 2026-09-30. - Starting the same built api locally with a hook on
inspector.openand production-like environment variables: the debugger never opened. - We deployed a change that removed Sentry's LocalVariables integration (commit b39ed1b). The lines still appeared on the new deployments (8ab65e75 and 1dc4a860), so we reverted it (commit 933a7b9).
Pinned Solution
2 months ago
Hey — those lines are Node’s inspector, not a Railway-documented tracing/profiling product. Here’s what published docs support vs leave open.
How Node opens / attaches
Node listens for a debugger when started with --inspect (default 127.0.0.1:9229), when it gets SIGUSR1 (unless --disable-sigusr1), or when inspect flags arrive via NODE_OPTIONS (Debugging Node.js, CLI --inspect / NODE_OPTIONS). Your logs match that default: Debugger listening on ws://127.0.0.1:9229/… then Debugger attached. — attached means a client connected to that local inspector WebSocket.
No --inspect on the argv from SSH does not rule out SIGUSR1, a transient NODE_OPTIONS, or code/-r preload calling the inspector API.
Intentional debug on Railway
To reach a non-HTTP debug port from outside the container, the published path is TCP Proxy (Settings → Networking → TCP Proxy to the internal port) (TCP Proxy, CLI tcp-proxy). That is opt-in networking — not evidence of auto-attach. Service variables can also carry Node flags such as NODE_OPTIONS (example pattern).
Undocumented (do not invent)
Railway docs do not define a platform feature that automatically opens or attaches the Node inspector. We cannot name “which Railway feature” did this, or what would break if you blocked it, from published material alone — that needs staff confirmation on your project.
Asker-side clues only
Your start uses node -r ./apps/api/dist/tracing.js …, and you saw OTEL_* vars you didn’t set. Treat those as app/deploy clues (preload/inspector.open, OTEL/Sentry injectors, anything sending SIGUSR1) — not proof of Railway auto-attach.
Next: inspect tracing.js / OTEL for inspector open, and ask Railway staff whether anything on ajoquest production api/worker injects inspect/SIGUSR1. Until documented, treat auto-attach as undocumented.
4 Replies
2 months ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 2 months ago
2 months ago
Hey — those lines are Node’s inspector, not a Railway-documented tracing/profiling product. Here’s what published docs support vs leave open.
How Node opens / attaches
Node listens for a debugger when started with --inspect (default 127.0.0.1:9229), when it gets SIGUSR1 (unless --disable-sigusr1), or when inspect flags arrive via NODE_OPTIONS (Debugging Node.js, CLI --inspect / NODE_OPTIONS). Your logs match that default: Debugger listening on ws://127.0.0.1:9229/… then Debugger attached. — attached means a client connected to that local inspector WebSocket.
No --inspect on the argv from SSH does not rule out SIGUSR1, a transient NODE_OPTIONS, or code/-r preload calling the inspector API.
Intentional debug on Railway
To reach a non-HTTP debug port from outside the container, the published path is TCP Proxy (Settings → Networking → TCP Proxy to the internal port) (TCP Proxy, CLI tcp-proxy). That is opt-in networking — not evidence of auto-attach. Service variables can also carry Node flags such as NODE_OPTIONS (example pattern).
Undocumented (do not invent)
Railway docs do not define a platform feature that automatically opens or attaches the Node inspector. We cannot name “which Railway feature” did this, or what would break if you blocked it, from published material alone — that needs staff confirmation on your project.
Asker-side clues only
Your start uses node -r ./apps/api/dist/tracing.js …, and you saw OTEL_* vars you didn’t set. Treat those as app/deploy clues (preload/inspector.open, OTEL/Sentry injectors, anything sending SIGUSR1) — not proof of Railway auto-attach.
Next: inspect tracing.js / OTEL for inspector open, and ask Railway staff whether anything on ajoquest production api/worker injects inspect/SIGUSR1. Until documented, treat auto-attach as undocumented.
a month ago
We're seeing the same thing on a production Node service, so it isn't specific to this project.
Right after a replica started during a normal image deploy, stderr showed:
Debugger listening on ws://127.0.0.1:9229/d256ecfe-f2b5-407d-8c88-a766859dc78f
For help, see: https://nodejs.org/learn/getting-started/debugging
Debugger attached.- Deployment
54c263fa-169e-4534-bd5e-1e6db65f3189, instancedc0b1150-b8a8-4245-980f-7468b844a7a9 - Time: 2026-10-07T12:42:20Z, about 3 seconds after the server finished starting
What we ruled out:
- The image runs
node dist/app.js. No--inspectin the start command or in the image, no start command override, no pre-deploy command. NODE_OPTIONScontains only--max-old-space-sizeand--trace-warnings.- We pulled the exact image digest and searched it. The app code has no inspector calls and no
SIGUSR1handler. The onlyinspector.open()innode_modulesis in@sentry/node, in two features we don't enable, and those would pick a random port, not 9229. - The audit log has no
ContainerAccessevents anywhere near that time, so nobody on our side had a shell in the container. - It happened once in the last 30 days.
Like here, the inspector opened on 127.0.0.1:9229 and a client attached within about 200 ms. That's the default port Node uses when a process receives SIGUSR1, and the connection had to come from inside the container.
@Railway team: does any part of the platform (deploy rollout, health checks, profiling, tracing) send SIGUSR1 to container processes or connect to the Node inspector? If not, can you see what triggered it in the deployment above? With two unrelated projects showing the same pattern in the same week, it looks platform-side.
Status changed to Solved niyexdroid • about 1 month ago
Status changed to Awaiting Conductor Response Railway • about 1 month ago
Status changed to Open dev • about 1 month ago
dev
Do you have tracing enabled? This may show up when you have tracing enabled
a month ago
I do, but i have been experiencing this before tracing was enabled on my project. Also nodejs also confirmed it. Sometimes IDE such as VSCODE does the auto attach (NODE OPTION).
Not related but i don't have tracing interface in my dashboard yet.
Status changed to Solved niyexdroid • about 1 month ago