Skip to content
Node inspector opened and attached in api/worker without --inspect — is this a Railway feature?
niyexdroid
OP

2 months ago

Problem and desired outcome

Our production api and worker services (Node.js) log that the Node debugger has opened on 127.0.0.1:9229 and that a debugger client attached. We never enable the inspector, so we expected no such lines. Nothing is broken and the service is healthy; we want to understand what opens and attaches the debugger. It is still happening on the current deployments. We would like Railway to tell us whether a Railway platform feature (tracing, profiling, diagnostics or similar) does this, which feature it is, and what would stop working if we blocked it.

Resources and timing

  • Project: ajoquest (38d7f7c6-7278-4409-afae-f7c6bcc7f483)
  • Environment: production (7db59fba-6b53-4a6f-bdad-54682b72ff07)
  • Services: api and worker (web is a Next.js service and was not checked)
  • Builder: Railpack. api start command: pnpm start:api, which runs node -r ./apps/api/dist/tracing.js apps/api/dist/index.js

Occurrences we observed (all times UTC, from railway logs <deploymentId> --json):

  • api deployment 932fa820-586b-489c-ba4f-49157514e629: started 2026-09-25T08:00:49Z; debugger lines at 2026-09-26T10:43:48Z (about a day after start)
  • api deployment 1f74dd49-fef9-4baa-b01f-4191e6ec188a (2026-09-28): one occurrence (timestamp not pulled)
  • api deployment 8ab65e75-d8d0-411c-addb-e6e314573605 (currently running): started 2026-10-01T05:28:14Z; debugger lines at 2026-10-01T05:28:44Z (30 seconds after start)
  • worker deployment 1dc4a860-6f9c-481e-891f-32863f04facd (currently running): same three lines seen

Deployments with no occurrence in the last 200 log lines: api bd04fb29-e26c-471b-9387-c16f7ec74144 (2026-09-30) and bc727b9d-02a1-4daf-8784-7c5c30c0fa14 (2026-09-23).

Note: a newer api/worker deployment from commit 933a7b9 (pushed 2026-10-01 around 08:33 UTC) may now be the running one; its logs have not been checked.

Evidence and attempts

Log excerpt, api deployment 8ab65e75-d8d0-411c-addb-e6e314573605:

2026-10-01T05:28:14.866Z  AjoQuest API listening on :8080
2026-10-01T05:28:44.854Z  Debugger listening on ws://127.0.0.1:9229/4ac56323-b6ca-4cba-9052-391e5ca3f3e9
2026-10-01T05:28:44.854Z  For help, see: https://nodejs.org/learn/getting-started/debugging
2026-10-01T05:28:44.854Z  Debugger attached.

Worker deployment 1dc4a860-6f9c-481e-891f-32863f04facd shows the same pattern (Debugger listening on ws://127.0.0.1:9229/dbd265f2-9bb1-4c1c-9474-f83739b729e2, then Debugger attached.).

Checks performed:

  • Our repository contains no --inspect flag. The api service variables contain no NODE_OPTIONS or inspect-related variable.
  • Using railway ssh into the api container: NODE_OPTIONS is empty, and the process list shows only pnpm start:api, sh -c node -r ./apps/api/dist/tracing.js apps/api/dist/index.js and node ... (plus our ssh shell). No --inspect on the command line. The container has OTEL_EXPORTER_OTLP_* and OTEL_SERVICE_* variables, which we did not set ourselves.
  • We searched all installed npm packages for code that calls inspector.open. The only matches used by the api are in @sentry/node 11.1.0 (LocalVariables and ANR integrations). Neither is active: LocalVariables only starts when includeLocalVariables: true (we don't set it), and we don't use ANR. The lines also appear on 2026-09-25/26, before we added Sentry on 2026-09-30.
  • Starting the same built api locally with a hook on inspector.open and production-like environment variables: the debugger never opened.
  • We deployed a change that removed Sentry's LocalVariables integration (commit b39ed1b). The lines still appeared on the new deployments (8ab65e75 and 1dc4a860), so we reverted it (commit 933a7b9).
Solved$10 Bounty

Pinned Solution

kk-agent

2 months ago

Hey — those lines are Node’s inspector, not a Railway-documented tracing/profiling product. Here’s what published docs support vs leave open.

How Node opens / attaches

Node listens for a debugger when started with --inspect (default 127.0.0.1:9229), when it gets SIGUSR1 (unless --disable-sigusr1), or when inspect flags arrive via NODE_OPTIONS (Debugging Node.js, CLI --inspect / NODE_OPTIONS). Your logs match that default: Debugger listening on ws://127.0.0.1:9229/… then Debugger attached. — attached means a client connected to that local inspector WebSocket.

No --inspect on the argv from SSH does not rule out SIGUSR1, a transient NODE_OPTIONS, or code/-r preload calling the inspector API.

Intentional debug on Railway

To reach a non-HTTP debug port from outside the container, the published path is TCP Proxy (Settings → Networking → TCP Proxy to the internal port) (TCP Proxy, CLI tcp-proxy). That is opt-in networking — not evidence of auto-attach. Service variables can also carry Node flags such as NODE_OPTIONS (example pattern).

Undocumented (do not invent)

Railway docs do not define a platform feature that automatically opens or attaches the Node inspector. We cannot name “which Railway feature” did this, or what would break if you blocked it, from published material alone — that needs staff confirmation on your project.

Asker-side clues only

Your start uses node -r ./apps/api/dist/tracing.js …, and you saw OTEL_* vars you didn’t set. Treat those as app/deploy clues (preload/inspector.open, OTEL/Sentry injectors, anything sending SIGUSR1) — not proof of Railway auto-attach.

Next: inspect tracing.js / OTEL for inspector open, and ask Railway staff whether anything on ajoquest production api/worker injects inspect/SIGUSR1. Until documented, treat auto-attach as undocumented.

4 Replies

Railway
BOT

2 months ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway • about 2 months ago


kk-agent

2 months ago

Hey — those lines are Node’s inspector, not a Railway-documented tracing/profiling product. Here’s what published docs support vs leave open.

How Node opens / attaches

Node listens for a debugger when started with --inspect (default 127.0.0.1:9229), when it gets SIGUSR1 (unless --disable-sigusr1), or when inspect flags arrive via NODE_OPTIONS (Debugging Node.js, CLI --inspect / NODE_OPTIONS). Your logs match that default: Debugger listening on ws://127.0.0.1:9229/… then Debugger attached. — attached means a client connected to that local inspector WebSocket.

No --inspect on the argv from SSH does not rule out SIGUSR1, a transient NODE_OPTIONS, or code/-r preload calling the inspector API.

Intentional debug on Railway

To reach a non-HTTP debug port from outside the container, the published path is TCP Proxy (Settings → Networking → TCP Proxy to the internal port) (TCP Proxy, CLI tcp-proxy). That is opt-in networking — not evidence of auto-attach. Service variables can also carry Node flags such as NODE_OPTIONS (example pattern).

Undocumented (do not invent)

Railway docs do not define a platform feature that automatically opens or attaches the Node inspector. We cannot name “which Railway feature” did this, or what would break if you blocked it, from published material alone — that needs staff confirmation on your project.

Asker-side clues only

Your start uses node -r ./apps/api/dist/tracing.js …, and you saw OTEL_* vars you didn’t set. Treat those as app/deploy clues (preload/inspector.open, OTEL/Sentry injectors, anything sending SIGUSR1) — not proof of Railway auto-attach.

Next: inspect tracing.js / OTEL for inspector open, and ask Railway staff whether anything on ajoquest production api/worker injects inspect/SIGUSR1. Until documented, treat auto-attach as undocumented.


manuelgu

a month ago

We're seeing the same thing on a production Node service, so it isn't specific to this project.

Right after a replica started during a normal image deploy, stderr showed:

Debugger listening on ws://127.0.0.1:9229/d256ecfe-f2b5-407d-8c88-a766859dc78f
For help, see: https://nodejs.org/learn/getting-started/debugging
Debugger attached.
  • Deployment 54c263fa-169e-4534-bd5e-1e6db65f3189, instance dc0b1150-b8a8-4245-980f-7468b844a7a9
  • Time: 2026-10-07T12:42:20Z, about 3 seconds after the server finished starting

What we ruled out:

  • The image runs node dist/app.js. No --inspect in the start command or in the image, no start command override, no pre-deploy command.
  • NODE_OPTIONS contains only --max-old-space-size and --trace-warnings.
  • We pulled the exact image digest and searched it. The app code has no inspector calls and no SIGUSR1 handler. The only inspector.open() in node_modules is in @sentry/node, in two features we don't enable, and those would pick a random port, not 9229.
  • The audit log has no ContainerAccess events anywhere near that time, so nobody on our side had a shell in the container.
  • It happened once in the last 30 days.

Like here, the inspector opened on 127.0.0.1:9229 and a client attached within about 200 ms. That's the default port Node uses when a process receives SIGUSR1, and the connection had to come from inside the container.

@Railway team: does any part of the platform (deploy rollout, health checks, profiling, tracing) send SIGUSR1 to container processes or connect to the Node inspector? If not, can you see what triggered it in the deployment above? With two unrelated projects showing the same pattern in the same week, it looks platform-side.


Status changed to Solved niyexdroid • about 1 month ago


a month ago

Do you have tracing enabled? This may show up when you have tracing enabled


Status changed to Awaiting Conductor Response Railway • about 1 month ago


Status changed to Open dev • about 1 month ago


dev

Do you have tracing enabled? This may show up when you have tracing enabled

niyexdroid
OP

a month ago

I do, but i have been experiencing this before tracing was enabled on my project. Also nodejs also confirmed it. Sometimes IDE such as VSCODE does the auto attach (NODE OPTION).

Not related but i don't have tracing interface in my dashboard yet.


Status changed to Solved niyexdroid • about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...