`railway ssh` fails for one production service (realtime-engine) — gateway auth OK, then connection error

railway ssh

fails for one production service (realtime-engine) — gateway auth OK, then connection error

majdkaid

PROOP

12 days ago

Hi Railway team,

SSH into one production service consistently fails, while every other service in the same project, and the same service in staging, works.

IDs

Workspace: 7f9a60b5-2030-455b-8d02-5c7ab9bb2a08
Project: 31abc521-9b9f-4186-910c-d6865896ba2a (mt5-backoffice)
Service: realtime-engine, 97e49a67-cccd-40fd-8bca-9f71be0a012d
Environment: production (failing); staging (works)
Service instance (SSH user from railway ssh config): 897d58d6-8e51-4e9a-9b03-db91af01d69a
Region: europe-west4-drams3a, 1 replica, Dockerfile builder (node:22-alpine, runs as root, /bin/sh present)
Deployments: da1a03ce-121d-4161-8d95-f277fc459226 (failed since at least 24 Oct ~03:05 CET), and a fresh deploy 1aacba82-803f-4777-b8fd-8d697fdb49df (25 Oct 01:20 UTC, commit 82899030f418feb0beab5123cdf2d59b6faa54d2) — still fails

Symptom

railway ssh -p 31abc521-... -e production -s realtime-engine -- id (CLI 5.71.0, macOS, two different Macs) returns

"An error occurred connecting to your service. Please try again." Once: "The agent path is temporarily unavailable. Try again later."

Same result with plain OpenSSH using the service instance ID and the service domain as the username:

$ ssh -v 897d58d6-8e51-4e9a-9[email protected] id
debug1: Server accepts key: ... ED25519 SHA256:G2dRuIock92gZrhLkHVAPZNsipbx6U9Oz5jTO0QzO40 explicit
Authenticated to ssh.railway.com ([66.33.22.3]:22) using "publickey".
debug1: Sending command: id
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
An error occurred connecting to your service. Please try again.
Connection to ssh.railway.com closed by remote host.
Transferred: sent 2512, received 1608 bytes, in 8.7 seconds
debug1: Exit status 1

So auth at the gateway succeeds; the gateway→container agent path fails after ~8.7 s.

What works (same key, same machine, same minute)

production: web, Postgres, Redis, onboarding-service — all return uid=0(root)
staging: realtime-engine (same Dockerfile, same image base, same deploy config), web, Postgres, Redis, onboarding-service

Already tried

Fresh deploy (1aacba82) of the same commit on a new container: healthy (/health 200, /readyz 200), but SSH still fails.
Railway status page shows no SSH/compute incident.
Found and deleted a malformed variable on this service (name 'DATABASE_PUBLIC_URL, leading apostrophe; the only invalid env name in the project). Redeployed (e84bc8ed-490d-4dc3-95a2-a115704f3cdf, 25 Oct 01:40 UTC, same commit 82899030): healthy (/health 200, /readyz 200, all connectors reconnected), but SSH still fails with the same error. Variables on this service now all have valid names.
25 Oct: found the service's newest deployments were 10+ failed CLI railway up one-off jobs (Dockerfile.tmpops, last 996f9a83-df9b-4175-9c45-ad9669f200d3, 07:22 UTC), while e84bc8ed kept serving. Made a clean deployment of commit 82899030f from GitHub with the normal /apps/realtime-engine/Dockerfile: 81256919-9454-405d-9c93-9d1f77355402 (instance f7d9bce7-5057-4ed2-a00f-bf97c1ff7aa8), SUCCESS, /health and /readyz 200. SSH still fails (also with -d f7d9bce7-...), at ~14:21 UTC. In the same minute, SSH to backfill-service in production works.

Could you check the SSH agent / routing for this service instance in production? Happy to provide anything else.

Thanks,

Majd Al Kaid

Awaiting User Response

4 Replies

Status changed to Awaiting Railway Response Railway • 12 days ago

chandrika

EMPLOYEE

12 days ago

Hi Majd,

If the service has static outbound IPs enabled, that's why SSH is failing: the two systems aren't currently compatible, and we don't have a time yet for a fix. In the meantime you can use a jumpbox: deploy a small service without static outbound IPs enabled, SSH into that, then SSH from it to the service you need.

If static outbound IPs aren't enabled on that service, reply with the service link and the time you last tried, and we'll look into it.

Status changed to Awaiting User Response Railway • 12 days ago

amnongraitzer

PRO

11 days ago

Same issue here, and yes: static outbound IPs are enabled on this service (since Sep 24). SSH worked with them until ~Oct 25, so this looks like a recent regression. Last attempt: 2026-10-26 15:50 IST.

Status changed to Awaiting Railway Response Railway • 11 days ago

dizzydes90

EMPLOYEE

11 days ago

We checked realtime-engine in production and it has static outbound IPs enabled, so that's why SSH fails there while staging and your other production services connect. The two don't work together right now: the container can't open its tunnel back to our SSH relay through the static-IP path, which is why the session drops after about 9 seconds even though your key is accepted, the ~8.7 s you measured. It isn't your image, variables or configuration, and redeploying won't change it, which matches what you saw across da1a03ce, 1aacba82, e84bc8ed and 81256919; 81256919 shows the same failure in our relay logs. Leave the static outbound IPs as they are. We aim to have this fixed in the coming weeks and we'll reply here when it ships. For anyone else reading with the same symptom: please open your own thread with a service link so we can check your service specifically.

Status changed to Awaiting User Response Railway • 11 days ago

Railway

BOT

4 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway • 4 days ago

brody

EMPLOYEE

2 days ago

SSH should now work with Static Outbound IPs enabled. Please try connecting again and let us know if you still run into any issues.

Thank you for your patience.

Status changed to Awaiting User Response Railway • 2 days ago

Welcome!