Intermittent SSH/SFTP timeouts blocking a production deployment
beallyy
OP

16 hours ago

Hello Railway team and community,

We are experiencing persistent, intermittent SSH/SFTP connectivity failures while preparing a controlled update to our production ERP.

Our application remains healthy and accessible over HTTPS, but the unreliable administrative connection is preventing us from safely proceeding with database maintenance and deployment.

Environment

  • Railway plan: Hobby
  • Environment: Production
  • Database: PostgreSQL 18.6
  • Railway CLI: 5.45.3
  • Local operating system: Windows 10
  • SSH endpoint: ssh.railway.com (port 22)

Observed behavior

  1. SSH connections sometimes authenticate successfully and execute commands normally.
  2. Subsequent connections may time out before authentication, even when no files are being transferred.
  3. SFTP transfers are interrupted by timeouts, unresponsive sessions, or broken connections.
  4. Both our backend and PostgreSQL services have experienced these intermittent connection issues.
  5. Our application services remain healthy throughout these failures.

Troubleshooting already performed

We have tested:

  • DNS resolution and TCP/22 connectivity.
  • IPv4 and IPv6 connections.
  • Explicit IPv4 connections using ssh -4.
  • SSH authentication and short administrative commands.
  • A different local connection using an iPhone mobile hotspot.
  • SFTP file transfers with resumable fragments and SHA-256 integrity checks.

The same type of failure occurred even when forcing IPv4 and using the mobile hotspot.

During a transfer of an approximately 173 MB administrative package, we switched to 4 MiB fragments to minimize retransmissions. Six fragments were successfully transferred and verified, but subsequent SSH/SFTP failures prevented completion.

The failures are not limited to large transfers: some connections time out before authentication.

We also verified that railway service files uses SFTP, so it does not appear to offer an independent transport path.

Production impact

Our production backend, frontend, and PostgreSQL services remain operational.

We have not initiated database maintenance, migrations, or application deployment.

We are deliberately avoiding disruptive operations because our maintenance procedure requires reliable administrative access for both execution and recovery.

Questions for Railway

  1. Are there any known intermittent issues affecting the SSH relay or SFTP gateway?
  2. Could routing, session affinity, connection limits, or gateway timeouts explain this behavior?
  3. Would explicitly targeting a deployment instance improve SSH reliability?
  4. Is there a supported way to transfer administrative files into an existing service container without relying on the same SSH/SFTP connection?
  5. Is there a recommended procedure for maintaining reliable administrative access during production PostgreSQL maintenance?
  6. Can Railway staff investigate relay-side failures using timestamps and sanitized logs?

We have found other community discussions reporting similar symptoms while production deployments remained healthy.

Important: We do not want to restart, redeploy, recycle, or modify any running service solely for troubleshooting without understanding the consequences and explicitly approving the action.

We can provide detailed timestamps, sanitized logs, and project/deployment identifiers privately to Railway staff if necessary.

Thank you for your assistance.

Awaiting User Response

7 Replies

Railway
BOT

16 hours ago

We have an open incident, External network blocks affecting access to Railway. Some network providers and security services are blocking Railway IP addresses, which affects public networking and railway.com from certain networks. Its status is Identified and we are still working on it. That could explain connections that succeed one moment and time out before authentication the next. Your hotspot test also failed, though, so we can't confirm it explains everything in your case.

On targeting, a plain railway ssh with the project, service and environment flags picks the running instance automatically. If you pass -d / --deployment-instance, the value has to be a running deployment instance ID, not a deployment ID. Pinning an instance is not a way to make the connection more reliable.

railway volume files and railway volume browse use SFTP over the same SSH connection, as you found, so they don't give you a separate transport path. Every SSH-backed tool, including dashboard Console and Files, also needs a healthy running instance of the target service. Nothing in this troubleshooting requires a restart or redeploy.

To look into failures that don't fit the incident pattern, please share the exact commands you ran, the full error text, and the UTC timestamps of both failed and successful attempts. Leave out project and deployment IDs, since this thread is public.


Status changed to Awaiting User Response Railway • about 16 hours ago


beallyy
OP

16 hours ago

Thank you for the response. The issue is not resolved.

We have confirmed intermittent TCP/22 failures before SSH authentication, including when forcing IPv4 and using a mobile hotspot.

Relevant observations (UTC, November 11, 2026):

  • 17:11 — TCP/SSH connection timeout before authentication.
  • 18:20–18:34 — SFTP transfer interruptions, including "server not responding" and "Broken pipe".
  • 19:31 — Short authenticated SSH commands succeeded for both backend and PostgreSQL.
  • A later IPv4-only SSH attempt failed with a TCP connection timeout after approximately 8 seconds.

During a resumable SFTP transfer, six 4 MiB fragments were successfully uploaded and verified by SHA-256. A subsequent transfer failed, and we stopped to avoid incomplete administrative operations.

We used Railway CLI SSH with explicit project, environment and service selection, as well as standard OpenSSH/SFTP tools. We can provide exact command lines with sensitive values removed.

Could a Railway staff member clarify whether there is currently an active SSH relay or external-network access incident, and whether these failures can be investigated using the timestamps above?

We specifically need reliable administrative access before starting a controlled PostgreSQL maintenance operation.

Our production services remain healthy. We have not started maintenance, migrations or deployment, and we do not authorize a restart or redeployment as a troubleshooting measure.

Thank you.


Status changed to Awaiting Railway Response Railway • about 16 hours ago


Railway
BOT

16 hours ago

Yes, the external network access incident we mentioned earlier, External network blocks affecting access to Railway, is still active with status Identified. Some network providers and security services are blocking Railway IP addresses, and access remains affected from some networks while that work continues. We can't give a specific ETA for when all of those blocks will be lifted.

Your 8 second TCP timeouts before authentication match that pattern, but we can't yet confirm that the incident caused every failure in your timeline, including the hotspot failure and the SFTP broken pipes.

The UTC timestamps help, but on their own they don't let us connect each failure to a specific cause. Please share the exact command lines with sensitive values removed, along with the full error output for the 17:11 timeout, the 18:20 to 18:34 SFTP interruptions, and the later IPv4-only failure. Leave project, service and deployment IDs out, since this thread is public. Nothing here requires a restart or redeploy of your services.


Status changed to Awaiting User Response Railway • about 16 hours ago


Railway

Yes, the external network access incident we mentioned earlier, External network blocks affecting access to Railway, is still active with status Identified. Some network providers and security services are blocking Railway IP addresses, and access remains affected from some networks while that work continues. We can't give a specific ETA for when all of those blocks will be lifted. Your 8 second TCP timeouts before authentication match that pattern, but we can't yet confirm that the incident caused every failure in your timeline, including the hotspot failure and the SFTP broken pipes. The UTC timestamps help, but on their own they don't let us connect each failure to a specific cause. Please share the exact command lines with sensitive values removed, along with the full error output for the 17:11 timeout, the 18:20 to 18:34 SFTP interruptions, and the later IPv4-only failure. Leave project, service and deployment IDs out, since this thread is public. Nothing here requires a restart or redeploy of your services.

beallyy
OP

12 hours ago

Thank you for your response. The issue remains unresolved.

We use Railway CLI 5.45.3 and Windows OpenSSH/SFTP. Below are the exact recorded errors and timestamps from November 11, 2026 (UTC). No new tests were performed. All private identifiers are redacted.

1. TCP timeout before authentication

At 17:11:37 UTC, a Python IPv4 socket connection to 66.33.22.3:22 timed out after 5.01 seconds.

Recorded error: TimeoutError (errno: null).

However, TCP probes at 17:11:44 and 17:11:47 succeeded, receiving SSH-2.0-Go.

Authenticated SSH commands to PostgreSQL and backend also succeeded between 17:11:49 and 17:11:55 (1.92–3.02 seconds, exit code 0).

SSH command format:

ssh.exe -T -i "<KEY>" -oBatchMode=yes -oIdentitiesOnly=yes -oStrictHostKeyChecking=yes -oConnectionAttempts=1 -oConnectTimeout=8 <SERVICE_INSTANCE>@ssh.railway.com "<READ_ONLY_CHECK>"

2. SFTP failures (18:20–18:34 UTC)

We transferred a 173 MB administrative archive in 42 fragments, mostly 4 MiB each, to a private temporary directory outside PostgreSQL's data volume.

The first successful fragment took 105.23 seconds.

Fragment 5 failed after 14.47 seconds (exit code 255).

Complete stderr:

Timeout, server ssh.railway.com not responding.
Couldn't send packet: Broken pipe

Local log timestamp: 18:27:44 UTC.

SFTP command format:

sftp.exe -i "<KEY>" -oBatchMode=yes -oIdentitiesOnly=yes -oConnectTimeout=8 -oConnectionAttempts=1 -b "<BATCH_FILE>" <POSTGRES_INSTANCE>@ssh.railway.com

A single retry succeeded. Six fragments were verified remotely using size and SHA-256.

After uploading fragment 7, a separate SSH connection for hash verification failed after 8.08 seconds (exit code 255).

Complete stderr:

ssh: connect to host ssh.railway.com port 22: Connection timed out

Local log timestamp: 18:33:36 UTC.

The unverified fragment was removed. Six verified fragments remain. The archive was never assembled or executed.

3. IPv4-only failure

We changed the Windows route to an iPhone hotspot. Carrier independence was not conclusively verified.

Four authenticated ssh.exe -4 checks succeeded between 20:21:49 and 20:22:06 UTC (4.60–5.51 seconds).

A subsequent IPv4-only backend SSH check failed before any file transfer.

Duration: 8.04 seconds. Exit code: 255.

Local log timestamp: 20:26:34 UTC.

Command:

ssh.exe -4 -T -i "<KEY>" -oBatchMode=yes -oIdentitiesOnly=yes -oConnectTimeout=8 -oConnectionAttempts=1 <BACKEND_INSTANCE>@ssh.railway.com "<READ_ONLY_CHECK>"

Complete stderr:

ssh: connect to host ssh.railway.com port 22: Connection timed out

4. Production impact

Our frontend/backend remain healthy (HTTP 200), and PostgreSQL remains operational.

No maintenance, migrations, deployments, restarts, or database permission changes were performed. Autodeploys are temporarily disabled.

The SSH instability prevents a safe production update because reliable administrative access is required for recovery.

5. Request for Railway engineering

Could a human Railway engineer please investigate:

  1. Relay/gateway logs for these UTC timestamps, including routing, filtering, rate limits, and disconnects?
  2. Why TCP/22 intermittently times out before authentication despite successful SSH sessions?
  3. Whether this relates to the reported external-network access incident?
  4. Whether there is a supported non-SSH/SFTP method to transfer files into an existing service container without additional infrastructure?

We can share sanitized logs and identifiers privately.

Please do not restart, redeploy, recycle, or modify our production services without explicit approval.

Thank you.


Status changed to Awaiting Railway Response Railway • about 12 hours ago


Railway
BOT

12 hours ago

Thank you for the detailed timeline. The external network access incident is still active with status Identified. Some networks are still blocking Railway IP addresses while that work continues, and we can't give a specific ETA. The address you probed at 17:11:37 UTC sits in one of our public edge ranges, and short TCP timeouts on port 22 that alternate with successful connections a few seconds later fit what blocking from some networks would cause. We can't confirm that the incident caused every failure, including the hotspot attempt at 20:26:34 UTC and the SFTP broken pipe at 18:27:44 UTC.

The file paths we can point to, including railway volume files, railway volume browse, and dashboard Console and Files, all run over that same SSH connection, so none of them gives you a separate route for the transfer.

Holding off on the PostgreSQL maintenance while the connection is unreliable is consistent with your recovery plan. Nothing here requires a restart or redeploy, and none has been done to your services.


Status changed to Awaiting User Response Railway • about 12 hours ago


beallyy
OP

2 hours ago

Thank you for the clarification.

We will keep our production services unchanged and postpone PostgreSQL maintenance until administrative connectivity is reliable.

Could you please provide the official status page link or incident identifier for the currently active external network access incident?

We would like to monitor its resolution before resuming our controlled deployment.

Thank you.


Status changed to Awaiting Railway Response Railway • about 2 hours ago


Railway
BOT

2 hours ago

The incident is External network blocks affecting access to Railway, and you can follow it at status.railway.com/incident/QBCLHG1V. Its current status is Identified. Access is still affected from some networks, and we can't give a specific ETA for when all of the blocks will be lifted.


Status changed to Awaiting User Response Railway • about 2 hours ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...