PostgreSQL SSH connection fails after Railway incident
pablogauzzi
HOBBYOP

10 days ago

Hello Railway Support,

I'm experiencing issues connecting to my PostgreSQL database hosted on Railway following the October 21 incident.

Project: creative-determination

Environment: production

Service: Postgres

Plan: Hobby

The PostgreSQL service and my FastAPI application are currently shown as Sleeping. Previously, when running, the application successfully connected to the database and its API endpoints returned HTTP 200.

However, I cannot establish an SSH connection to access PostgreSQL for a database backup.

Commands attempted:

railway connect Postgres --ssh --tunnel-only

Error:

channel 1: open failed: connect failed: port forwarding rides your dev.new session: run ssh -L ... dev.new with a shell or command, not -N

I also tried:

railway ssh --service Postgres -- pg_dump --version

Error:

Sorry Railway TUI not currently available

The Railway CLI is installed, authenticated and linked to the correct production project.

Could you please help clarify:

  1. Why SSH tunneling and remote commands are failing?
  2. Whether these errors are related to the recent Railway incident or current service sleeping state?
  3. What is the recommended secure procedure to create a PostgreSQL backup without enabling public database access?

I want to preserve all existing production data and avoid any destructive operations.

I also reviewed the Railway community thread "Least-privilege method for a verified Production PostgreSQL backup". However, the suggested SSH procedures are failing in my environment with the errors described above.

I would appreciate guidance on how to resolve these specific SSH errors and safely create a backup of my production database.

Awaiting User Response

11 Replies

Status changed to Awaiting Railway Response Railway • 10 days ago


10 days ago

On the SSH errors: since the October 21 incident, SSH is unavailable on any service with Static Outbound IPs enabled. If your Postgres service has that turned on, that limitation would explain why both railway connect --ssh and railway ssh fail. Your Postgres deployment also currently shows as Sleeping, and SSH sessions connect to a running container.

You can back up without SSH or public access. Your Postgres service has a volume attached, so you can take a volume backup from the service's Backups tab, or run railway postgres pitr backup create --service Postgres --name pre-backup. That snapshot stays on our platform and doesn't change your existing data. Point-in-time recovery is available from the same tab. Restoring that way creates a separate new service and leaves the original untouched. See Point-in-Time Recovery and the railway postgres reference.


Status changed to Awaiting User Response brody • 10 days ago


pablogauzzi
HOBBYOP

8 days ago

Hi Brody,

Thank you for your response.

I checked the Backups tab for my PostgreSQL service, but it shows "No Backups" and states that upgrading to Pro is required to create new backups.

I'm currently on the Hobby plan.

Could you please confirm whether the following command is supported on Hobby and can create a backup without modifying or restarting the production database?

railway postgres pitr backup create --service Postgres --name pre-backup

If this command also requires Pro, what is the safest supported backup alternative for Hobby users while SSH access is unavailable?

Thank you.


Status changed to Awaiting Railway Response Railway • 8 days ago


8 days ago

railway postgres pitr backup create is the CLI version of the Backups tab's on-demand volume backup, so it has the same plan requirement as the Backups tab. On Hobby it won't create a backup. Our earlier reply pointed you to that command without mentioning this, sorry for the confusion.

Volume backups and point-in-time recovery are the supported ways to back up a database on Railway without SSH or public access, and both need Pro. Upgrading to Pro unlocks on-demand backups from the Backups tab or the CLI. Taking a volume backup doesn't change your existing data, and restores from it go to a new volume instead of overwriting your current one. Enabling PITR is different, because it redeploys the Postgres service, and its restore window only starts from the first base backup taken after it's turned on.

Staying on Hobby without SSH leaves a standard pg_dump over a database connection. That needs the Postgres service running instead of Sleeping, and if you connect from outside the project it goes over the database's public connection, which you said you want to avoid.


Status changed to Awaiting User Response brody • 8 days ago


brody

`railway postgres pitr backup create` is the CLI version of the Backups tab's on-demand volume backup, so it has the same plan requirement as the Backups tab. On Hobby it won't create a backup. Our earlier reply pointed you to that command without mentioning this, sorry for the confusion. Volume backups and point-in-time recovery are the supported ways to back up a database on Railway without SSH or public access, and both need Pro. Upgrading to Pro unlocks on-demand backups from the Backups tab or the CLI. Taking a volume backup doesn't change your existing data, and restores from it go to a new volume instead of overwriting your current one. Enabling PITR is different, because it redeploys the Postgres service, and its restore window only starts from the first base backup taken after it's turned on. Staying on Hobby without SSH leaves a standard `pg_dump` over a database connection. That needs the Postgres service running instead of Sleeping, and if you connect from outside the project it goes over the database's public connection, which you said you want to avoid.

pablogauzzi
HOBBYOP

8 days ago

Hi Brody,

Thank you for clarifying the backup limitations on the Hobby plan.

Before considering an upgrade to Pro, I would like to understand whether we can restore SSH access safely.

You mentioned that SSH is unavailable for services with Static Outbound IPs enabled following the October 21 incident.

Could you please confirm:

  1. How can I verify whether Static Outbound IP is enabled on my PostgreSQL service?
  2. If it is enabled, would disabling it restore SSH connectivity?
  3. Would disabling it cause a redeployment, downtime, or any risk to my existing PostgreSQL data?
  4. Is there an estimated timeline for restoring SSH functionality for affected services?

My priority is to create a safe backup without modifying or risking my production database.

Thank you.


Status changed to Awaiting Railway Response Railway • 8 days ago


8 days ago

Static Outbound IPs is a per-service setting, so you can check whether it is on from your Postgres service's settings. Our read of the service's configuration didn't let us confirm either way, so we can't say for certain that it's the cause in your case.

If it is enabled, SSH and the dashboard console are currently unavailable on that service, and that is the limitation turning it off would remove. SSH also needs a running container, and your Postgres deployment was showing as Sleeping, so the service has to be running before any SSH session can connect, whatever this setting says. Turning Static Outbound IPs off doesn't touch the data on your volume, but it does remove the service's assigned static addresses, and turning it back on later doesn't guarantee the same ones. That only matters if something outside Railway allowlists those addresses.

If you'd rather leave the setting alone, another route is to deploy a separate small service without Static Outbound IPs in the same project and use it as a jumpbox. Note that a jumpbox on its own doesn't open a shell inside the Postgres container.

We aim to resolve the SSH and Static Outbound IPs incompatibility in the coming weeks, but we can't give a concrete timeline.


Status changed to Awaiting User Response brody • 8 days ago


brody

Static Outbound IPs is a per-service setting, so you can check whether it is on from your Postgres service's settings. Our read of the service's configuration didn't let us confirm either way, so we can't say for certain that it's the cause in your case. If it is enabled, SSH and the dashboard console are currently unavailable on that service, and that is the limitation turning it off would remove. SSH also needs a running container, and your Postgres deployment was showing as Sleeping, so the service has to be running before any SSH session can connect, whatever this setting says. Turning Static Outbound IPs off doesn't touch the data on your volume, but it does remove the service's assigned static addresses, and turning it back on later doesn't guarantee the same ones. That only matters if something outside Railway allowlists those addresses. If you'd rather leave the setting alone, another route is to deploy a separate small service without Static Outbound IPs in the same project and use it as a jumpbox. Note that a jumpbox on its own doesn't open a shell inside the Postgres container. We aim to resolve the SSH and Static Outbound IPs incompatibility in the coming weeks, but we can't give a concrete timeline.

pablogauzzi
HOBBYOP

7 days ago

Hi Brody,

I checked the Postgres service settings and searched for "Static Outbound", but Railway shows "No settings matching your search".

Both my Postgres service and application are currently showing as Sleeping.

Could you please confirm whether Static Outbound IPs are enabled for my Postgres service, and where I can verify that setting?

My priority is to create a safe PostgreSQL backup on the Hobby plan without exposing the database publicly or risking production data.

If Static Outbound IPs are not enabled, would you recommend using a temporary jumpbox service to access Postgres through Railway's private network?

Thanks for your help.


Status changed to Awaiting Railway Response Railway • 7 days ago


7 days ago

Static Outbound IPs are only available on the Pro plan, and your workspace is on Hobby, so the setting is not shown in your Postgres service's settings. That explains the empty search, and it means Static Outbound IPs are not the reason SSH is failing for you.

The remaining blocker is that SSH connects to a running container, and your Postgres service is currently Sleeping. Once it is running again, railway ssh and railway connect can reach it directly, so a separate jumpbox is not needed for that.

For a backup on Hobby without SSH or public access, a pg_dump run from inside the same project connects over the private network rather than the database's public connection. Like any pg_dump, it only reads the database and does not modify your existing data, but it does need the Postgres service running.


Status changed to Awaiting User Response brody • 7 days ago


brody

Static Outbound IPs are only available on the Pro plan, and your workspace is on Hobby, so the setting is not shown in your Postgres service's settings. That explains the empty search, and it means Static Outbound IPs are not the reason SSH is failing for you. The remaining blocker is that SSH connects to a running container, and your Postgres service is currently Sleeping. Once it is running again, `railway ssh` and `railway connect` can reach it directly, so a separate jumpbox is not needed for that. For a backup on Hobby without SSH or public access, a `pg_dump` run from inside the same project connects over the private network rather than the database's public connection. Like any `pg_dump`, it only reads the database and does not modify your existing data, but it does need the Postgres service running.

pablogauzzi
HOBBYOP

6 days ago

Hi Brody,

I checked the Postgres service settings and confirmed that Serverless is already disabled.

However, the current deployment still shows Sleeping.

I also reviewed the deployment logs. PostgreSQL 18.6 started successfully, recognized the existing database volume, and reported "database system is ready to accept connections". Later, it received a fast shutdown request and shut down cleanly.

Given that Serverless is disabled, could you clarify why the deployment remains Sleeping?

What is the safest supported way to bring this existing PostgreSQL deployment back online without risking the persistent volume or database contents?

Would you recommend Redeploy in this situation, or is there a safer way to wake the existing deployment?

My goal is to perform a verified pg_dump backup before deploying application changes.

Thank you.


Status changed to Awaiting Railway Response Railway • 6 days ago


6 days ago

Your current Postgres deployment was created on October 26 at 19:23 UTC, shortly after the workspace moved onto Hobby, and its sleep state is fixed when a deployment's container is created. Changing the Serverless setting afterwards doesn't reach an existing deployment, so the setting now showing as disabled won't wake this one on its own.

Redeploying is the supported way to bring it back: it creates a new deployment using the current settings, with Serverless off. Your Postgres service has a persistent volume attached, and that volume stays attached to the new deployment, so the database contents carry over. Once it shows as running, you can take your pg_dump from inside the project over the private network before deploying your application changes.


Status changed to Awaiting User Response brody • 6 days ago


brody

Your current Postgres deployment was created on October 26 at 19:23 UTC, shortly after the workspace moved onto Hobby, and its sleep state is fixed when a deployment's container is created. Changing the Serverless setting afterwards doesn't reach an existing deployment, so the setting now showing as disabled won't wake this one on its own. Redeploying is the supported way to bring it back: it creates a new deployment using the current settings, with Serverless off. Your Postgres service has a persistent volume attached, and that volume stays attached to the new deployment, so the database contents carry over. Once it shows as running, you can take your pg_dump from inside the project over the private network before deploying your application changes.

pablogauzzi
HOBBYOP

an hour ago

Hi Brody,

Thank you for clarifying the PostgreSQL sleep-state issue and confirming that the persistent volume remains attached during redeployment.

Before proceeding, could you please confirm:

  1. Will redeploying the PostgreSQL service preserve the existing database volume and all stored data without any migration, reset, or initialization?
  2. Is there any risk that the redeployment could create or attach a different volume?
  3. Is there a way to back up the current persistent volume before redeploying, given that the PostgreSQL service is currently sleeping?
  4. What precautions should we take before redeploying this production PostgreSQL service?
  5. After redeployment, will Serverless remain disabled so PostgreSQL can stay available for the application?

We will not perform the redeployment until these points are clarified and we have an approved recovery procedure.

Thank you for your assistance.


Status changed to Awaiting Railway Response Railway • about 1 hour ago


Railway
BOT

26 minutes ago

Your Postgres service has one volume, mounted at /var/lib/postgresql/data, holding about 243 MB. It is the only volume in the project. A redeploy starts a new container on that same volume, so Postgres starts from its existing data directory and doesn't create a new database. Your earlier logs show this already: Postgres recognized the existing volume and reported it was ready for connections.

A redeploy doesn't create or attach a different volume. The volume binding only changes when you deploy a volume change yourself, like the staged swap after a backup restore, and there's none pending.

On Hobby you can't back up the volume before redeploying. Volume backups need Pro, and pg_dump needs Postgres running. If you want a snapshot first, upgrading to Pro lets you take one from the Backups tab without starting the service.

Before redeploying, check that postgres-volume still shows as attached to the service. Afterwards, check that the logs say "database system is ready to accept connections" and don't show a fresh initialization. Then run your pg_dump from inside the project before you deploy any application changes.

Serverless is currently off on your Postgres service, and the new deployment uses that setting, so it will stay running. Your FastAPI service is also Sleeping from a deployment made at the same time, so it needs a redeploy too.


Status changed to Awaiting User Response Railway • 26 minutes ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...