Postgres crashes on startup with
autogrpaph
HOBBYOP

3 months ago

Postgres crashes on startup with
PANIC: could not find free replication state, increase max_replication_slots
Please help recover or fix the database. I cannot change max_replication_slots from the UI.

Solved$10 Bounty

3 Replies

3 months ago

This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.

Status changed to Open noahd 3 months ago


Status changed to Awaiting Conductor Response noahd 3 months ago


3 months ago

Railway lets you modify PostgreSQL config using ALTER SYSTEM commands. Since your database is stuck in a crash loop, you'll need to SSH in and adjust the settings directly.

Let me know if this works for you! I've seen a few other's having this issue lately so may be an underlying Railway problem but this should resolve things regardless.

Fixing it with SSH sunglasses emoji:

Grab the SSH command from Railway (right-click your PostgreSQL service -> "Copy SSH Command") or use:

railway ssh --service=<your-postgres-service-name>

then bump up the replication slots; connect to postgresql via the CLI. Just use the username/password in your service's environment variables. should work fine.

psql -U postgres

and increase the limit:

ALTER SYSTEM SET max_replication_slots = '10';
SELECT pg_reload_conf();

-- if you're hitting other replication limits in your logs you can fix those too iirc 
ALTER SYSTEM SET max_wal_senders = '10';
ALTER SYSTEM SET max_replication_slots = '10';
SELECT pg_reload_conf();

then all you should have to do is restart the PostgreSQL service. (click the 3-dot menu -> Restart).

According to the Railway PostgreSQL documentation, the ALTER SYSTEM command persists configuration changes across restarts. The changes are stored in postgresql.auto.conf and will remain even after restart.

Another tip

I found a few articles that mention this error is caused by PostgreSQL having orphaned replication slots in the system. You should be able to check and drop the orphaned slots:

SELECT * FROM pg_replication_slots; 

-- then drop the ones you don't want 
SELECT pg_drop_replication_slot('slot_name');

autogrpaph
HOBBYOP

3 months ago

I cannot apply your suggested fix because the PostgreSQL service does not reach a running state.

It crashes during recovery BEFORE SSH becomes available.

Railway CLI always returns:

"Expected welcome message... Your application is not running or in an unexpected state"

This means I cannot:

- ssh into the container

- run psql

- run ALTER SYSTEM

- drop replication slots

The database never reaches "ready to accept connections".

This is a crash during recovery, not a runtime config issue.

The PANIC happens because of corrupted replication state files inside PGDATA (pg_replslot).

To fix this I need one of the following:

1. Railway team to clear pg_replslot/* inside the data volume

2. OR mount the data volume (vol_wjme57m3jkwex0ce) into a temporary container so I can repair it manually and dump the data.

This database contains important production data and has no backups.

Please help repair the damaged PGDATA volume or allow me access to it.


3 months ago

You can manage to SSH in by changing your start command to be sleep 9999999. This will put the container in an idle state and allow you to get in and rummage around.


Status changed to Awaiting User Response Railway 3 months ago


Railway
BOT

3 months ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 3 months ago


Loading...