3 months ago
My Postgres service stopped working after environment variables were removed by mistake and redeployed.
Now the container keeps printing:
"initdb: error: directory '/var/lib/postgresql/data' exists but is not empty"
This indicates the data volume already contains a Postgres cluster, but the container is repeatedly running initdb instead of using the existing data directory.
This database contains production client data. I need the Railway team to inspect the volume and prevent initdb from running. Please DO NOT wipe or recreate the volume.
Technical details:
- Postgres service name: clinic-management-db
- Issue started around: 2025-12-08 ~18:30 IST
- Logs show continuous initdb failure due to non-empty data directory.
I need:
1) Volume inspection (PG_VERSION, pg_wal, base).
2) Confirmation whether the existing cluster is intact.
3) Fix startup so Postgres uses the existing cluster.
4) Or snapshot recovery if available.
Please help urgently. This is data-critical.
Pinned Solution
3 months ago
I doubt railway can do anything in this case...this is a human error not an infrastructure issue. I would probably try to add those envs, there is no reason why it should work if, but maybe let's see if other people has better ideas
7 Replies
3 months ago
Which env did you removed?
fra
Which env did you removed?
3 months ago
Update / Clarification:
I accidentally removed environment variables for the Postgres service, then I re-added them and redeployed the DB service, but the container still fails to start and shows initdb errors.
What I removed earlier (before redeploy):
- PGHOST, PGPORT, PGUSER, PGPASSWORD, PGDATABASE, DATABASE_URL
- POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB
After re-adding these variables I redeployed. Redeploy / restart attempts happened around: 2025-12-08T18:36:00 → 2025-12-08T18:41:00 IST (see logs).
Current recurring log lines:
- "Error: Database is uninitialized and superuser password is not specified." (before I re-added POSTGRES_PASSWORD)
- After adding password: "initdb: error: directory '/var/lib/postgresql/data' exists but is not empty"
So sequence is:
1) I deleted vars by mistake.
2) I re-added vars (including POSTGRES_PASSWORD = bkxKgtWEdUzTQAbmODBlsXAbLlOWfkqN) and redeployed.
3) Now Postgres init is trying to run initdb against a non-empty data directory.
This DB contains client production data. Please do NOT wipe or reinitialize the volume.
Requested immediate actions from Railway team:
1) Mount/inspect the volume (read-only) and report:
- Output of: /var/lib/postgresql/data/PG_VERSION
- ls -la /var/lib/postgresql/data/pg_wal
- ls -la /var/lib/postgresql/data/base | head -n 20
2) If PG_VERSION exists and data looks like a valid cluster, please prevent initdb from running and start Postgres against the existing data directory.
3) If the volume is partially initialized/corrupted, please check for snapshots/backups and restore the latest good snapshot to a new instance for me to verify.
4) If you need deployment id / full logs / anything else from my side, tell me exactly what to paste here (I will supply immediately).
Thank you — this is data-critical.
3 months ago
I doubt railway can do anything in this case...this is a human error not an infrastructure issue. I would probably try to add those envs, there is no reason why it should work if, but maybe let's see if other people has better ideas
fra
I doubt railway can do anything in this case...this is a human error not an infrastructure issue. I would probably try to add those envs, there is no reason why it should work if, but maybe let's see if other people has better ideas
3 months ago
.
fra
I doubt railway can do anything in this case...this is a human error not an infrastructure issue. I would probably try to add those envs, there is no reason why it should work if, but maybe let's see if other people has better ideas
3 months ago
Thanks for the input.
Just to clarify: the environment variables were re-added immediately, but the PostgreSQL container is still running initdb repeatedly and refuses to start because the existing data directory is non-empty.
This is no longer an application-level issue. The volume now contains an existing Postgres cluster that the container is not able to use, which puts data at risk. This requires volume inspection (PG_VERSION, base/, pg_wal/) from Railway’s side.
If a snapshot exists, restoration would also require staff involvement.
So I am waiting for a Railway team member to check the volume state. Any guidance from staff would be appreciated.
3 months ago
Is the postgres version the same as before?
shashipareta12-stack
Thanks for the input.Just to clarify: the environment variables were re-added immediately, but the PostgreSQL container is still running initdb repeatedly and refuses to start because the existing data directory is non-empty.This is no longer an application-level issue. The volume now contains an existing Postgres cluster that the container is not able to use, which puts data at risk. This requires volume inspection (PG_VERSION, base/, pg_wal/) from Railway’s side.If a snapshot exists, restoration would also require staff involvement.So I am waiting for a Railway team member to check the volume state. Any guidance from staff would be appreciated.
3 months ago
for clarity, just because those postgres env vars were re-added later does not guarantee the container will ever start again by itself. likely, your only two options are:
start postgres directly against that volume in a fresh clean container, or
remount the volume onto a new postgres service and let the official entrypoint detect PG_VERSION without trying initdb. I added the fix below should you wish to remount the volume onto a new postgres service
if you didnt have any backups previously, you cannot restore data https://docs.railway.com/reference/backups. you have to ssh into the postgres service (right click and copy ssh command). check whether postgres is actually running or not, and inspect the data dir:
ps aux | grep postgres
ls -la /var/lib/postgresql/data
test -f /var/lib/postgresql/data/PG_VERSION && echo "PG cluster present"
if postgres isn't initilizing, then spin up a brand new postgres service, disconnect the existing volume from the failing service, and then attach the volume to the new clean postgres service and let it start up. the official postgres entrypoint will see PG_VERSION and start the existing cluster automatically, because this new instance isn’t stuck in an init loop. create a pgdump of your existing postgres db
pg_dump -U <root> -d <db> -f dump.sql
pg_dumpall -U <root> -f alldb.sql
and then reimport it to the new service psql -U <user> -d <db> -f dump.sql
id look into https://docs.railway.com/reference/support. railway doesnt provide application level support
Status changed to Open brody • 3 months ago
Status changed to Solved brody • 3 months ago