21 days ago
Hello Railway Team!
My Postgres service has crashed after reaching 100% disk usage.
I have already increased the volume to 50GB, but the database still fails to start with a “postmaster.pid” lock.
Please remove the PID lock and restart the container — I need to recover the data from this volume.
Project link: https://railway.com/project/5532db3f-1268-4148-9fa5-a2a3df392989
Thank you!
8 Replies
21 days ago
Hey there! We've found the following might help you get unblocked faster:
🧵 I ran a recursive query and now there is no more space on device
🧵 Postgres went unresponsive, now frozen. Most likely issue with attached volume.
If you find the answer from one of these, please let us know by solving the thread!
21 days ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open brody • 21 days ago
21 days ago
hey, you can do this yourself by detatching the volume to PG service and mount it to another PG service, hook up to it via the public network, and run
ls -l /var/lib/postgresql/data/postmaster.pid || true
rm -f /var/lib/postgresql/data/postmaster.pid
you can read up on it here. then just remount the volume back to your PG service and start it. alternatively if you just want the data from the volume, you can use a template to get that data
20 days ago
Hello Railway Support Team,
I’m experiencing a critical issue preventing my PostGIS service from launching and accessing the existing PostgreSQL data volume that contains my n8n workflows.
Restore and start the PostGIS service without losing the existing database, so my n8n instance (Primary) can reconnect and load previous workflows.
Current Status
Project: courageous-creativity / production
Service: postgis
Region: EU West (Amsterdam)
Volume:
postgres-volume(~6 GB used — confirms database data exists)Mount Path: confirmed as
/mnt/railway/volumeinside the container
Core Problem
PostgreSQL cannot start using the existing data due to permission and context issues related to the mounted volume.postgres user cannot access /mnt/railway/volume even though the volume is attached and visible to root context.
Troubleshooting Steps Performed (all failed)
Command Attempted Run As Result / Error
su - postgres -c postgres could not access directory "/mnt/railway/volume": No such file or directory
"/usr/lib/postgresql/14/bin/postgres
-D /mnt/railway/volume"
chown -R postgres:postgres root cannot access '/mnt/railway/volume': No such file or directory
/mnt/railway/volume
/usr/lib/postgresql/14/bin/postgres root "root" execution of PostgreSQL server is not permitted
-D /mnt/railway/volume
PGDATA=/mnt/railway/volume entrypoint FATAL: database "railway" does not exist
(no custom command)
Diagnosis
It seems the Railway volume is mounted too late or with restricted permissions, preventing the postgres user from initializing or reading the existing data directory.
This blocks any normal recovery process or pg_dump access from inside the container.
Request
Could your team please help with one of the following options:
Adjust permissions or remount the volume
/mnt/railway/volumewithpostgres:postgres(UID/GID = 70) ownership,
so the container can access the existing cluster.If not possible, please provide a method to execute a
pg_dumpor read the data from the existing volume manually — so we can migrate to a new PostGIS instance safely.
Thank you very much for your assistance with this — this database contains essential production workflows, and any help to recover access to it would be greatly appreciated.
Best regards,
Evgeny Koptyakov
Project: courageous-creativity / production
koptyakov2709-ctrl
Hello Railway Support Team,I’m experiencing a critical issue preventing my PostGIS service from launching and accessing the existing PostgreSQL data volume that contains my n8n workflows.Restore and start the PostGIS service without losing the existing database, so my n8n instance (Primary) can reconnect and load previous workflows.Current StatusProject: courageous-creativity / productionService: postgisRegion: EU West (Amsterdam)Volume:postgres-volume (~6 GB used — confirms database data exists)Mount Path: confirmed as /mnt/railway/volume inside the containerCore ProblemPostgreSQL cannot start using the existing data due to permission and context issues related to the mounted volume.postgres user cannot access /mnt/railway/volume even though the volume is attached and visible to root context.Troubleshooting Steps Performed (all failed)Command Attempted Run As Result / Errorsu - postgres -c postgres could not access directory "/mnt/railway/volume": No such file or directory"/usr/lib/postgresql/14/bin/postgres-D /mnt/railway/volume"chown -R postgres:postgres root cannot access '/mnt/railway/volume': No such file or directory/mnt/railway/volume/usr/lib/postgresql/14/bin/postgres root "root" execution of PostgreSQL server is not permitted-D /mnt/railway/volumePGDATA=/mnt/railway/volume entrypoint FATAL: database "railway" does not exist(no custom command)DiagnosisIt seems the Railway volume is mounted too late or with restricted permissions, preventing the postgres user from initializing or reading the existing data directory.This blocks any normal recovery process or pg_dump access from inside the container.RequestCould your team please help with one of the following options:Adjust permissions or remount the volume /mnt/railway/volume with postgres:postgres (UID/GID = 70) ownership,so the container can access the existing cluster.If not possible, please provide a method to execute a pg_dump or read the data from the existing volume manually — so we can migrate to a new PostGIS instance safely.Thank you very much for your assistance with this — this database contains essential production workflows, and any help to recover access to it would be greatly appreciated.Best regards,Evgeny KoptyakovProject: courageous-creativity / production
20 days ago
Hey, please consider following the reply above. If you're unable to SSH into your service due to your PostgreSQL service crashing immediately, modify your startup command in the settings tab to "sleep 10000000", deploy the changes, and then try again. Re-deploy after making the necessary changes.
20 days ago
The PostGIS container is now running with sleep 10000000 and remains active, but there is no “Shell” or “Debug” option available in the UI.
Could your team please SSH into the container on my behalf and verify whether the volume (/mnt/railway/volume or /var/lib/postgresql/data) is accessible to the postgres user?
It contains ~6 GB of n8n workflow data.
Project: courageous-creativity / production
Service: postgis
koptyakov2709-ctrl
The PostGIS container is now running with sleep 10000000 and remains active, but there is no “Shell” or “Debug” option available in the UI.Could your team please SSH into the container on my behalf and verify whether the volume (/mnt/railway/volume or /var/lib/postgresql/data) is accessible to the postgres user?It contains ~6 GB of n8n workflow data.Project: courageous-creativity / productionService: postgis
19 days ago
For SSH you'll need to install the Railway's CLI: https://docs.railway.com/guides/cli#installing-the-cli and then SSH into your service: https://docs.railway.com/guides/cli#ssh
18 days ago
Hello Railway Support Team,
I’m writing to express my deep disappointment with how my support request was handled.
Despite following all the suggested steps provided in this thread — including detaching and remounting the volume, using the “sleep” command, and attempting to access the data via CLI — the issue with my PostGIS service and its attached volume was never resolved.
No member of the support team actually connected to the container to verify or adjust permissions on the mounted volume, even though this would have been a simple action from your side. As a result, I was left completely blocked and eventually had to delete all services and start over, losing more than two months of production data and configuration work.
I genuinely appreciate the platform and the community, but I’m very disappointed that direct assistance was not provided when it was clearly within Railway’s capabilities.
Please escalate this feedback to your technical management. I hope that in the future, your support team will take a more proactive approach when customers face critical production data access issues.
Best regards,
Evgeny Koptyakov
Project: courageous-creativity / production
koptyakov2709-ctrl
Hello Railway Support Team,I’m writing to express my deep disappointment with how my support request was handled.Despite following all the suggested steps provided in this thread — including detaching and remounting the volume, using the “sleep” command, and attempting to access the data via CLI — the issue with my PostGIS service and its attached volume was never resolved.No member of the support team actually connected to the container to verify or adjust permissions on the mounted volume, even though this would have been a simple action from your side. As a result, I was left completely blocked and eventually had to delete all services and start over, losing more than two months of production data and configuration work.I genuinely appreciate the platform and the community, but I’m very disappointed that direct assistance was not provided when it was clearly within Railway’s capabilities.Please escalate this feedback to your technical management. I hope that in the future, your support team will take a more proactive approach when customers face critical production data access issues.Best regards,Evgeny KoptyakovProject: courageous-creativity / production
17 days ago
Hey, unfortunately the team isn't able to provide application-level support, and this was clearly an application-level support issue.
If you still had any issues at that time, sending us a message on this thread would have allowed us to guide you toward the right solution. Deleting your production data definitely wasn't the right approach. If you still have that volume or service, we can help you recover it. I'm even able to record a video if necessary.
Let me know how you want to approach this! Again, we're here to help.