3 months ago
Hi Railway Support,
I’m experiencing a critical issue with a Railway Postgres service: our database suddenly has no application tables (e.g., credit no longer exists). We also observed an unexpected deploy/restart that no one triggered.
Project/Env/Service details:
Project ID:
e6677ad8-a8d5-4708-b993-1027bc71c750Environment ID:
4d525208-c560-4b1c-b1bf-136ce061b3f7Postgres service ID:
5421a792-d8bf-473a-80d6-cda0bae2c713Volume:
vol_ml9vpxc49q16nmh0(mounted on multiple replicas)
What we see in logs (UTC):
2025-12-18 15:34:05a client attemptedCREATE TABLE IF NOT EXISTS arc (...)but it failed with a syntax error (trailing comma).logs.1766089095058
Shortly after, Postgres shows “database system was interrupted / not properly shut down; automatic recovery in progress”.
Starting around
2025-12-18 16:00:00, the DB repeatedly logs:ERROR: relation "credit" does not exist, while our app continues querying it.There are multiple container stop/start events with SIGTERM.
Request:
Can you confirm what triggered the unexpected redeploy/restarts? (platform action, health check, config change, etc.)
Can you verify whether the volume was reattached, rolled back, or recreated, or if there were any storage incidents affecting this Postgres service?
Are there any available snapshots/backups or ways to recover the previous data state?
Is it possible the service is pointing to a different database inside the same cluster (dbname changed), or can you confirm the configured database name didn’t change?
This is production-impacting. Any help or guidance to restore data (or confirm root cause) would be appreciated.
6 Replies
3 months ago
Hello.
Railway won't restart your databases or services for any health check or config change. Any restart or redeploy would have to be done by the end user.
Unfortunately, unless you used Railways backup features or stored a local copy of you're data theres no way for us to restore the data as we simply don't store it.
As for a separate db name, we haven't modified or changed any databases. You would need to investigate that on your end to be sure. It does appear the volume attached has ~536mb so there might be some data.
I can open this up to the community to help debug if you'd like! Unfortunately due to support volume I won't be able to help debug this fully.
Status changed to Awaiting User Response Railway • 3 months ago
noahd
Hello.Railway won't restart your databases or services for any health check or config change. Any restart or redeploy would have to be done by the end user. Unfortunately, unless you used Railways backup features or stored a local copy of you're data theres no way for us to restore the data as we simply don't store it. As for a separate db name, we haven't modified or changed any databases. You would need to investigate that on your end to be sure. It does appear the volume attached has ~536mb so there might be some data. I can open this up to the community to help debug if you'd like! Unfortunately due to support volume I won't be able to help debug this fully.
3 months ago
How is it possible that the attached file has a size of 536 MB if it’s supposedly empty? Is it possible that the data is still there?
Status changed to Awaiting Railway Response Railway • 3 months ago
noahd
Hello.Railway won't restart your databases or services for any health check or config change. Any restart or redeploy would have to be done by the end user. Unfortunately, unless you used Railways backup features or stored a local copy of you're data theres no way for us to restore the data as we simply don't store it. As for a separate db name, we haven't modified or changed any databases. You would need to investigate that on your end to be sure. It does appear the volume attached has ~536mb so there might be some data. I can open this up to the community to help debug if you'd like! Unfortunately due to support volume I won't be able to help debug this fully.
3 months ago
I’d also like to open this up to the community to see if this has happened to anyone else and to try to find a solution.
3 months ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open brody • 3 months ago
3 months ago
Did you try to connect to the dB from your local machine with something like tableplus/pgadmin? Do you have a custom docker image for the dB? I wonder if by any cache the Pg version changed after the redeploy, maybe you can try do download the Pg data from the volume (so at least you have some "broken" backup)
Have a look to this thread, they suggest to use pg_resetxlog:
I'm not expert and I never done this before, so do your research as well
fra
Did you try to connect to the dB from your local machine with something like tableplus/pgadmin? Do you have a custom docker image for the dB? I wonder if by any cache the Pg version changed after the redeploy, maybe you can try do download the Pg data from the volume (so at least you have some "broken" backup)Have a look to this thread, they suggest to use pg_resetxlog:https://stackoverflow.com/questions/598200/how-do-i-fix-postgres-so-it-will-start-after-an-abrupt-shutdownI'm not expert and I never done this before, so do your research as well
3 months ago
And how can I download the data from the volume?
3 months ago
In theory you should be able to ssh into the postgres container (left click on the service, copy ssh command) and then use something like scp? This is what I would do, but I'm not a server admin, only a self taught, so as I said do some research 