Database lost
frankieboxx
PROOP

a month ago

Dear Railway support, 

today we noticed that our database has been lost. We tried to restore from backups but seems there is no data in previous backups, what is really strange. 

Please look at attached screenshot of service where that happens. 

Service name is tendo-pgvector, project is  

Username is Frano Hartman, workspace is CAAS, project id 98a08102-3927-4206-bae3-06e3570b0943 (tendo-pgvector).

We had a scheduled backup on railway, but now backup list are empty what is also strange. What we did we recovered two volumes from backup but is seems that there are no data at all there.

Please help us solve this situation as we do not have another backup.

Best regards, Frano Hartman.

$20 Bounty

3 Replies

chandrika
EMPLOYEE

a month ago

We can see that your tendo-pgvector service is currently running, but the database has no tables - queries for tables like "Review" and "MasterCategory" are all returning "relation does not exist" errors. We also see your restore attempts from earlier today, including creating a volume from a snapshot.

To help investigate further, we need a few details: When did you last confirm the database had data in it? Did anything change recently, such as a redeployment, volume configuration change, or migration? Was the backup schedule configured on the original volume before the data went missing, or was it set up after?


Status changed to Awaiting User Response Railway about 1 month ago


Railway
BOT

a month ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 26 days ago


chandrika

We can see that your tendo-pgvector service is currently running, but the database has no tables - queries for tables like "Review" and "MasterCategory" are all returning "relation does not exist" errors. We also see your restore attempts from earlier today, including creating a volume from a snapshot. To help investigate further, we need a few details: When did you last confirm the database had data in it? Did anything change recently, such as a redeployment, volume configuration change, or migration? Was the backup schedule configured on the original volume before the data went missing, or was it set up after?

frankieboxx
PROOP

5 hours ago

Dear chandrika, thank you for your support and sorry for late response. I have investigate this data loss incident, so here are my experiences :

- database loss incidend happened when I tried to changed region for custom postgress volume with vector search : pgvector/pgvector:pg17

- at that time, I had a failover database on supabase, +one week old. I tried to restore volume from railway dashboard, without success.

- I restored +one week old failover database from supabase to railway, and continue as usual, except I wrote custom scripts to dump database from railway to local disk on a daily base.

- Few days ago, railway service stopped due to balance, I updated budget, services started to redeploy and same data loss incident happened again. Database was empty on custom pg volume. So I realized that I do not have redeploy persistence on my service : vectorpg2 (54e39607-419a-4d65-af84-5487aaabb9f8) due it is a custom docker image, not standard postgress deployment.

I restored my database from backup. So my question is :

- is this a normal situation with custom docker images?

-- if so can be avoid somehow?

Anyway, I just replied in case you did not experience such issue from other customers and you may instruct me how to avoid this issue in future.

I really this that railway offers great service, keep going on.

Thank, Frano.


Status changed to Awaiting Railway Response Railway about 5 hours ago


Railway
BOT

5 hours ago

This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.

Status changed to Open Railway about 5 hours ago


frankieboxx
PROOP

3 hours ago

Hi ryzon11,

I noticed that first data was missing on Sunday, 22.03.2026, 11:00 AM (GMT +2).
After I changed a docker file, railway deploys a new volume to tendo-pgvector, ID: 98a08102-3927-4206-bae3-06e3570b0943 and data was overwritten by a new volume. Tried to restore from backups, but as you mention backups on Railway are tied to volumes — volume was overwritten by new deployment and prior backups would did not work. Restored data from my previous strapi instance (old MySql) and +7 days old supabase database (11 hours of continuous work, so frustrating). I was trying all methods to recreate volume, nothing, even installed service to download volume, nothing. Restore backup (volume), nothing.
Yesterday, as I already write about, my budget was exhausted, and all services were paused. I update budget value, services started to deploy, tendo-pgvector also, and that service wipe out attached volume with pg data. Thanks god I had a working cron job which dumps tendo-pgvector to disk & github account. Again, need to reinstall volume, add vector search extension and restore backup from dump file.

My conclusion is that all redeployment (even change of region, GB, CPU-us, suspending due low budget) if service is custom docker image (or pgvector image) volume is overwritten and data is lost.

Regards, Frano.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...