3 months ago
I woke up today to customers complaining that many documents were not accessible in our app. I quickly checked the logs and found what was happening, especially since we hadn’t made any changes that could have caused this.
I reviewed several backend logs indicating that the documents requested by clients were not available. At first, I thought there might have been some data corruption or that the customers had deleted the files themselves. However, more than 10 customers started calling me, saying their documents were not appearing.
What happened? Our attached volume was simply erased. It went from more than 150 MB to 0. And this didn’t affect only production — our staging and development environments were impacted as well.
What is happening with Railway? I’m going to migrate to a real cloud provider.
Below are the graphs from 7 days ago and from 1 day ago. I don’t know what happened or whether this is a temporary issue, but I’m losing customer confidence due to Railway outages and related issues.
6 Replies
Status changed to Awaiting Railway Response Railway • 3 months ago
3 months ago
Hello!
We've escalated your issue to our engineering team.
We aim to provide an update within 1 business day.
Please reply to this thread if you have any questions!
Status changed to Awaiting User Response Railway • 3 months ago
3 months ago
We're investigating what happened to your volume data and have escalated this to our engineering team. To help us narrow down the root cause, could you share your Dockerfile (at /app/apa-backend/deploy/Dockerfile)? We want to rule out whether the build process interacts with the /app/uploads directory. Also, do you know approximately when documents stopped being accessible - was it right after the Feb 28 deploys, or did it happen later? If you have any volume backups configured, check the Backups tab on your backend service for a possible restore while we investigate.
3 months ago
I don't have volumes deploy since I'm using hobby plan. I noticed the volume erased today, so I can't say it was related to deploy. Below is my Dockerfile.
Attachments
Status changed to Awaiting Railway Response Railway • 3 months ago
3 months ago
The 0B metrics you're seeing is a visual issue we're looking into at the moment. There is still data in your volume.
Is your data selectively missing? I can see that there's a bunch of data in your database. Can you clarify which service data is missing on, since you also have a service with a volume attached (https://railway.com/project/af1e9fea-23a6-45b5-bf5d-82a84be6f22c/volume/d0f39fe5-8317-4701-b264-06f6ec375b43/metrics?environmentId=682886ba-cfa0-4341-8e8f-c87bfc296250)?
Status changed to Awaiting User Response Railway • 3 months ago
3 months ago
The service is backend, I checked that database was unnafected
Status changed to Awaiting Railway Response Railway • 3 months ago
3 months ago
/app # du -sh /app/uploads
90M /app/uploadsYour data appears to be intact? If any data is selectively missing, it's highly unlikely any erasure came from our end - in those events, files wouldn't be missing individually, and the entire volume would have been wiped/corrupted, but I see no evidence of that on our end
Status changed to Awaiting User Response Railway • 3 months ago
kaioenzo
The service is backend, I checked that database was unnafected
3 months ago
Hey there! As my colleague Ray pointed out, your volume wasn’t wiped. This was a visual glitch in the Volume Usage chart affecting certain time ranges. We’ve just pushed a fix, and it should now look as expected.
2 months ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 2 months ago