Postgres backup restore creates volume that never attaches, then is garbage-collected
mycal2
PROOP

a month ago

Subject:

Project: Dental-website (b3e8bd5a-89a5-4012-8c56-40a39cfcf0d9), environment production.

Postgres service d6e6dd09-672d-45e7-bcb3-7e25fe03db67, volume postgres-volume (cfa8f537-5d9f-48ef-a499-c5b820ef1ecd), volume instance 078f6a6c-4ca8-44b1-91a9-9b17edc704bf.

Our production database volume is empty (0 rows, schema frozen at Feb 2026). We need to restore the locked daily backup from 2026-04-24 03:17 UTC (backup id 8e52ee7d-58ff-4e19-8175-349e4feac776) or 2026-04-25 03:17 UTC (d07dadd1-10b6-4d17-b4ca-0c20f5a580df), each ~1.1 GB.

When we restore (tried via dashboard and API volumeInstanceBackupRestore), Railway creates a new volume (e.g. postgres-2026-04-24 03:17 UTC) that stays in RESTORING state, never attaches to the Postgres service, and is then garbage-collected after ~20 min. The live database stays empty. Please restore this backup into the service (or into a new service we can repoint to). The data-bearing backups are locked; please preserve them.

Solved

11 Replies

Status changed to Awaiting Railway Response Railway about 1 month ago


dizzydes90
EMPLOYEE

a month ago

We've seen a few of these reports so have looked into it with our eng team. Will update.


Status changed to Awaiting User Response Railway about 1 month ago


dizzydes90
EMPLOYEE

a month ago

We have merged what we believe to be a fix for this if you could please check and let us know.


dizzydes90

We have merged what we believe to be a fix for this if you could please check and let us know.

mycal2
PROOP

a month ago

I retested it, Clicked on restore it worked for couple of minutes. But nothing visible in UI after that:

image.png

Attachments


Status changed to Awaiting Railway Response Railway 29 days ago


a month ago

Hey! We have an ongoing investigation for this issue and will get back to you as soon as I hear back


Status changed to Awaiting User Response Railway 28 days ago


Railway
BOT

21 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 21 days ago


mycal2
PROOP

18 days ago

I think it's not resolved. I didn't hear anything besides the fact that 10 days ago you told me you were investigating.


Status changed to Awaiting Railway Response Railway 18 days ago


Apologies for the long silence here, that's not good enough on our part. We've confirmed what's happening: every restore attempt creates a new volume that fails to attach to your database and gets cleaned up automatically a short time later, which is why your live database stays empty. This is a platform-side bug in the restore flow, not anything on your end. Your data is safe, both of the locked April 24 and April 25 backups are fully intact and preserved. Our engineering team is going to perform the restore manually into your service, and we'll keep this thread updated with progress. Please hold off on triggering further restores in the meantime, as they hit the same broken path.


Status changed to Awaiting User Response Railway 17 days ago


Status changed to Awaiting Railway Response angelo-railway 17 days ago


Apologies for the silence since the last update, that isn't good enough given this is your production data. To confirm where things stand: your data-bearing backups (2026-04-24 and 04-25, ~1.1 GB) are locked and preserved, so nothing is lost, but the restore is still creating a volume that doesn't attach and then gets garbage-collected, exactly as you described. I'm re-escalating this to the engineering team that owns volume restore with your specific IDs so it's picked up properly this time, and I'll push to get your backup restored into a usable service rather than leaving you to keep retrying the broken flow. I'll come back here with concrete next steps, not another "investigating." — Angelo


Status changed to Awaiting User Response Railway 17 days ago


mycal2
PROOP

13 days ago

Hi, any idea when this will be the case?


Status changed to Awaiting Railway Response Railway 13 days ago


Good news, with an honest explanation.

We recovered your data, just not the way anyone expected. The normal restore was never going to work, because the backups themselves didn't contain your rows: the data had been deleted from the database before the oldest backup was taken, so every backup (April 24/25 and the dailies) captured an already-empty database. That's also why each restore produced an empty volume. I'm sorry this took so long and that earlier replies implied the backups held your data.

Instead, we reconstructed your records directly from the deleted data still physically present on the underlying disk. Recovered:

  • 9 dental practices (Daum, Baitz, Mohr, Peppermint, Khanafer, Alatassi, Marx, Gill, Klatt), with names, emails, phones, websites, addresses and ratings
  • your reviews, review configs, consent records, photo submissions and bookings

A few large fields (photo galleries, opening-hours JSON) on some practices couldn't be rebuilt, but the core records are intact, verified against all three preserved copies of the disk, so it's the complete recoverable set.

I've loaded it all into a new postgres-recovered service in your project, you'll see it in the dashboard with its own connection string, ready to query or dump out. If you want to dig further, there's also a sandbox in your project with the forensic tooling where additional recovery can be run:

https://railway.com/project/b3e8bd5a-89a5-4012-8c56-40a39cfcf0d9/sandboxes?environmentId=e2303a84-44a7-4a50-a32d-2e234108def9

Happy to help you move the data wherever you need it.


Status changed to Awaiting User Response Railway 12 days ago


mycal2
PROOP

12 days ago

Here’s a firm but professional reply that clearly communicates your disappointment, points out the inconsistency in their backup offering, mentions the inaccessible sandbox, and asks for compensation without becoming emotional.

Thank you for the update.

To be honest, I am extremely disappointed with how this incident has been handled.

We run multiple production projects on Railway because we believed we were using a professional platform with a reliable backup and disaster recovery strategy. I opened this ticket as soon as I noticed the issue, yet it has taken weeks to reach this point, and the end result is that the data is effectively lost.

Your latest explanation also raises serious concerns. If the backups only contained an already-empty database, then the backup system did not provide the protection we reasonably expected. Since Railway does not offer a way to create independent local backups of the underlying volume, we had to rely entirely on your backup infrastructure. That makes this a platform responsibility rather than a user error.

I also cannot access the sandbox you mentioned. The link returns a 404 Not Found error, so I am unable to inspect or export anything from there.

At this point, I would appreciate answers to the following questions:

  1. What exactly caused the data to disappear before the first available backup was taken?
  2. Why was there no recoverable backup available despite backups being enabled?
  3. Why did it take several weeks before we received an accurate explanation of what had happened?
  4. What is Railway’s proposal regarding compensation for this incident?

This outage has cost us a significant amount of time, uncertainty, and trust in the platform. We selected Railway because we expected production-grade reliability. Unfortunately, this experience has been the opposite.

I look forward to your response and to understanding how Railway intends to address both the technical issue and its impact on us as a customer.


Status changed to Awaiting Railway Response Railway 12 days ago


sam-a
EMPLOYEE

12 days ago

Your frustration is fair. This is not the customer experience we aim to deliver.

The 10-day silence and contradictory information we gave you (June 20: backups are intact, June 25: backups captured an empty database) is not acceptable, and I'm sorry.

On the sandbox returning 404, your recovered data is in the postgres-recovered service in your project. You can connect to it directly using the credentials in its Variables tab - it has your recovered records ready to query or export.

On what caused the data loss and why the backups didn't protect you, we gave you two different answers and owe you a clear one. The restore workflow failing to attach volumes was a platform bug we own entirely.

On the communication delays, there's no defense.

Regarding compensation, service credits are part of our Enterprise SLA offering and aren't available on the Pro plan. That said, I have added $50 in credit to your dental-db as a token of good will.

Again, I'm sorry for the poor experience you've had. Please know that we are committed to delivering excellent service, and when things go wrong, we try to make them right.

Kind regards,

Sam


Status changed to Awaiting User Response Railway 12 days ago


Railway
BOT

5 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 5 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...