11 days ago
I've completed a full investigation with Railway Agent and confirmed this is a platform storage issue, not user error.
Evidence:
Volume mount is correct: bodasofhiyalex-volume (ID: cfc8778c-fbf7-49ae-9705-23d0e9cfeb2e) mounted to /app/data
Verified through console file browser: invitations.json exists on the volume but is completely empty (0 bytes)
Critical timestamp: File metadata shows 2026-08-09 22:22 — 2 hours and 17 minutes BEFORE the US West outage started (03:39 UTC)
My application code does NOT reinitialize or overwrite this file on startup (verified in codebase)
This was the ONLY copy of guest RSVP data (not in git, not backed up elsewhere)
What happened:
The file existed on the volume with data before the outage. After Railway's service recovery at 05:09 UTC, the file still exists but the content was completely wiped. The file metadata timestamp remained unchanged, proving the file wasn't deleted, it was truncated/corrupted during the storage backend recovery process.
What I need from Railway:
The US West outage incident affected our block storage backend. Can the Railway team check if there are snapshots or backups of the bodasofhiyalex-volume volume from before 2026-08-10 03:39 UTC? My invitations.json data should be recoverable from a pre-incident snapshot.
This is unacceptable for production data. I was about to migrate to Vercel Postgres for exactly this reason — to avoid data loss when using persistent volumes on Railway.
12 Replies
11 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 11 days ago
11 days ago
Was the JSON data stored in a dedicated volume or just within the container?
11 days ago
It was stored in a dedicated volume. The volume is named 'bodasofhiyalex-volume' and I had it correctly mounted to the path /app/data from day one. However, after today's US West outage and a subsequent container restart, the volume appears to be completely empty or corrupted. The filesystem mounted at /app/data no longer has my invitations.json file. Could the outage or the redeploy have wiped the persistent volume data?
onepenguinstudio
It was stored in a dedicated volume. The volume is named 'bodasofhiyalex-volume' and I had it correctly mounted to the path /app/data from day one. However, after today's US West outage and a subsequent container restart, the volume appears to be completely empty or corrupted. The filesystem mounted at /app/data no longer has my invitations.json file. Could the outage or the redeploy have wiped the persistent volume data?
11 days ago
Was the file stored under /app/data though? (Just a sanity check)
0x5b62656e5d
Was the file stored under `/app/data` though? (Just a sanity check)
11 days ago
Yes! The code uses path.join(process.cwd(), 'data', 'invitations.json'). Since the working directory in Railway's container is /app, the absolute path where the file is written is exactly /app/data/invitations.json.
11 days ago
I've confirmed the volume is correctly mounted (cfc8778c-fbf7-49ae-9705-23d0e9cfeb2e to /app/data) and the volume ID hasn't changed — no duplicate was created.
I also checked my code and confirmed it does NOT reinitialize or overwrite the file on startup. The only code that touches invitations.json is a read-only rescue endpoint. So the file wasn't cleared by my application.
The empty file came back after the service recovered from the outage at 05:09 UTC. This points to a storage backend issue during the incident recovery.
Request for Railway team: Can you check the underlying block storage snapshot for bodasofhiyalex-volume (ID: cfc8778c-fbf7-49ae-9705-23d0e9cfeb2e) from before the US West outage (before 2026-08-10 03:39 UTC)? My invitations.json data was there before the incident — it should be recoverable from the snapshot if it wasn't permanently lost.
11 days ago
I've confirmed through Railway Agent support chat:
Volume is correctly mounted (ID: cfc8778c-fbf7-49ae-9705-23d0e9cfeb2e to /app/data)
My code does NOT reinitialize or create the file on startup (verified in codebase)
The file was there before the outage, came back empty after recovery at 05:09 UTC
This is a data loss issue from the platform outage. Railway team: Can you check the block storage snapshot for this volume from before the incident started (2026-08-10 03:39 UTC)? My invitations.json data should be recoverable.
11 days ago
Can a Railway person help me please? 😔 My wedding invitations got lost and I’m really worried 😟
onepenguinstudio
I've confirmed through Railway Agent support chat: Volume is correctly mounted (ID: cfc8778c-fbf7-49ae-9705-23d0e9cfeb2e to /app/data) My code does NOT reinitialize or create the file on startup (verified in codebase) The file was there before the outage, came back empty after recovery at 05:09 UTC This is a data loss issue from the platform outage. Railway team: Can you check the block storage snapshot for this volume from before the incident started (2026-08-10 03:39 UTC)? My invitations.json data should be recoverable.
11 days ago
I’d suggest maybe disconnecting the volume from your service and trying to use the filebrowser template to see if anything is present in the volume. For a guide, look at this template
11 days ago
Following the milo's suggestion, I checked the volume using the file browser console and confirmed:
Volume is correctly mounted (ID: cfc8778c-fbf7-49ae-9705-23d0e9cfeb2e to /app/data)
My code does NOT reinitialize or create the file on startup (verified in codebase)
The file exists with timestamp 2026-08-09 22:22 — before the outage started at 03:39 UTC
The file is completely empty (0 bytes)
This proves the file was there with data, but the CONTENT was wiped during/after the outage recovery while the file metadata remained intact.
Railway team: Can you check the block storage snapshot for this volume from before the incident (before 2026-08-10 03:39 UTC)? My invitations.json data should be recoverable from the snapshot.
Attachments
11 days ago
Please Railway employees help me :( This are my wedding invitees and I really need to recover this file 😭
Volume ID: cfc8778c-fbf7-49ae-9705-23d0e9cfeb2e
File timestamp: 2026-08-09 22:22 (BEFORE outage)
Current state: 0 bytes (AFTER recovery)
Outage window: 2026-08-10 03:39-05:20 UTC
Request: Check block storage snapshots from before 03:39 UTC
11 days ago
I've completed a full investigation with Railway Agent and confirmed this is a platform storage issue, not user error.
Evidence:
Volume mount is correct: bodasofhiyalex-volume (ID: cfc8778c-fbf7-49ae-9705-23d0e9cfeb2e) mounted to /app/data
Verified through console file browser: invitations.json exists on the volume but is completely empty (0 bytes)
Critical timestamp: File metadata shows 2026-08-09 22:22 — 2 hours and 17 minutes BEFORE the US West outage started (03:39 UTC)
My application code does NOT reinitialize or overwrite this file on startup (verified in codebase)
This was the ONLY copy of guest RSVP data (not in git, not backed up elsewhere)
What happened:
The file existed on the volume with data before the outage. After Railway's service recovery at 05:09 UTC, the file still exists but the content was completely wiped. The file metadata timestamp remained unchanged, proving the file wasn't deleted, it was truncated/corrupted during the storage backend recovery process.
What I need from Railway:
The US West outage incident affected our block storage backend. Can the Railway team check if there are snapshots or backups of the bodasofhiyalex-volume volume from before 2026-08-10 03:39 UTC? My invitations.json data should be recoverable from a pre-incident snapshot.
This is unacceptable for production data. I was about to migrate to Vercel Postgres for exactly this reason — to avoid data loss when using persistent volumes on Railway.
Status changed to Awaiting Railway Response milo • 11 days ago
11 days ago
We're really sorry, but the RSVP data isn't recoverable on our side. We checked the storage host directly: there are no snapshots of this volume, and no backup schedules were ever configured, so there's no earlier copy to restore from.
We did check the platform side thoroughly. The volume was never moved or recreated, and it's still on the same host it's been on since July. The incident itself was network congestion in US West, which made services in the region slow or unreachable, and hosts in the region were under heavy load during that window. One thing worth noting from your screenshot: the file's last-modified time is about two hours before the outage began and didn't change through recovery. That would be unusual if the content had been wiped during the incident, since emptying a file normally updates its timestamp.
We know how much this data mattered, and we're sorry we can't bring it back. Enabling volume backups on anything you need to keep will give you a restore point going forward.
Status changed to Awaiting User Response Railway • 11 days ago
4 days ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 4 days ago