5 months ago
Hello Railway Support Team,
I’m currently using Railway's PostgreSQL managed service, and we’ve encountered critical data loss on our production DB.
Problem Summary:
Database: PostgreSQL on Railway
Problem: All data between
2025-04-25and2025-06-24is missingSuspected Cause: PostgreSQL log shows WAL (Write-Ahead Log) corruption and automatic recovery behavior
Key log output:
LOG: database system was interrupted; last known up at 2025-06-24 14:58:00 UTC LOG: invalid record length at 0/303B580: expected at least 24, got 0 LOG: redo done at 0/303B548
The system appears to have rolled back to a much earlier checkpoint (around April 24), losing nearly 2 months of transactional data.
6 Replies
5 months ago
Hello!
We've escalated your issue to our engineering team.
We aim to provide an update within 1 business day.
Please reply to this thread if you have any questions!
Status changed to Awaiting User Response Railway • 5 months ago
5 months ago
Hello there,
I am extremely sorry that we have placed your volume into an incorrect state. I have contacted the platform team to initiate a disaster recovery, as such, you should see a volume called: data-recovery
This volume should have your data pre-recovery. After we confirm that your data is in a good spot, we can talk about after action.
angelo-railway
Hello there,I am extremely sorry that we have placed your volume into an incorrect state. I have contacted the platform team to initiate a disaster recovery, as such, you should see a volume called: data-recovery This volume should have your data pre-recovery. After we confirm that your data is in a good spot, we can talk about after action.
5 months ago
The volume called 'data-recovery' only has data up to April 24th. I need data from April 24th to June 24th.
Status changed to Awaiting Railway Response Railway • 5 months ago
Status changed to Awaiting User Response itsrems • 5 months ago
ray-chen
Hey! Did you restore a backup prior to the migration?
5 months ago
what do you mean? i restored data-recovery but it doesn't have data what i want
Status changed to Awaiting Railway Response Railway • 5 months ago
5 months ago
Hello,
Based on the project activity logs, I can clarify what occurred with your PostgreSQL database.
On July 24th, you initiated a backup restoration from April 24th. When restoring a backup, a new volume is created with the name corresponding to the backup date, and the original volume remains on your project canvas.
The backup volume (2025-04-24 at 15:07) is currently connected to your Postgres service, which is why you're seeing data from April 24th. Your original volume (kitty-volume) is still available on your canvas, currently connected to your kuhidbs3 service, and contains your data through June 24th.
This backup restoration that you performed is what caused the apparent data loss you're experiencing - the data from April 25th through June 24th appears missing because your service is now connected to the earlier backup rather than your current data.
I should also note that while Railway provides the ability to host a PostgreSQL database, the databases themselves are entirely self-managed.
Best regards,
Brody
Status changed to Awaiting User Response Railway • 5 months ago
4 months ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 4 months ago
3 months ago
❌ The ticket Data restoration issue has been marked as canceled.