a month ago
Production data in our Lockflow job-management app has reverted to an old database and we urgently need a storage-level recovery.
Project: c9be8119-cf0c-4baa-91e4-3fd1cf67d57f
Environment: production (bb13066d-01e1-457f-80e6-67319557163d)
Service: accurate-recreation (8fdf6751-f519-4414-91e0-d50c3ebf168c)
Volume: accurate-recreation-volume (3e322fcc-71e6-4908-804f-b20eccae89f7)
Volume instance: 7a37fb61-feaf-40a4-abbc-15a2385be073
Mount: /lockflow
Database path: /lockflow/data/jobs.json
The production database previously contained jobs through ASJ-1082. It now contains only 21 records and stops at ASJ-1021. The current jobs.json timestamp is 2026-07-26 23:07 UTC. The volume was effectively full (500 MB allocation; currently ext4 reports about 8.3 MB available), and the app had previously logged disk-full errors while writing automatic backups.
We checked Railway volume backups via the API; volumeInstanceBackupList is empty. We also checked all surviving files under /lockflow/data/backups and the rest of the mounted volume. Only post-reversion backups remain. debugfs lsdel returns no recoverable deleted inodes because the mounted ext4 device uses discard. The prior deployment containers are REMOVED and cannot be reached by deployment-instance SSH.
Please urgently check Railway infrastructure-level snapshots/backups for this volume from BEFORE 2026-07-26 22:00 UTC, ideally the newest copy containing ASJ-1082. Do not overwrite the current volume without confirming the recovery point first. This is live business data and exact recovery is critical.
2 Replies
Status changed to Awaiting Railway Response Railway • 26 days ago
25 days ago
URGENT FOLLOW-UP: This is active production business data and the surviving local files do not contain the original records after ASJ-1030. We have reconstructed what can be proven, but exact customer names, job descriptions, financial fields, assignments, notes and statuses remain missing. Please escalate this immediately to the infrastructure/storage team and check for any platform-level volume snapshot from before 2026-07-26 22:00 UTC, preferably the newest snapshot containing jobs through ASJ-1082. Please do not overwrite or roll back the current live volume. First confirm whether a snapshot exists and provide a safe clone/export or exact recovery procedure so we can verify it before applying anything.
25 days ago
Your volume was at capacity (500 MB) when the application attempted to write to jobs.json, and that write failed due to the disk being full. This is the most likely cause of the data reverting to an older state - the on-disk file was left with whatever the last successful write contained. Volume data is unmanaged, and without a backup solution configured there isn't anything we can do to recover the lost records. Going forward, upgrading to Pro unlocks two features that would prevent this: native volume backups for self-serve restore points, and disk usage monitors that alert you before a volume fills up.
Status changed to Awaiting User Response Railway • 25 days ago
Status changed to Solved mykal • 25 days ago