22 days ago
URGENT — PRODUCTION DATA LOSS
Our production Postgres database was accidentally wiped today.
We urgently need recovery help.
Incident time: 22 May 2026, ~08:20 UTC
Project: caring-cat (ID: 65b00a59-8915-4c0d-ab6c-e3fcf4e4551c)
Environment: production (ID: 5feee148-bd5a-41bc-bec6-fff406128635)
Service: Postgres (ID: 6dc93ebe-12da-4a47-bdf5-e6326212a5d0)
Volume: postgres-volume (ID: 6708c4ab-4aa1-4397-a16f-f5753b090e97, 50 GB)
Region: EU West (europe-west4-drams3a)
WHAT HAPPENED:
A Prisma command (prisma migrate diff --shadow-database-url) was
accidentally pointed at the production database. It dropped the entire
public schema — all tables and data. The database is now empty
(9.5 MB, 0 tables).
BACKUPS:
No backup schedule was configured. The only manual backup was created
AFTER the incident and is empty — it cannot be used.
WHAT WE NEED:
-
Restore from any infrastructure / GCP persistent-disk snapshot of the
postgres-volume taken BEFORE 08:18 UTC, 22 May 2026.
-
If no snapshot exists — block-level / forensic recovery of the volume.
The DROP only unlinked the files; the data is likely still physically
present on disk until overwritten.
CRITICAL:
Please do NOT let the volume be overwritten or re-initialised.
Please either stop the Postgres service on your side, or confirm it is
safe for us to stop it to preserve the volume.
This is a full production outage. Please respond as soon as possible.
2 Replies
Status changed to Awaiting Railway Response Railway • 22 days ago
Status changed to Solved yura250292 • 21 days ago
21 days ago
URGENT — PRODUCTION DATA LOSS
Our production Postgres database was accidentally wiped today.
We urgently need recovery help.
Incident time: 22 May 2026, ~08:20 UTC
Project: caring-cat (ID: 65b00a59-8915-4c0d-ab6c-e3fcf4e4551c)
Environment: production (ID: 5feee148-bd5a-41bc-bec6-fff406128635)
Service: Postgres (ID: 6dc93ebe-12da-4a47-bdf5-e6326212a5d0)
Volume: postgres-volume (ID: 6708c4ab-4aa1-4397-a16f-f5753b090e97, 50 GB)
Region: EU West (europe-west4-drams3a)
WHAT HAPPENED:
A Prisma command (prisma migrate diff --shadow-database-url) was
accidentally pointed at the production database. It dropped the entire
public schema — all tables and data. The database is now empty
(9.5 MB, 0 tables).
BACKUPS:
No backup schedule was configured. The only manual backup was created
AFTER the incident and is empty — it cannot be used.
WHAT WE NEED:
Restore from any infrastructure / GCP persistent-disk snapshot of the
postgres-volume taken BEFORE 08:18 UTC, 22 May 2026.
If no snapshot exists — block-level / forensic recovery of the volume.
The DROP only unlinked the files; the data is likely still physically
present on disk until overwritten.
CRITICAL:
Please do NOT let the volume be overwritten or re-initialised.
Please either stop the Postgres service on your side, or confirm it is
safe for us to stop it to preserve the volume.
This is a full production outage. Please respond as soon as possible.
Status changed to Awaiting Railway Response Railway • 21 days ago
19 days ago
We don't offer data restoration for user-initiated actions like this. Going forward, enable volume backups on your service so you can self-restore if it happens again.
Status changed to Awaiting User Response Railway • 19 days ago
Status changed to Solved mykal • 19 days ago