Production Data Loss — Postgres database wiped (caring-cat)
yura250292
PROOP

22 days ago

URGENT — PRODUCTION DATA LOSS

Our production Postgres database was accidentally wiped today.

We urgently need recovery help.

Incident time: 22 May 2026, ~08:20 UTC

Project: caring-cat (ID: 65b00a59-8915-4c0d-ab6c-e3fcf4e4551c)

Environment: production (ID: 5feee148-bd5a-41bc-bec6-fff406128635)

Service: Postgres (ID: 6dc93ebe-12da-4a47-bdf5-e6326212a5d0)

Volume: postgres-volume (ID: 6708c4ab-4aa1-4397-a16f-f5753b090e97, 50 GB)

Region: EU West (europe-west4-drams3a)

WHAT HAPPENED:

A Prisma command (prisma migrate diff --shadow-database-url) was

accidentally pointed at the production database. It dropped the entire

public schema — all tables and data. The database is now empty

(9.5 MB, 0 tables).

BACKUPS:

No backup schedule was configured. The only manual backup was created

AFTER the incident and is empty — it cannot be used.

WHAT WE NEED:

  1. Restore from any infrastructure / GCP persistent-disk snapshot of the

    postgres-volume taken BEFORE 08:18 UTC, 22 May 2026.

  2. If no snapshot exists — block-level / forensic recovery of the volume.

    The DROP only unlinked the files; the data is likely still physically

    present on disk until overwritten.

CRITICAL:

Please do NOT let the volume be overwritten or re-initialised.

Please either stop the Postgres service on your side, or confirm it is

safe for us to stop it to preserve the volume.

This is a full production outage. Please respond as soon as possible.

Solved

2 Replies

Status changed to Awaiting Railway Response Railway 22 days ago


Status changed to Solved yura250292 21 days ago


yura250292
PROOP

21 days ago

URGENT — PRODUCTION DATA LOSS

Our production Postgres database was accidentally wiped today.

We urgently need recovery help.

Incident time: 22 May 2026, ~08:20 UTC

Project: caring-cat (ID: 65b00a59-8915-4c0d-ab6c-e3fcf4e4551c)

Environment: production (ID: 5feee148-bd5a-41bc-bec6-fff406128635)

Service: Postgres (ID: 6dc93ebe-12da-4a47-bdf5-e6326212a5d0)

Volume: postgres-volume (ID: 6708c4ab-4aa1-4397-a16f-f5753b090e97, 50 GB)

Region: EU West (europe-west4-drams3a)

WHAT HAPPENED:

A Prisma command (prisma migrate diff --shadow-database-url) was

accidentally pointed at the production database. It dropped the entire

public schema — all tables and data. The database is now empty

(9.5 MB, 0 tables).

BACKUPS:

No backup schedule was configured. The only manual backup was created

AFTER the incident and is empty — it cannot be used.

WHAT WE NEED:

Restore from any infrastructure / GCP persistent-disk snapshot of the

postgres-volume taken BEFORE 08:18 UTC, 22 May 2026.

If no snapshot exists — block-level / forensic recovery of the volume.

The DROP only unlinked the files; the data is likely still physically

present on disk until overwritten.

CRITICAL:

Please do NOT let the volume be overwritten or re-initialised.

Please either stop the Postgres service on your side, or confirm it is

safe for us to stop it to preserve the volume.

This is a full production outage. Please respond as soon as possible.


Status changed to Awaiting Railway Response Railway 21 days ago


19 days ago

We don't offer data restoration for user-initiated actions like this. Going forward, enable volume backups on your service so you can self-restore if it happens again.


Status changed to Awaiting User Response Railway 19 days ago


Status changed to Solved mykal 19 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...