Production Data Loss — Postgres database wiped (caring-cat)
yura250292
PROOP

a month ago

URGENT — PRODUCTION DATA LOSS

Our production Postgres database was accidentally wiped today.

We urgently need recovery help.

Incident time: 22 May 2026, ~08:20 UTC

Project: caring-cat (ID: 65b00a59-8915-4c0d-ab6c-e3fcf4e4551c)

Environment: production (ID: 5feee148-bd5a-41bc-bec6-fff406128635)

Service: Postgres (ID: 6dc93ebe-12da-4a47-bdf5-e6326212a5d0)

Volume: postgres-volume (ID: 6708c4ab-4aa1-4397-a16f-f5753b090e97, 50 GB)

Region: EU West (europe-west4-drams3a)

WHAT HAPPENED:

A Prisma command (prisma migrate diff --shadow-database-url) was

accidentally pointed at the production database. It dropped the entire

public schema — all tables and data. The database is now empty

(9.5 MB, 0 tables).

BACKUPS:

No backup schedule was configured. The only manual backup was created

AFTER the incident and is empty — it cannot be used.

WHAT WE NEED:

  1. Restore from any infrastructure / GCP persistent-disk snapshot of the

    postgres-volume taken BEFORE 08:18 UTC, 22 May 2026.

  2. If no snapshot exists — block-level / forensic recovery of the volume.

    The DROP only unlinked the files; the data is likely still physically

    present on disk until overwritten.

CRITICAL:

Please do NOT let the volume be overwritten or re-initialised.

Please either stop the Postgres service on your side, or confirm it is

safe for us to stop it to preserve the volume.

This is a full production outage. Please respond as soon as possible.

Solved

2 Replies

Status changed to Awaiting Railway Response Railway about 1 month ago


Status changed to Solved yura250292 about 1 month ago


yura250292
PROOP

a month ago

URGENT — PRODUCTION DATA LOSS

Our production Postgres database was accidentally wiped today.

We urgently need recovery help.

Incident time: 22 May 2026, ~08:20 UTC

Project: caring-cat (ID: 65b00a59-8915-4c0d-ab6c-e3fcf4e4551c)

Environment: production (ID: 5feee148-bd5a-41bc-bec6-fff406128635)

Service: Postgres (ID: 6dc93ebe-12da-4a47-bdf5-e6326212a5d0)

Volume: postgres-volume (ID: 6708c4ab-4aa1-4397-a16f-f5753b090e97, 50 GB)

Region: EU West (europe-west4-drams3a)

WHAT HAPPENED:

A Prisma command (prisma migrate diff --shadow-database-url) was

accidentally pointed at the production database. It dropped the entire

public schema — all tables and data. The database is now empty

(9.5 MB, 0 tables).

BACKUPS:

No backup schedule was configured. The only manual backup was created

AFTER the incident and is empty — it cannot be used.

WHAT WE NEED:

Restore from any infrastructure / GCP persistent-disk snapshot of the

postgres-volume taken BEFORE 08:18 UTC, 22 May 2026.

If no snapshot exists — block-level / forensic recovery of the volume.

The DROP only unlinked the files; the data is likely still physically

present on disk until overwritten.

CRITICAL:

Please do NOT let the volume be overwritten or re-initialised.

Please either stop the Postgres service on your side, or confirm it is

safe for us to stop it to preserve the volume.

This is a full production outage. Please respond as soon as possible.


Status changed to Awaiting Railway Response Railway about 1 month ago


a month ago

We don't offer data restoration for user-initiated actions like this. Going forward, enable volume backups on your service so you can self-restore if it happens again.


Status changed to Awaiting User Response Railway about 1 month ago


Status changed to Solved mykal about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...