Production Postgres tables missing after Prisma shadow DB command
bushmasterhtps
PROOP

16 days ago

Project: sublime-gratitude

Environment: production

Service: Postgres

Database: railway

Host: interchange.proxy.rlwy.net:43835

Our production app is down. The database previously contained all production application tables such as User, Task, RawMessage, etc. It now contains only _prisma_migrations.

Incident timeline:

- App was working earlier today.

- A Prisma command was accidentally run with the production DATABASE_URL as the shadow DB URL:

npx prisma migrate diff --from-migrations prisma/migrations --to-schema-datamodel prisma/schema.prisma --shadow-database-url "$DATABASE_URL"

- Afterward, the DB no longer showed application tables.

- A later npx prisma migrate deploy failed on:

20241229180000_add_task_performance_index

ERROR: relation "public.Task" does not exist

- Current _prisma_migrations only shows the failed migration row from 2026-05-07 02:12 UTC.

- Railway Backups UI shows no backups.

We urgently need to know if Railway can recover this Postgres volume/database to a point before 2026-05-07 02:12 UTC, using any platform-level snapshot, volume recovery, WAL/PITR, or internal recovery option. This is a production outage affecting tomorrow’s operations.

Solved

7 Replies

Status changed to Awaiting Railway Response Railway 16 days ago


bushmasterhtps
PROOP

15 days ago

Hi Railway Support or anybody,

We urgently need help determining whether recovery is possible for our Postgres database.

A Prisma command was run where the production database URL may have been used as the shadow database URL during a migration diff/schema troubleshooting step. After that, our production database appears to have lost historical application data, including task/productivity records going back to last year.

Database details:

- Host: interchange.proxy.rlwy.net

- Database: railway

- Service: Railway Postgres

- Approximate incident window: May 6–7, 2026, around the evening/night PT

We understand scheduled backups may not have been enabled. Can you please confirm whether there are any platform-level snapshots, volume backups, WAL/archive recovery, internal disaster recovery options, or any other way to restore or clone the database to a point before the incident?

This database contained operational productivity/task history used for department reporting, so even a partial restore or clone would be extremely valuable. We do not want to overwrite the current database if recovery is possible; ideally we would like a restored copy/clone to inspect first.

Please help ☹


bushmasterhtps
PROOP

15 days ago

URGENT UPDATE — DATABASE STATE CONFIRMED

We connected directly to the active Railway Postgres instance via psql and confirmed the following:

- Current application tables exist because we rebuilt the schema after the incident to restore operations.

- However, the active data only begins on 2026-05-07.

- Historical production records from the prior ~9 months are no longer present in the active database.

Examples confirmed via SQL:

- Task oldest createdAt: 2026-05-07 07:40 UTC

- RawMessage oldest createdAt: 2026-05-07 07:40 UTC

- User oldest createdAt: 2026-05-07 03:37 UTC

We also confirmed:

- Postgres itself is healthy/responding

- The current database is operational

- No alternate schemas containing the old tables were found

- Current data appears to be from the rebuilt environment after the incident

We are urgently requesting confirmation whether Railway has ANY infrastructure-level recovery options available, including:

- WAL recovery

- PITR (point-in-time recovery)

- storage snapshots

- volume backups

- internal disaster recovery

- cloned restore from a previous state

IMPORTANT:

Please do NOT overwrite the current database.

If recovery is possible, we would strongly prefer:

- a cloned/restored Postgres instance

- restored from ANY point prior to approximately:

May 6, 2026 around 4:00 PM PT

An exact timestamp is not necessary. Even a partial or slightly older restore would be extremely valuable.

Project details:

- Project: sublime-gratitude

- Environment: production

- Database: railway

- Host: interchange.proxy.rlwy.net

This database contained operational productivity/task history used for reporting and department operations. Recovering the historical data would be critically important.

Thank you.


bushmasterhtps
PROOP

15 days ago

Additional note:

I attempted to join the Railway Discord to escalate this production recovery request, but the Railway ↔ Discord OAuth verification flow is currently failing with:

“Problem completing OAuth login”

So I am unable to post in the support/help channels there at the moment.

Please let me know if there is another direct escalation/support route available for urgent production Postgres recovery requests.


15 days ago

We don't offer data restoration for user-initiated actions like this. Going forward, enable volume backups on your Postgres service so you can self-restore if it happens again.


Status changed to Awaiting User Response Railway 15 days ago


bushmasterhtps
PROOP

14 days ago

Hi Railway Support,

Thank you for clarifying that Railway does not offer data restoration for user-initiated actions and that we should enable volume backups going forward.

Before we fully close this out, I wanted to ask one final recovery-related question:

Even if a managed restore is not available, is there anything Railway can provide that may help us reconstruct or audit the lost Postgres data?

Specifically, are any of the following available on your side?

1. Any internal database volume snapshots, temporary snapshots, filesystem-level remnants, or retained volume artifacts from before the incident.

2. Any deployment/database event logs showing when the destructive action occurred and what database/volume object was affected.

3. Any connection/query/audit metadata that could confirm whether the data was dropped, truncated, overwritten, or affected by a migration/shadow DB operation.

4. Any way to export raw remnants, partial volume contents, WAL/archive data, or diagnostic metadata, even if Railway cannot perform an official restore.

5. Any recommended next steps for forensic review or partial reconstruction based on the incident details.

We understand this may still not be recoverable, but even raw metadata or confirmation of exactly what happened would help us rebuild reporting history and prevent this from happening again.

For context, this affected a production Postgres service, and we are trying to determine whether any historical operational data can be reconstructed or whether we should treat it as permanently unrecoverable.

Thank you again for checking.


Status changed to Awaiting Railway Response Railway 14 days ago


14 days ago

  1. None of those would be made available for user-initiated deletions.
  2. The databases are completely unmanaged, so that isn't available.
  3. Same as above.
  4. Same as above.
  5. Nothing on our end. This was something you initiated on your side.

Status changed to Awaiting User Response Railway 14 days ago


bushmasterhtps
PROOP

14 days ago

Understood.

So to confirm: Railway has no recoverable artifacts, no available diagnostic metadata, and no recommended recovery path for this incident.

We’ll proceed accordingly.


Status changed to Awaiting Railway Response Railway 14 days ago


Status changed to Solved bushmasterhtps 14 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...