16 days ago
Project: sublime-gratitude
Environment: production
Service: Postgres
Database: railway
Host: interchange.proxy.rlwy.net:43835
Our production app is down. The database previously contained all production application tables such as User, Task, RawMessage, etc. It now contains only _prisma_migrations.
Incident timeline:
- App was working earlier today.
- A Prisma command was accidentally run with the production DATABASE_URL as the shadow DB URL:
npx prisma migrate diff --from-migrations prisma/migrations --to-schema-datamodel prisma/schema.prisma --shadow-database-url "$DATABASE_URL"
- Afterward, the DB no longer showed application tables.
- A later npx prisma migrate deploy failed on:
20241229180000_add_task_performance_index
ERROR: relation "public.Task" does not exist
- Current _prisma_migrations only shows the failed migration row from 2026-05-07 02:12 UTC.
- Railway Backups UI shows no backups.
We urgently need to know if Railway can recover this Postgres volume/database to a point before 2026-05-07 02:12 UTC, using any platform-level snapshot, volume recovery, WAL/PITR, or internal recovery option. This is a production outage affecting tomorrow’s operations.
7 Replies
Status changed to Awaiting Railway Response Railway • 16 days ago
15 days ago
Hi Railway Support or anybody,
We urgently need help determining whether recovery is possible for our Postgres database.
A Prisma command was run where the production database URL may have been used as the shadow database URL during a migration diff/schema troubleshooting step. After that, our production database appears to have lost historical application data, including task/productivity records going back to last year.
Database details:
- Host: interchange.proxy.rlwy.net
- Database: railway
- Service: Railway Postgres
- Approximate incident window: May 6–7, 2026, around the evening/night PT
We understand scheduled backups may not have been enabled. Can you please confirm whether there are any platform-level snapshots, volume backups, WAL/archive recovery, internal disaster recovery options, or any other way to restore or clone the database to a point before the incident?
This database contained operational productivity/task history used for department reporting, so even a partial restore or clone would be extremely valuable. We do not want to overwrite the current database if recovery is possible; ideally we would like a restored copy/clone to inspect first.
Please help ☹
15 days ago
URGENT UPDATE — DATABASE STATE CONFIRMED
We connected directly to the active Railway Postgres instance via psql and confirmed the following:
- Current application tables exist because we rebuilt the schema after the incident to restore operations.
- However, the active data only begins on 2026-05-07.
- Historical production records from the prior ~9 months are no longer present in the active database.
Examples confirmed via SQL:
- Task oldest createdAt: 2026-05-07 07:40 UTC
- RawMessage oldest createdAt: 2026-05-07 07:40 UTC
- User oldest createdAt: 2026-05-07 03:37 UTC
We also confirmed:
- Postgres itself is healthy/responding
- The current database is operational
- No alternate schemas containing the old tables were found
- Current data appears to be from the rebuilt environment after the incident
We are urgently requesting confirmation whether Railway has ANY infrastructure-level recovery options available, including:
- WAL recovery
- PITR (point-in-time recovery)
- storage snapshots
- volume backups
- internal disaster recovery
- cloned restore from a previous state
IMPORTANT:
Please do NOT overwrite the current database.
If recovery is possible, we would strongly prefer:
- a cloned/restored Postgres instance
- restored from ANY point prior to approximately:
May 6, 2026 around 4:00 PM PT
An exact timestamp is not necessary. Even a partial or slightly older restore would be extremely valuable.
Project details:
- Project: sublime-gratitude
- Environment: production
- Database: railway
- Host: interchange.proxy.rlwy.net
This database contained operational productivity/task history used for reporting and department operations. Recovering the historical data would be critically important.
Thank you.
15 days ago
Additional note:
I attempted to join the Railway Discord to escalate this production recovery request, but the Railway ↔ Discord OAuth verification flow is currently failing with:
“Problem completing OAuth login”
So I am unable to post in the support/help channels there at the moment.
Please let me know if there is another direct escalation/support route available for urgent production Postgres recovery requests.
15 days ago
We don't offer data restoration for user-initiated actions like this. Going forward, enable volume backups on your Postgres service so you can self-restore if it happens again.
Status changed to Awaiting User Response Railway • 15 days ago
14 days ago
Hi Railway Support,
Thank you for clarifying that Railway does not offer data restoration for user-initiated actions and that we should enable volume backups going forward.
Before we fully close this out, I wanted to ask one final recovery-related question:
Even if a managed restore is not available, is there anything Railway can provide that may help us reconstruct or audit the lost Postgres data?
Specifically, are any of the following available on your side?
1. Any internal database volume snapshots, temporary snapshots, filesystem-level remnants, or retained volume artifacts from before the incident.
2. Any deployment/database event logs showing when the destructive action occurred and what database/volume object was affected.
3. Any connection/query/audit metadata that could confirm whether the data was dropped, truncated, overwritten, or affected by a migration/shadow DB operation.
4. Any way to export raw remnants, partial volume contents, WAL/archive data, or diagnostic metadata, even if Railway cannot perform an official restore.
5. Any recommended next steps for forensic review or partial reconstruction based on the incident details.
We understand this may still not be recoverable, but even raw metadata or confirmation of exactly what happened would help us rebuild reporting history and prevent this from happening again.
For context, this affected a production Postgres service, and we are trying to determine whether any historical operational data can be reconstructed or whether we should treat it as permanently unrecoverable.
Thank you again for checking.
Status changed to Awaiting Railway Response Railway • 14 days ago
14 days ago
- None of those would be made available for user-initiated deletions.
- The databases are completely unmanaged, so that isn't available.
- Same as above.
- Same as above.
- Nothing on our end. This was something you initiated on your side.
Status changed to Awaiting User Response Railway • 14 days ago
14 days ago
Understood.
So to confirm: Railway has no recoverable artifacts, no available diagnostic metadata, and no recommended recovery path for this incident.
We’ll proceed accordingly.
Status changed to Awaiting Railway Response Railway • 14 days ago
Status changed to Solved bushmasterhtps • 14 days ago