4 months ago
Hello Railway Support,
I need urgent help recovering production PostgreSQL data.
Project: spirited-radiance (production)
Project ID: 8073fd28-63fb-461b-8582-b2d79fffc21b
Website: https://khanbabadryfruits.com
API: https://api.khanbabadryfruits.com
WHAT HAPPENED:
- June 11, 2026 ~12:50 PM PKT: postgres-volume hit 100% (5 GB full)
- Postgres crashed: "No space left on device", "recovery mode"
- Website went down — products/categories HTTP 500
WHAT WE DID:
- Upgraded to Pro Plan, resized volume
- PITR restore created: Postgres-restored-20260611-0826
- api-server connected — health check OK: {"ok":true,"status":"ready"}
CURRENT PROBLEM:
- Database connects but ALL data is missing (0 products, 0 categories)
- Website shows "0 products found"
- pgBackRest backup (20260611-082553F) is only ~7 MB — taken AFTER crash, empty DB
- Original production data was ~5 GB before crash
WHAT WE NEED:
Please recover PostgreSQL data from BEFORE June 11 disk-full crash.
Ideal recovery point: June 10, 2026, 11:59 PM PKT
WAL archiving was enabled on original Postgres (WAL_ARCHIVE_* variables set).
Please use WAL archive bucket for PITR if available.
IMPORTANT: Do NOT delete postgres-volume or either Postgres service until recovery confirmed.
This is a live e-commerce business. Please prioritize.
Thank you.
4 Replies
4 months ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 4 months ago
4 months ago
i need urgently help
Status changed to Open brody • 4 months ago
4 months ago
Hello Railway Support Team,
I urgently need your assistance regarding a serious data loss issue after restoring my project.
My project and PostgreSQL database were restored successfully, and I can see that all database tables still exist. However, the actual data inside those tables appears to be missing.
The affected data includes:
- Products
- Categories
- Blog Posts
- Banners
- Store Content
- Other production data
Before the incident, my store was fully operational and contained a large amount of manually uploaded content. I have spent significant time and effort building this data over many months.
I recently upgraded my Railway plan to the $20/month plan, and after the project restore process, the database structure remained intact, but the records themselves appear to be gone.
Could you please investigate the following:
- Whether the database was restored from the correct snapshot.
- Whether an older backup containing my production data is available.
- Whether the project was connected to a new/empty database during the restore process.
- Whether any automatic migration, reset, or restore operation may have caused the records to disappear.
- Whether Railway can recover the most recent backup containing my original production data.
This is a production e-commerce system, and the missing data includes products, categories, blog posts, and other business-critical content.
Please escalate this case and help me recover the latest available backup containing my original data.
I would greatly appreciate an urgent investigation, as this data represents a substantial amount of work and business value.
Thank you.
2 months ago
The safest path is to treat the original volume and the PITR archive as evidence and restore only into a new sibling. Do not run migrations, initdb, pg_restore, or a manual pgBackRest restore against either existing service.
First, freeze the situation:
- Stop application writes or point the API at maintenance mode. Keep the original Postgres service, original
postgres-volume, restored service, and PITR bucket. Do not wipe/delete a volume or disable PITR; Railway documents that wiping a volume deletes its backups, and disabling PITR on a single-node service can stage deletion of its bucket. - In the original Postgres service's Backups tab, lock every surviving backup and inspect the PITR available restore range. The requested
2026-06-10 23:59 PKTis2026-06-10 18:59:00 UTC; use the exact instant with an explicit timezone. - If that instant is inside the displayed range, use Restore to this moment from the original service. Railway's PITR flow creates a brand-new Postgres sibling with a new empty volume, restores the latest base backup at/before the target, replays archived WAL to the target, and leaves the source untouched. If more than one WAL history is offered, choose the pre-crash cluster lifetime.
Before switching the API, verify the fork directly with a read-only client:
- record
current_database(),inet_server_addr()andSHOW data_directoryso you know which service you queried; - list databases/schemas and compare row counts for products, categories, posts and banners;
- check min/max business timestamps and a few known primary keys;
- only after those checks, change the API's
DATABASE_URLreference to the verified restored service and deploy that one staged change.
A 7 MB pgBackRest object is not by itself proof that the backup is empty: pgBackRest backups are compressed and can be incremental. The decisive facts are the PITR range, the selected archive history, the base-backup metadata, WAL continuity and verified row counts in the restored fork.
If the PITR picker does not include 18:59 UTC, or a restore from the original service/bucket produces only schema, do not keep retrying against the post-crash service. WAL segments cannot reconstruct a cluster without a compatible base backup. At that point Railway staff must preserve and inspect:
- the detached original 5 GB volume and its mount association;
- the original
Postgres-PITRbucket and pgBackRest stanza/history; - every pre-crash base/incremental backup and the WAL continuity through the target;
- which Postgres service the API reference actually points to.
Also check the original volume in the project canvas: Railway's native backup restore mounts a new dated volume and retains the previous volume unmounted. If the old ~5 GB volume still exists, do not attach it to two live Postgres instances; ask Railway to clone/recover it into an isolated service using the same Postgres major version and mount path. The fact that tables exist but rows do not can simply mean migrations ran against a newly initialized cluster.
Official recovery behavior and safeguards:
2 months ago
Small correction to my preservation wording above: Railway's current backup docs do not expose a backup "lock" control. I meant preserve/no mutation. Before the intended PITR restore, do not delete or wipe the source, disable PITR, or run a native volume-backup Restore. Railway's native Backup flow removes newer backups after the selected point; by contrast, PITR "Restore to this moment" creates a new sibling service and leaves the source untouched. Capture the visible restore range and relevant timestamps first, then use only the sibling PITR procedure described above.
Docs: https://docs.railway.com/volumes/backups and https://docs.railway.com/volumes/point-in-time-recovery