PostgreSQL data recovery - production e-commerce data lost
khandryfruit
PROOP

4 months ago

Hello Railway Support,

I need urgent help recovering production PostgreSQL data.

Project: spirited-radiance (production)

Project ID: 8073fd28-63fb-461b-8582-b2d79fffc21b

Website: https://khanbabadryfruits.com

API: https://api.khanbabadryfruits.com

WHAT HAPPENED:

  • June 11, 2026 ~12:50 PM PKT: postgres-volume hit 100% (5 GB full)
  • Postgres crashed: "No space left on device", "recovery mode"
  • Website went down — products/categories HTTP 500

WHAT WE DID:

  • Upgraded to Pro Plan, resized volume
  • PITR restore created: Postgres-restored-20260611-0826
  • api-server connected — health check OK: {"ok":true,"status":"ready"}

CURRENT PROBLEM:

  • Database connects but ALL data is missing (0 products, 0 categories)
  • Website shows "0 products found"
  • pgBackRest backup (20260611-082553F) is only ~7 MB — taken AFTER crash, empty DB
  • Original production data was ~5 GB before crash

WHAT WE NEED:

Please recover PostgreSQL data from BEFORE June 11 disk-full crash.

Ideal recovery point: June 10, 2026, 11:59 PM PKT

WAL archiving was enabled on original Postgres (WAL_ARCHIVE_* variables set).

Please use WAL archive bucket for PITR if available.

IMPORTANT: Do NOT delete postgres-volume or either Postgres service until recovery confirmed.

This is a live e-commerce business. Please prioritize.

Thank you.

$20 Bounty

4 Replies

Railway
BOT

4 months ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway • 4 months ago


khandryfruit
PROOP

4 months ago

i need urgently help


Status changed to Open brody • 4 months ago


khandryfruit
PROOP

4 months ago

Hello Railway Support Team,

I urgently need your assistance regarding a serious data loss issue after restoring my project.

My project and PostgreSQL database were restored successfully, and I can see that all database tables still exist. However, the actual data inside those tables appears to be missing.

The affected data includes:

  • Products
  • Categories
  • Blog Posts
  • Banners
  • Store Content
  • Other production data

Before the incident, my store was fully operational and contained a large amount of manually uploaded content. I have spent significant time and effort building this data over many months.

I recently upgraded my Railway plan to the $20/month plan, and after the project restore process, the database structure remained intact, but the records themselves appear to be gone.

Could you please investigate the following:

  1. Whether the database was restored from the correct snapshot.
  2. Whether an older backup containing my production data is available.
  3. Whether the project was connected to a new/empty database during the restore process.
  4. Whether any automatic migration, reset, or restore operation may have caused the records to disappear.
  5. Whether Railway can recover the most recent backup containing my original production data.

This is a production e-commerce system, and the missing data includes products, categories, blog posts, and other business-critical content.

Please escalate this case and help me recover the latest available backup containing my original data.

I would greatly appreciate an urgent investigation, as this data represents a substantial amount of work and business value.

Thank you.


ayalaa12
FREE

2 months ago

The safest path is to treat the original volume and the PITR archive as evidence and restore only into a new sibling. Do not run migrations, initdb, pg_restore, or a manual pgBackRest restore against either existing service.

First, freeze the situation:

  1. Stop application writes or point the API at maintenance mode. Keep the original Postgres service, original postgres-volume, restored service, and PITR bucket. Do not wipe/delete a volume or disable PITR; Railway documents that wiping a volume deletes its backups, and disabling PITR on a single-node service can stage deletion of its bucket.
  2. In the original Postgres service's Backups tab, lock every surviving backup and inspect the PITR available restore range. The requested 2026-06-10 23:59 PKT is 2026-06-10 18:59:00 UTC; use the exact instant with an explicit timezone.
  3. If that instant is inside the displayed range, use Restore to this moment from the original service. Railway's PITR flow creates a brand-new Postgres sibling with a new empty volume, restores the latest base backup at/before the target, replays archived WAL to the target, and leaves the source untouched. If more than one WAL history is offered, choose the pre-crash cluster lifetime.

Before switching the API, verify the fork directly with a read-only client:

  • record current_database(), inet_server_addr() and SHOW data_directory so you know which service you queried;
  • list databases/schemas and compare row counts for products, categories, posts and banners;
  • check min/max business timestamps and a few known primary keys;
  • only after those checks, change the API's DATABASE_URL reference to the verified restored service and deploy that one staged change.

A 7 MB pgBackRest object is not by itself proof that the backup is empty: pgBackRest backups are compressed and can be incremental. The decisive facts are the PITR range, the selected archive history, the base-backup metadata, WAL continuity and verified row counts in the restored fork.

If the PITR picker does not include 18:59 UTC, or a restore from the original service/bucket produces only schema, do not keep retrying against the post-crash service. WAL segments cannot reconstruct a cluster without a compatible base backup. At that point Railway staff must preserve and inspect:

  • the detached original 5 GB volume and its mount association;
  • the original Postgres-PITR bucket and pgBackRest stanza/history;
  • every pre-crash base/incremental backup and the WAL continuity through the target;
  • which Postgres service the API reference actually points to.

Also check the original volume in the project canvas: Railway's native backup restore mounts a new dated volume and retains the previous volume unmounted. If the old ~5 GB volume still exists, do not attach it to two live Postgres instances; ask Railway to clone/recover it into an isolated service using the same Postgres major version and mount path. The fact that tables exist but rows do not can simply mean migrations ran against a newly initialized cluster.

Official recovery behavior and safeguards:


ayalaa12
FREE

2 months ago

Small correction to my preservation wording above: Railway's current backup docs do not expose a backup "lock" control. I meant preserve/no mutation. Before the intended PITR restore, do not delete or wipe the source, disable PITR, or run a native volume-backup Restore. Railway's native Backup flow removes newer backups after the selected point; by contrast, PITR "Restore to this moment" creates a new sibling service and leaves the source untouched. Capture the visible restore range and relevant timestamps first, then use only the sibling PITR procedure described above.

Docs: https://docs.railway.com/volumes/backups and https://docs.railway.com/volumes/point-in-time-recovery


Welcome!

Sign in to your Railway account to join the conversation.

Loading...