Postgres volume reports ~843MB but mounted cluster is empty after major upgrade — need block-level restore
eduardojonas244-star
PROOP

6 days ago

Hi Railway team — this is urgent. We need help recovering Postgres data from a volume that still reports ~843MB in the UI, but the live mounted filesystem only has ~47MB and an empty application database after a PostgreSQL major upgrade.

Context

On or around 2026-08-15 ~14:04 UTC, during a project rebuild / Postgres template major upgrade, our historical Postgres volume appears to have been remounted with a fresh PG18 cluster. The volume size in the dashboard still shows ~843MB, but inside the container there is no recoverable old data directory (no data-old / backup cluster), only a new empty PG 18.x datadir.

We have NOT deleted this volume. We are keeping Postgres-Recovery online as-is and have not run destructive volume deletes. We need a platform-side block/snapshot restore of the volume contents from BEFORE the major upgrade.

Project / IDs

  • Workspace: eduardojonas244-star's Projects
  • Account email: eduardojonas244@gmail.com
  • Project name: positive-reverence
  • Project ID: 97e4e572-fa41-4298-a858-7a1da1da1c60
  • Environment: production
  • Environment ID: 9297cdd4-941b-441c-90ad-f7e084e25003

Volume with missing data (please restore this one)

  • Volume name: postgres-volume
  • Volume ID: 3ff7a25a-53a8-41e2-9f37-ce21b7b3c766
  • Volume instance ID: 972e06f2-b660-40b8-a396-a32a3c6f874f
  • UI reported size: ~843 MB
  • Currently attached to service: Postgres-Recovery
  • Postgres-Recovery service ID: b6008d65-9f54-4a25-8ea5-7e552cb8a0ee
  • Mount path: /var/lib/postgresql/data

What we observe on the mounted volume

  • Live mount usage: ~47 MB (df / du)
  • Only current cluster under something like .../data/pgdata
  • PG_VERSION: 18
  • Presence of .railway-major-upgrade.lock (around 2026-08-15 14:04 UTC)
  • Live database has 0 application tables in public schema
  • No data-old / data.bak / second PG_VERSION / leftover cluster found
  • volumeInstanceBackupList / backup schedules: empty (no user-configured snapshots)

Current (new) DB — do not wipe

We also have a separate new Postgres used by the app:

  • Service: Postgres
  • Service ID: 4f13ea15-e6a2-4ae7-982d-90a1ca70df15
  • Volume: postgres-volume-waU4
  • Volume ID: a779e655-2c1f-46bc-b69a-015e44645e6c

This has only seed/new schema data. Please do NOT restore over / delete this unless necessary — we need the OLD data from postgres-volume (3ff7a25a-...).

Related deleted project

A previous project named desirable-emotion was deleted during the incident rebuild. If any soft-deleted volumes/backups from that project still exist in the retention window, please check those too.

Ask

  1. Can you restore postgres-volume (3ff7a25a-53a8-41e2-9f37-ce21b7b3c766) from an internal block snapshot / pre-upgrade state (before 2026-08-15 ~14:04 UTC)?
  2. If yes, please restore it as a separate volume or rehydrate readable PG data without destroying the current empty cluster attachment until we can verify.
  3. If no snapshot exists, please confirm explicitly so we can stop waiting.

This volume held production-sandbox player wallets / ledger data. Time-sensitive.

Thank you.

Solved

4 Replies

Railway
BOT

6 days ago

The volume is attached and reporting usage, but our databases are unmanaged, so automatic snapshots, WAL recovery, and PITR are not available to restore your data unless you had previously enabled them on the service. We do not recover data lost to user-initiated actions, including major version upgrades. Going forward, enable native volume backups and point-in-time recovery on your Postgres services from the service's Backups tab so you can self-restore if this happens again.


Status changed to Awaiting User Response Railway 6 days ago


Railway

The volume is attached and reporting usage, but our databases are unmanaged, so automatic snapshots, WAL recovery, and PITR are not available to restore your data unless you had previously enabled them on the service. We do not recover data lost to user-initiated actions, including major version upgrades. Going forward, enable native volume backups and point-in-time recovery on your Postgres services from the service's Backups tab so you can self-restore if this happens again.

eduardojonas244-star
PROOP

6 days ago

Thanks for clarifying that unmanaged Postgres has no automatic snapshots / WAL / PITR unless previously enabled, and that major-version upgrades are treated as user-initiated.

We need to correct the root-cause framing:

This was NOT a user-initiated delete of the original production sandbox

The original project was removed/destroyed when our Railway trial ended and payment method / account balance was not funded in time. Railway automatically deleted or deprovisioned the project because the trial expired — we did not intentionally delete the live sandbox that ran our casino operation (player accounts, wallets, ledger, custom domains).

That automatic trial/billing teardown effectively wiped our operation. Afterward we tried to rebuild in a new project (positive-reverence) and attempted recovery from a remaining volume; a Postgres major upgrade then left that volume reporting ~843MB in the UI while the mounted cluster is empty (~47MB, 0 app tables). We understand you may not undo major upgrades via PITR — but restoring data lost to trial expiry / automatic account teardown is a different request.

Ask — restore from Railway trial expiry / soft-delete retention

  1. Please search this workspace for projects/volumes soft-deleted or deprovisioned due to trial expiry / unpaid account around early/mid August 2026 (especially ~2026-08-15), including any original project named desirable-emotion that hosted:
  • Services: casinha-sandbox, casinha-front-sandbox, casinha-adm-sandbox, Postgres
  • Domains: api.ricaobet.com, www.ricaobet.com, admin.ricaobet.com / ricaobet.com
  • Workspace ID: 366b4ee7-0450-45a3-b58d-382ba2701da8
  • Account email: eduardojonas244@gmail.com
  1. Soft-deleted project still visible:
  • Project name: desirable-emotion
  • Project ID: 802ce642-40bb-4e20-a4d8-4f01b9f772ab
  • deletedAt: 2026-08-17T14:26:39.156Z
  • Environment ID: f95a4d39-d1b0-47c3-9404-574325251445

Please restore this if it still contains the original services/volumes. If it was only an empty recreate, please still locate the original trial-expired project/volumes.

  1. Please resend any “Restore Volume” / “Restore Project” emails to eduardojonas244@gmail.com and confirm whether the 48h (or any extended) retention window still applies for trial-expiry deletions.

  2. Volume still attached in the rebuild project (possible remnant of the old DB):

  • Volume ID: 3ff7a25a-53a8-41e2-9f37-ce21b7b3c766 (postgres-volume)
  • UI used size ~843MB vs mounted empty PG18 ~47MB

If Railway retains a pre-teardown or pre-upgrade block image of this volume, please expose it as a restorable volume.

This held live player balances and accounts. The outage was caused by trial expiry automation, not an intentional wipe by us. We are asking for platform recovery under soft-delete / trial-deletion retention.

If recovery is impossible, please say so explicitly and confirm the exact retention policy for trial-expired projects/volumes so we can close this path.

Thank you.


Status changed to Awaiting Railway Response Railway 6 days ago


6 days ago

To address each point directly: this workspace was created and upgraded to Pro on 2026-08-15, with no prior trial period on this account, so there was no trial-expiry teardown to recover from. The project desirable-emotion was deleted from the dashboard on 2026-08-17 and contained no services or volumes at that time; its 48-hour restore window has since closed. Dashboard-deleted projects carry a 48-hour window to undo the deletion, after which the resources are permanently removed. The volume postgres-volume is still attached and reporting usage, but the in-place data loss from the PostgreSQL 18 major upgrade is not something we can restore. We do not recover data lost to user-initiated actions, and no volume backups or point-in-time recovery had been enabled on this service beforehand. Going forward, enable native volume backups and PITR on your Postgres services from the service's Backups tab so you can self-restore if this happens again.


Status changed to Awaiting User Response brody 6 days ago


dizzydes90
EMPLOYEE

3 days ago

Following up with a definitive answer, since you asked for one. We went through the backend records directly. Railway's automated major-version upgrade never ran on this service or anywhere in this workspace, and the .railway-major-upgrade.lock file is a marker the Postgres image writes on every boot, not evidence of an upgrade. The PG18 cluster on postgres-volume was initialized fresh on Aug 15 when the Postgres-Recovery service was created and the volume attached, and all of the activity in that window came from your own account's sessions.

On recovery: no snapshot of this volume exists on our side. Backups and point-in-time recovery were never enabled on it, and we don't keep block-level copies otherwise. The 843MB in the dashboard versus 47MB inside the container is storage accounting for data that's already been deleted at the file level, not a hidden recoverable copy. As my colleague said, this workspace was created on Aug 15 with no prior trial, desirable-emotion held no services or volumes when it was deleted, and its 48-hour window has closed. So, explicitly: there's nothing on our side to restore, and you can stop waiting on this path. Any recovery would have to come from copies outside Railway.


Status changed to Solved dizzydes90 3 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...