Urgent: Unacceptable Data Loss on Production PostgreSQL – Immediate Action Required
jsj3473
PROOP

7 months ago

Hello Railway Support Team,

I’m reaching out again with serious concern regarding the PostgreSQL managed service on Railway. We have experienced critical and unacceptable data loss on our production database — and despite prior communication, we have not received any meaningful resolution or accountability from your side.

mag emojiProblem Summary

  • Database: PostgreSQL (Railway managed)

  • Issue: All data from 2025-04-25 to 2025-06-24 is completely missing.

  • Suspected Cause: PostgreSQL WAL (Write-Ahead Log) corruption. The system attempted recovery but instead rolled back to a checkpoint from April 24.

  • Key log output:

    LOG: database system was interrupted; last known up at 2025-06-24 14:58:00 UTC

  • LOG: invalid record length at 0/303B580: expected at least 24, got 0

  • LOG: redo done at 0/303B548

Due to what appears to be a severe failure in your infrastructure or backup retention policies, nearly two full months of production data have been irreversibly lost. This is not a minor issue — it has directly affected our business operations, and no valid recovery path has been provided.

The so-called data-recovery volume contains only data up to April 24 — it does not include the data we desperately need. We explicitly communicated that our required window is from April 25 to June 24, yet no further action has been taken.

We Need an Answer

We urgently demand the following:

  1. Immediate clarification on whether any WAL archives or snapshots exist between April 25 and June 24.

  2. A clear explanation of how such an extensive rollback occurred without warning.

  3. A formal response regarding compensation or remediation for this severe data loss.

  4. Transparent disclosure of Railway’s backup, retention, and recovery policies — as it currently stands, they are not reliable for production use.

This is a production-grade managed service. Losing weeks of data — and getting no concrete response beyond vague apologies — is not acceptable.

We expect a proper resolution from your team immediately.

Sincerely,
jsj3473

Closed

0 Replies

Status changed to Closed itsrems 7 months ago


Loading...