a month ago
Environment: PostgreSQL 18.4 (Debian 18.4-1.pgdg13+1), Railway Postgres with PITR enabled, Pro plan. Two independent restores performed on 2026-07-08/09 UTC.
Summary
When a PITR restore is provisioned, the resulting service becomes reachable over its public TCP endpoint several minutes before WAL replay reaches the requested recovery target. During that window the database answers queries normally, returning a partial, monotonically growing subset of the data — with no error, no warning, and no indication in the Railway UI that recovery is still in progress. An operator validating the restore during this window will measure a row count that looks like catastrophic data loss.
In our case the restored copy reported its most recent row as ~15 hours older than the requested recovery point. We were minutes away from declaring a PITR failure and a 24h RPO. The restore was in fact correct; it simply had not finished replaying.
Minimal reproduction
Trigger a PITR restore of a Postgres service to a target timestamp T.
As soon as the restored service exposes a connection string, connect over DATABASE_PUBLIC_URL and run:
sqlselect pg_is_in_recovery(),
pg_last_xact_replay_timestamp(),
(select max(created_at) from <any_high_write_table>),
(select count(*) from <any_high_write_table>);Repeat every 15 seconds.
Observed — pg_is_in_recovery() returns t for several minutes. pg_last_xact_replay_timestamp() advances steadily. count(*) grows on every sample. Nothing in the dashboard distinguishes "restoring" from "ready".
Our measurements
Restore ARestore BRequested target (UTC)2026-07-08 23:062026-07-09 00:11 ("5 minutes ago" preset)Postmaster start (UTC)00:08:3400:18:23End-of-recovery checkpoint (UTC)00:21:1200:26:25Time serving partial data12m38s~8m02sEffective recovery point, from pg_wal/00000002.historybefore 2026-07-08 23:08:12.731362+00before 2026-07-09 00:12:21.0238+00Correct after promotion?Yes — exact row-count match vs source at targetYes — exact row-count match vs source at target
Both restores were data-correct. Row counts across users/chats/messages/customers/appointments matched the source database filtered at the effective recovery point, exactly, with zero discrepancies.
Impact
This is not a data-integrity bug — it is a disaster-recovery usability bug, and it fires precisely when the operator is under maximum pressure. The natural validation step after a restore (select count(*)) silently returns a wrong answer, and the natural conclusion is "PITR lost my data."
What we would like
Do not expose the public/private endpoint until recovery completes (i.e. until pg_is_in_recovery() returns f), or
Surface recovery state in the dashboard: a "Replaying WAL — replayed through " state distinct from "Ready", driven by pg_last_xact_replay_timestamp() vs the requested target, and
Display the effective recovery point of a completed restore (the value in the new timeline's .history file) next to the requested one, so an operator can confirm the restore landed where they asked.
Item 2 alone would have prevented this incident entirely.
Secondary question
Restore B targeted "5 minutes ago" yet still replayed ~19 hours of WAL (~8 minutes), implying the base snapshot is nightly and every restore replays from it regardless of target proximity. Can you confirm the base snapshot cadence, and whether restore time is expected to grow linearly with WAL accumulated since the last snapshot? This determines our RTO budget as write volume scales.
0 Threads mention this feature
0 Replies