Skip to content
production data deleted on two Postgres services
Anonymous
OP

25 days ago

Project ID: c2ee1419-47e5-4bfb-bee3-78327f51e0e4 · Environment: production · Region: US East

Services: Postgres-aXUE (volume postgres-volume-8J-j) and Postgres (volume postgres-volume)

On Oct 25, 2026, all data in both databases was deleted. The public schema was dropped and recreated over a direct connection; the servers did not restart. Metrics show the drop around 06:00 UTC (Postgres-aXUE) and 15:20 UTC (Postgres). Our only volume backups are from Aug 27, and PITR was off.

We know you don't normally restore data. We're asking for a human to please check:

Is there any internal volume snapshot of either service from Oct 24–25 (before those times) that can be restored or exported?

Are there TCP proxy connection logs (client IPs and times) for both services on Oct 24 22:00 – Oct 25 16:30 UTC?

Please preserve anything you have for these volumes while this is looked at.

Our students can't sign in, so any help is hugely appreciated.

Solved

3 Replies

Status changed to Awaiting Railway Response Railway • 25 days ago


dizzydes90
EMPLOYEE

25 days ago

Here's what exists. Postgres-aXUE (postgres-volume-8J-j) has two manual backups: Aug 27 20:30 UTC, and one taken from your workspace on Oct 26 04:07 UTC, after the deletion, so it holds the empty state. Postgres (postgres-volume) has one manual backup, Aug 27 20:31 UTC. PITR was never enabled on either, and no other snapshots of these volumes exist. We don't restore data on a customer's behalf. Restoring a backup mounts a new volume at the same path, leaves the old volume in the project unmounted, and keeps newer backups on it, so restoring loses nothing: https://docs.railway.com/volumes/backups

We don't have customer-facing TCP proxy logs, and the proxy doesn't keep client IPs in our network records, so we can't say who connected. Every connection in that window came through the public TCP proxy. Besides your app's long-lived connections, each database got one short connection at 15:22:50 and 15:22:51 UTC on Oct 25, and another at 15:23:38 and 15:23:39. Each server's logs show the first "relation does not exist" errors after that: 15:22:53 UTC on Postgres-aXUE and 15:39:26 UTC on Postgres, with no earlier errors on either.

The database password was stored as a Railway variable, so treat it as exposed: regenerate it under Database, Config, then redeploy the services that use it. If nothing outside Railway needs these databases, remove the TCP proxy under Settings, Networking. Don't wipe the volumes; that deletes the backups too.


Status changed to Awaiting User Response Railway • 25 days ago


dizzydes90

Here's what exists. Postgres-aXUE (postgres-volume-8J-j) has two manual backups: Aug 27 20:30 UTC, and one taken from your workspace on Oct 26 04:07 UTC, after the deletion, so it holds the empty state. Postgres (postgres-volume) has one manual backup, Aug 27 20:31 UTC. PITR was never enabled on either, and no other snapshots of these volumes exist. We don't restore data on a customer's behalf. Restoring a backup mounts a new volume at the same path, leaves the old volume in the project unmounted, and keeps newer backups on it, so restoring loses nothing: https://docs.railway.com/volumes/backups We don't have customer-facing TCP proxy logs, and the proxy doesn't keep client IPs in our network records, so we can't say who connected. Every connection in that window came through the public TCP proxy. Besides your app's long-lived connections, each database got one short connection at 15:22:50 and 15:22:51 UTC on Oct 25, and another at 15:23:38 and 15:23:39. Each server's logs show the first "relation does not exist" errors after that: 15:22:53 UTC on Postgres-aXUE and 15:39:26 UTC on Postgres, with no earlier errors on either. The database password was stored as a Railway variable, so treat it as exposed: regenerate it under Database, Config, then redeploy the services that use it. If nothing outside Railway needs these databases, remove the TCP proxy under Settings, Networking. Don't wipe the volumes; that deletes the backups too.

Anonymous
OP

25 days ago

Thank you, this is very helpful. A few follow-ups:

  • When we restore the Aug 27 backups on Postgres-aXUE and Postgres, will the connection details (host, port, password, TCP proxy) stay the same, so our API reconnects without changes? Or should we update variables after the restore?
  • For the short connections at 15:22:50–51 and 15:23:38–39 UTC, do your records show the bytes transferred? We'd like to know whether data was downloaded before the drop.
  • Do your records show similar short connections to these databases in the 7 days before Oct 25? That would tell us whether this was a recurring script.
  • Please keep the current (post-wipe) volumes after we restore, in case we need them.

Status changed to Awaiting Railway Response Railway • 25 days ago


dizzydes90
EMPLOYEE

24 days ago

Connection details don't change. Host, port, password and the TCP proxy belong to the service, not the volume; the restore mounts a new volume at the same path and redeploys the service, so your API reconnects as is. One catch: if you regenerate the password before restoring, the August 27 data still holds the old one, so run Regenerate again after the restore; it resets the role inside the database to match the variable.

Bytes sent by the database on the short connections: Postgres-aXUE about 6 KB at 15:22:50 and 4 KB at 15:23:38; Postgres about 4.6 KB at 15:22:51 and 4 KB at 15:23:39. Each lasted about a second, a login and a few statements, not a copy of the data.

Earlier in the week, the same one-second shape hit Postgres-aXUE on Oct 21 at 06:19 and 08:47, six times between 02:30 and 02:32 on Oct 22, and at 04:40 on Oct 22 UTC, and Postgres once on Oct 21 at 06:19. Postgres also gets a 2 to 4 second connection every hour all week, which looks scheduled. Separately, Postgres-aXUE sent roughly 1 GB over five connections on Oct 20 between 15:21 and 16:27 UTC and about 470 MB over two on Oct 22 between 18:56 and 20:10 UTC; if those weren't your own exports, treat the data as copied. We can't see client IPs behind any of them.

The current volumes stay in the project after a restore, unmounted, until you delete them.


Status changed to Awaiting User Response Railway • 24 days ago


Status changed to Solved Anonymous • 23 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...