Production Postgres database tables wiped and overwritten – need audit log help
huypxbaolam
PROOP

24 days ago

Hi Railway Team,

We are facing a severe issue where our production PostgreSQL database was suddenly wiped and overwritten with tables from a completely different service (a chat/messenger service) in our project. We have checked our deployment logs but cannot find any manual restore or administrative action on the dashboard that triggered this.

Could you please help us check the internal audit logs on your end to see what event or who/which connection triggered this database reset/overwrite?

Project Technical Details:

  • Database Service: Postgres
  • Impacted Services: erp-backend, baolam-messenger-fe, baolam-erp-engine

Timeline of the Incident:

  • Time: May 25, 2026, around 07:02:32 AM (GMT+7) / 00:02:32 AM (UTC).
  • What happened:
    1. At 07:01:59 AM (GMT+7), our background cron job was running normally.
    2. At exactly 07:02:32 AM (GMT+7), the Postgres logs recorded dozens of connection resets: "could not receive data from client: Connection reset by peer".
    3. Right after this event, all our core ERP tables (such as daily_crew_reports, daily_tasks, grn_line_items, suppliers, warehouse_export_slips...) were completely gone.
    4. Instead, the database now only contains chat-related tables (conversations, messages, sessions...) belonging to the different service (baolam-messenger-fe).
    5. We checked the Project "Activity" feed but found no manual "Restore Backup" or administrative recreate logs around that timestamp.

Our request:

Could you please look into your backend control-plane logs for our Postgres instance around 00:00 UTC - 00:10 UTC on May 25, 2026, and help us identify:

  1. What triggered the massive connection disconnect at 00:02:32 UTC?
  2. Was there a cross-service migration/sync that went wrong and ran DDL statements (DROP/CREATE) on this database?
  3. If it was triggered by an external client/connection, could you provide the origin IP address or the actor?

Thank you so much!

Solved

1 Replies

Status changed to Awaiting Railway Response Railway 24 days ago


23 days ago

Our databases are completely unmanaged at the storage layer, so internal snapshots, WAL recovery, or PITR aren't available. We don't help with data recovery for user-initiated actions either way.


Status changed to Awaiting User Response Railway 23 days ago


Status changed to Solved mykal 23 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...