18 days ago
Project: wholesome-mercy
Environment: production
Affected service: Postgres
Region: US West
WHAT IS HAPPENING
Our Postgres service currently shows TWO deployments marked ACTIVE at the
same time:
ACTIVE "[our application commit message]" via GitHub
ACTIVE ghcr.io/railwayapp-templates/postgres-ssl:18 via Docker
The GitHub one is a commit of our APPLICATION repository. We never
intentionally connected that repository to this database service, and its
Settings > Source page shows NO repository connected. Yet every push to
our app repo deploys onto this Postgres service and replaces the database.
The Postgres service logs confirm it: instead of Postgres starting, we see
our application's startup script running and aborting, followed by
"Stopping Container".
So the database service is running our app code, not PostgreSQL. That is
why it is unreachable from every route.
CONSEQUENCE
-
Private network (postgres.railway.internal:5432): timeout expired
-
Public TCP proxy: ECONNRESET on read, with the original proxy
(tokaido.proxy.rlwy.net:51302) and with a regenerated one
(hayabusa.proxy.rlwy.net:27291)
-
Tested from inside Railway and from an external machine: identical
TIMELINE
-
Until 13:33 UTC on 2026-08-02: Postgres 18.4 healthy, normal checkpoints
-
13:52 UTC: all client connections dropped simultaneously
("could not receive data from client: Connection reset by peer"),
followed by "Stopping Container"
-
Since then: repeated "Mounting volume on: /var/lib/containers/
railwayapp/bind-mounts//vol_kep6ixr24ymeklpl" with a
different UUID each restart
-
The service has never accepted a connection again
WHAT WE TRIED
-
Restarted the Postgres service
-
Deleted and recreated the public TCP proxy (new host and port)
-
Redeployed the app service many times
-
Verified the Postgres service has no GitHub source connected
-
Confirmed SSL, port, IPv6 resolution and credentials are all correct on
our side. The failures happen at TCP level, before authentication.
WHAT WE NEED
-
Please make ghcr.io/railwayapp-templates/postgres-ssl:18 the running
deployment again on this service, and remove the GitHub deployment
that should never have been there.
-
Please confirm why our application repository is deploying onto this
database service when no source is connected to it, and stop it from
happening again.
-
Can you confirm there is no data loss? Volume vol_kep6ixr24ymeklpl
holds our production data.
-
If recovery stalls, what is the fastest safe path to restore from our
own encrypted backup onto a healthy volume? We would rather restore
now than keep waiting.
This is a production system and it has been down for over 14 hours. We
have verified encrypted backups, so data loss is not our main concern —
we need the fastest safe path back to production.
1 Replies
Status changed to Awaiting Railway Response Railway • 18 days ago
18 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 18 days ago
18 days ago
Unfortunately, your only option would be to delete your current Postgres service (not the volume), create a new Postgres service, and swap volumes by dragging the old volume onto the new one.