Production down 14h — Postgres service running app code instead of Postgres
Anonymous
FREEOP

18 days ago

Project: wholesome-mercy

Environment: production

Affected service: Postgres

Region: US West

WHAT IS HAPPENING

Our Postgres service currently shows TWO deployments marked ACTIVE at the

same time:

ACTIVE "[our application commit message]" via GitHub

ACTIVE ghcr.io/railwayapp-templates/postgres-ssl:18 via Docker

The GitHub one is a commit of our APPLICATION repository. We never

intentionally connected that repository to this database service, and its

Settings > Source page shows NO repository connected. Yet every push to

our app repo deploys onto this Postgres service and replaces the database.

The Postgres service logs confirm it: instead of Postgres starting, we see

our application's startup script running and aborting, followed by

"Stopping Container".

So the database service is running our app code, not PostgreSQL. That is

why it is unreachable from every route.

CONSEQUENCE

  • Private network (postgres.railway.internal:5432): timeout expired

  • Public TCP proxy: ECONNRESET on read, with the original proxy

    (tokaido.proxy.rlwy.net:51302) and with a regenerated one

    (hayabusa.proxy.rlwy.net:27291)

  • Tested from inside Railway and from an external machine: identical

TIMELINE

  • Until 13:33 UTC on 2026-08-02: Postgres 18.4 healthy, normal checkpoints

  • 13:52 UTC: all client connections dropped simultaneously

    ("could not receive data from client: Connection reset by peer"),

    followed by "Stopping Container"

  • Since then: repeated "Mounting volume on: /var/lib/containers/

    railwayapp/bind-mounts//vol_kep6ixr24ymeklpl" with a

    different UUID each restart

  • The service has never accepted a connection again

WHAT WE TRIED

  • Restarted the Postgres service

  • Deleted and recreated the public TCP proxy (new host and port)

  • Redeployed the app service many times

  • Verified the Postgres service has no GitHub source connected

  • Confirmed SSL, port, IPv6 resolution and credentials are all correct on

    our side. The failures happen at TCP level, before authentication.

WHAT WE NEED

  1. Please make ghcr.io/railwayapp-templates/postgres-ssl:18 the running

    deployment again on this service, and remove the GitHub deployment

    that should never have been there.

  2. Please confirm why our application repository is deploying onto this

    database service when no source is connected to it, and stop it from

    happening again.

  3. Can you confirm there is no data loss? Volume vol_kep6ixr24ymeklpl

    holds our production data.

  4. If recovery stalls, what is the fastest safe path to restore from our

    own encrypted backup onto a healthy volume? We would rather restore

    now than keep waiting.

This is a production system and it has been down for over 14 hours. We

have verified encrypted backups, so data loss is not our main concern —

we need the fastest safe path back to production.

$10 Bounty

1 Replies

Status changed to Awaiting Railway Response Railway 18 days ago


Railway
BOT

18 days ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 18 days ago


Unfortunately, your only option would be to delete your current Postgres service (not the volume), create a new Postgres service, and swap volumes by dragging the old volume onto the new one.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...