8 days ago
Railway started the new container while the old one was still running, with the same volume mounted in both. Two live postmasters, one data directory.
03:19:24 — new container boots. Its wrapper checks for a running Postgres, sees none in its own container, concludes the pid file is stale, and logs removing stale postmaster.pid (no postgres running at container start). It deletes the file and starts postmaster pid 26, which writes a fresh postmaster.pid. The old postmaster pid 7 is very much alive on the other side of the same volume.
03:19:34 — the old postmaster runs its routine lock-file check, reads postmaster.pid, and finds pid 26 where it expects 7. It concludes its data directory has been hijacked and shuts down. On exit, Postgres unlinks its lock files unconditionally — no check that the contents still belong to it. So the dying old postmaster deleted the new postmaster's pid file.
03:20:24 — the new postmaster runs its own lock-file check. The file is gone: could not open file "postmaster.pid": No such file or directory. Shuts down. Nothing left running.
Both shutdowns are Postgres's own safety interlock working correctly — the one that exists precisely to stop two postmasters sharing a data directory. It fired twice, ten seconds apart, and took out both.
The 60-second delays that looked mysterious are just the schedule: the postmaster rechecks that lock file once a minute. Old container ready → 60 s → death. New container ready 03:19:24 → 60 s → death at 03:20:24.
Railway's wrapper is the accomplice. Its removing stale postmaster.pid step defeats the interlock. Without it, the new postmaster would have refused to start — "lock file already exists" — the old one would have kept serving, and you'd have had no outage at all.
This whole issue caused a 1h+ outage in my system that could be a lot more if I haven't checked it early and this will continue to happen everytime railway applies a patch to a postgresql database. I need this to either be fixed or that the patchings are triggered by me and only me.
2 Replies
Status changed to Awaiting Railway Response Railway • 8 days ago
8 days ago
Volume-backed services are never served by two containers at once. Deploys are serialized: the previous container is stopped and confirmed stopped before the new one starts, so two postmasters cannot share a data directory. The "removing stale postmaster.pid" log line is expected on every container start - it clears a pid file left behind by a previous container that was killed rather than shut down gracefully, not one belonging to a running process. Interleaved timestamps across the two deployments in the log stream are a property of how the log view merges output from consecutive containers, not evidence of concurrency. For controlling when image updates happen, you can configure auto-updates in the service's Settings tab under Source: you can turn them off entirely, limit them to a category, or confine them to a maintenance window on specific days and hours (UTC). Two things that setting does not cover: we still redeploy services for maintenance on the underlying host, and a severe security issue can still be patched on our initiative.
Status changed to Awaiting User Response Railway • 8 days ago
8 days ago
Hi, I'm really sorry about that, we have just fixed this. It won't happen again.
Status changed to Solved Railway • 8 days ago