19 days ago
My Postgres database service has been crash-looping for the past ~40 minutes, restarting every 9-11 minutes (Restart Policy: On Failure, max retries: 10 — currently on attempt 4/10).
What I've ruled out:
- Resource limits: Plan limit is 8 vCPU / 8GB memory, actual usage is ~100MB memory and ~0% CPU throughout — nowhere near OOM territory.
- Health checks: No custom health check configured, restart policy is "On Failure" only.
- External triggers: Service is deployed via a fixed Docker image (ghcr.io/railwayapp-templates/postgres-ssl:18), not connected to a GitHub repo, no auto-deploy source that could be triggering redeploys.
- Password/config issues: Verified credentials are correct and match DATABASE_PUBLIC_URL.
Every restart shows the same pattern in deploy logs:
database system was interrupted; last known up at [timestamp]
database system was not properly shut down; automatic recovery in progress
This indicates the container process is being killed abruptly (not a clean shutdown) despite low resource usage. Between crashes I also see repeated "password authentication failed" and "invalid length of startup packet" entries on the public proxy port, which I believe is unrelated scanning/bot traffic, not the cause.
Service ID: c92536cf
Region: US West (California, USA)
Proxy domain: thomas.proxy.rlwy.net
I'm concerned about running out of restart retries (currently 4/10) and potential downtime/data loss. Could you please check the host-level logs for this instance? Full deploy logs available on request.
5 Replies
19 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 19 days ago
0x5b62656e5d
Does a redeploy fix it?
19 days ago
I already tried a redeploy earlier — it didn't fix it. The new deployment crashed again a few minutes later with the same "database system was interrupted / not properly shut down" error. I'm currently on attempt 4 of my 10 restart retries, and resource usage (CPU/memory) stays well below plan limits throughout, so it doesn't look resource-related.
19 days ago
Critical finding: Even running psql directly from inside the Postgres container via Railway's own Console (which should auto-use the container's env PGPASSWORD) fails with "FATAL: password authentication failed for user postgres" when connecting to postgres.railway.internal. This rules out any client-side, app-side, or network config issue — the authentication is broken at the instance level itself. This matches the symptoms in "Production incident: PostgreSQL authentication rejected after password regeneration" posted around the same time. Screenshot attached. This needs Railway engineering to look at the instance directly — I cannot self-diagnose or fix this from my side.
19 days ago
Try this:
- Click into your database service and go to the console tab
- Run this command:
sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf(This will bypass user authentication) - Redeploy your database
- Console again, and run the command
psql - Run
ALTER USER postgres with password '<PASSWORD>';where<PASSWORD>is the value of the variablePGPASSWORDin your Railway dashboard - Type
exit - Run
sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf(This will re-enable user authentication) - Redeploy your database
0x5b62656e5d
Try this: 1. Click into your database service and go to the console tab 2. Run this command: `sed -i 's/host all all all scram-sha-256/host all all ::\/0 trust/' /var/lib/postgresql/data/pgdata/pg_hba.conf` (This will bypass user authentication) 3. Redeploy your database 4. Console again, and run the command `psql` 5. Run `ALTER USER postgres with password '<PASSWORD>';` where `<PASSWORD>` is the value of the variable `PGPASSWORD` in your Railway dashboard 6. Type `exit` 7. Run `sed -i 's/host all all ::\/0 trust/host all all all scram-sha-256/' /var/lib/postgresql/data/pgdata/pg_hba.conf` (This will re-enable user authentication) 8. Redeploy your database
19 days ago
Update: The suggested fix (temporarily switching pg_hba.conf to trust, resetting the password via ALTER USER, then reverting to scram-sha-256) worked — authentication is now fixed and I can connect normally both via psql and DBeaver. Thank you for that.
However, this did not recover the missing data. I ran a direct check:
SELECT pg_size_pretty(pg_database_size('railway'));
→ 7774 kB
This is essentially an empty database (default system catalog size only). \l shows only the standard postgres/railway/template0/template1 databases, and \dn shows only an empty public schema — no application tables, no data.
To be clear: all of this data existed and was actively tested (including a SignalR feature) just a few hours before the crash-loop and auth issues started. I did not run any DROP, migration, or manual deletion. This appears to be genuine data loss, not a self-inflicted change or a migration that was never applied.
Given the timing overlap with other threads posted around the same window (private networking failures, auth rejected after password regeneration), this looks like it may be connected to a broader incident.
My plan doesn't include paid Backups/PITR, but I wanted to ask directly: is there any infrastructure-level snapshot, volume state, or internal recovery mechanism Railway can check on their end that predates this incident? Any guidance on next steps would be greatly appreciated — this is genuine lost production data, not something I'm trying to recover from my own mistake.