3 months ago
Hi, guys!
I changed one of my Postgres' service's env vars (I removed the TCP proxy and the DATABASE_PUBLIC_URL, since I don't need my DB to be externally accessible) and triggered a redeployment of the service (so that the env var changes would get applied).
I then noticed these errors in the logs:
```
2025-09-02 12:17:25.065 UTC [6] LOG: starting PostgreSQL 17.6 (Debian 17.6-1.pgdg13+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
2025-09-02 12:17:25.066 UTC [6] LOG: listening on IPv4 address "0.0.0.0", port 5432
2025-09-02 12:17:25.066 UTC [6] LOG: listening on IPv6 address "::", port 5432
2025-09-02 12:17:25.098 UTC [6] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2025-09-02 12:17:25.146 UTC [29] LOG: database system was interrupted; last known up at 2025-09-02 11:20:26 UTC
2025-09-02 12:17:25.601 UTC [29] LOG: database system was not properly shut down; automatic recovery in progress
2025-09-02 12:17:25.611 UTC [29] LOG: redo starts at 0/2290088
2025-09-02 12:17:25.611 UTC [29] LOG: invalid record length at 0/22901C8: expected at least 24, got 0
2025-09-02 12:17:25.611 UTC [29] LOG: redo done at 0/2290190 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
2025-09-02 12:17:25.638 UTC [27] LOG: checkpoint starting: end-of-recovery immediate wait
2025-09-02 12:17:25.699 UTC [27] LOG: checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.012 s, sync=0.017 s, total=0.081 s; sync files=2, longest=0.011 s, average=0.009 s; distance=0 kB, estimate=0 kB; lsn=0/22901C8, redo lsn=0/22901C8
2025-09-02 12:17:25.710 UTC [6] LOG: database system is ready to accept connections
```
Now, from what I can tell, this means that the redeployment caused the process to be killed (maybe via a SIGQUIT instead of a SIGTERM?) before it had a chance to shutdown properly, but the recovery process was successful and no data was lost. I found people with similar issues in other Station threads, but in their case the database seemed to have actually lost data and refused to start again, so my case is a bit different.
I have two questions:
1) Is my reading of this log correct, i.e.: no data was lost?
2) Can I do anything to configure the service in such a way that when redeploying again it shuts down properly, so that replaying the WAL log isn't needed when it starts up again? (https://www.postgresql.org/docs/current/server-shutdown.html)
Thanks!
4 Replies
3 months ago
Hey there! We've found the following might help you get unblocked faster:
If you find the answer from one of these, please let us know by solving the thread!
3 months ago
Hello,
Railway sends SIGTERM to processes and waits 60 seconds before force-killing them. This applies to all applications, including databases, since our databases are completely unmanaged.
If a database doesn't respect SIGTERM or can't flush its data to disk within 60 seconds, issues can occur. However, PostgreSQL is generally very resilient to this.
1. After reading your logs, yes, you're correct - no data was lost. The automatic recovery completed successfully.
2. You have a couple of options:
- SSH into the service before redeploying and manually shut down PostgreSQL cleanly using the appropriate psql command
- If your database requires more than 60 seconds to shut down properly, you can enable the teardown timeout option in your service settings and increase it accordingly
Best,
Brody
Status changed to Awaiting User Response Railway • 3 months ago
Status changed to Solved cipclever • 3 months ago
3 months ago
@brody I see, thanks! Interesting that it's taking so long to shutdown properly -- it's a just-created database with very little data and no active connections for most of the day. Is this a common issue for the stock* Postgres image?
(*I guess my configuration isn't 100% stock though, since I removed the TCP Proxy and the DATABASE_PUBLIC_URL env var, but I highly doubt that would be affecting shutdown times since it's not a DB setting.)
Status changed to Awaiting Railway Response Railway • 3 months ago
3 months ago
Hello,
Since we offer unmanaged databases, I don't have visibility into what's typical for PostgreSQL shutdown behavior across different configurations. The shutdown time can depend on various factors specific to your database's state and configuration.
The 60-second timeout we provide should generally be sufficient for most PostgreSQL instances to shut down cleanly, but individual cases can vary.
Best,
Brody
Status changed to Awaiting User Response Railway • 3 months ago
Status changed to Solved brody • 3 months ago