4 days ago
Hello my db times out throughout the day sometimes and it seems to be a disk issue problem. Would a redeploy solve this or wha do you reccomend?
16 Replies
4 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 4 days ago
4 days ago
You can try selecting the service, then press Cmd/Ctrl + K, then select redeploy source image.
0x5b62656e5d
You can try selecting the service, then press Cmd/Ctrl + K, then select redeploy source image.
4 days ago
Thanks. Before I restart anything I'd like a look at the host, because this looks like the volume's storage rather than Postgres itself. Two events from the Postgres logs, same fingerprint both times: reads unaffected, writes frozen for 70-90 seconds, recovers on its own.
Sep 5, 16:13 UTC: reads 3 ms, writes stalled 88 s (about 110 s total). The checkpoint that finished during recovery: write=4.660 s, sync=0.927 s, total=33.725 s.
Sep 7, 17:25 UTC: reads 4 ms, writes stalled 73 s. The checkpoint: wrote 209 buffers (about 2 MB of WAL), write=15.8 s, sync=0.4 s, total=87.6 s. 2 MB should take milliseconds. Every waiting backend during the stall was on IO:WalSync / LWLock:WALWrite, not application queries. App CPU and memory were flat and our bulk-write job was idle during the second event.
Questions:
- Does "redeploy source image" move the volume to a different host, or restart the container on the same one?
- Can you check the host this volume sits on for I/O latency or neighbor contention around those two timestamps?
- If the host is the problem, can the volume be migrated to another host, and what downtime does that involve?
a day ago
Please someone help
Status changed to Awaiting Railway Response Railway • 1 day ago
a day ago
We are looking into this and will follow up as soon as we have more information to share.
Status changed to Awaiting User Response Railway • 1 day ago
a day ago
We have migrated your production database service to a different host, so you will no longer experience disk stalls.
brody
We have migrated your production database service to a different host, so you will no longer experience disk stalls.
a day ago
Awesome thank you! Since we are growing I do have. few questions. For enterprise I see the max for the server increases but it still seems to small for us long term. Are my only options AWS or is there a way to get more information.
Status changed to Awaiting Railway Response Railway • about 24 hours ago
Status changed to Awaiting User Response Railway • about 24 hours ago
brody
What kind of capacity do you see yourself needing in all aspects?
a day ago
Also does my Ip change here? Do I need to update my other servers? I have a pretty complex system and it seems just with the clients I have at least with the previous server were hitting closer to the max and I expect to 10x in growth within 12 months.
Status changed to Awaiting Railway Response Railway • about 24 hours ago
brody
What kind of capacity do you see yourself needing in all aspects?
a day ago
is it normal ive gotten 3 disk stalls since the move lsting 25 second and freezes everything at once atleast ai is saying that as the logs show db timeout stall from what I saw. it said it froze i was not on the page. everything seems fine just wanted to inform
anon12345
Also does my Ip change here? Do I need to update my other servers? I have a pretty complex system and it seems just with the clients I have at least with the previous server were hitting closer to the max and I expect to 10x in growth within 12 months.
13 hours ago
Nothing I did would have changed your IP.
Status changed to Awaiting User Response Railway • about 13 hours ago
anon12345
is it normal ive gotten 3 disk stalls since the move lsting 25 second and freezes everything at once atleast ai is saying that as the logs show db timeout stall from what I saw. it said it froze i was not on the page. everything seems fine just wanted to inform
13 hours ago
That is not normal, your database is on much newer hardware now, I will look into that.
Status changed to Awaiting Railway Response brody • about 13 hours ago
8 hours ago
Looked at both the host and the database. There's no disk problem on the new machine: write latency is averaging 1.25ms, you're using about 1% of your volume's throughput limit, and the host is close to idle.
The numbers reading as stalls are checkpoint logs, and total= isn't freeze time. Postgres paces checkpoint writes across most of your 5 minute checkpoint interval on purpose. Every checkpoint in your logs works out to exactly 100ms per buffer, which is that pacing timer, not disk. The field that does measure disk is sync=, and yours runs 0.007 to 0.4 seconds. Over the last 14 hours your checkpoints spent 1503 seconds pacing and 17 seconds writing.
I also pulled the window where you saw the three 25 second freezes. Every checkpoint in it finished between 0.9 and 8.3 seconds.
What's real is your config. shared_buffers is 160MB against a 37GB database on a container with 32GB of RAM, so your buffer pool turns over completely every couple of seconds and your hit rate sits at 87.6%. The background writer keeps hitting its page ceiling and stopping mid round, which pushes dirty page writes onto your query backends. That's what puts them on WALWrite and WalSync waits, and it's the symptom you described from the start.
For 32GB and 32 vCPU, try shared_buffers 8GB, effective_cache_size 24GB, maintenance_work_mem 2GB, bgwriter_lru_maxpages 1000, max_wal_size 8GB, checkpoint_timeout 15min, random_page_cost 1.1.
On capacity, you're nowhere near the IO you're allowed today. Get the config right first, then tell me the numbers you're planning for and I'll tell you what we can do.
Status changed to Awaiting User Response Railway • about 8 hours ago
brody
Looked at both the host and the database. There's no disk problem on the new machine: write latency is averaging 1.25ms, you're using about 1% of your volume's throughput limit, and the host is close to idle. The numbers reading as stalls are checkpoint logs, and `total=` isn't freeze time. Postgres paces checkpoint writes across most of your 5 minute checkpoint interval on purpose. Every checkpoint in your logs works out to exactly 100ms per buffer, which is that pacing timer, not disk. The field that does measure disk is `sync=`, and yours runs 0.007 to 0.4 seconds. Over the last 14 hours your checkpoints spent 1503 seconds pacing and 17 seconds writing. I also pulled the window where you saw the three 25 second freezes. Every checkpoint in it finished between 0.9 and 8.3 seconds. What's real is your config. shared_buffers is 160MB against a 37GB database on a container with 32GB of RAM, so your buffer pool turns over completely every couple of seconds and your hit rate sits at 87.6%. The background writer keeps hitting its page ceiling and stopping mid round, which pushes dirty page writes onto your query backends. That's what puts them on WALWrite and WalSync waits, and it's the symptom you described from the start. For 32GB and 32 vCPU, try shared_buffers 8GB, effective_cache_size 24GB, maintenance_work_mem 2GB, bgwriter_lru_maxpages 1000, max_wal_size 8GB, checkpoint_timeout 15min, random_page_cost 1.1. On capacity, you're nowhere near the IO you're allowed today. Get the config right first, then tell me the numbers you're planning for and I'll tell you what we can do.
7 hours ago
So to clarify I have a config server setting issue? Can I change it without resetting server since clients are using right now. What would I change it too? How much room would I have to grow because It seemed were hitting 26 gb ram at the peak recently.
Status changed to Awaiting Railway Response Railway • about 7 hours ago
3 hours ago
The databases you deploy on Railway are unmanaged. We run the infrastructure underneath them, but the database itself is yours, so I'm not able to make configuration changes on your behalf or advise on how to apply them.
The values I gave earlier were a high level suggestion based on what your logs and the host metrics showed, and that's as far as I can take it. Whether they require a restart, and how to roll them out while clients are connected, is outside what I'm able to advise on. Apologies for proposing the lower level configuration without being clear about that up front.
Status changed to Awaiting User Response Railway • about 3 hours ago
brody
The databases you deploy on Railway are unmanaged. We run the infrastructure underneath them, but the database itself is yours, so I'm not able to make configuration changes on your behalf or advise on how to apply them. The values I gave earlier were a high level suggestion based on what your logs and the host metrics showed, and that's as far as I can take it. Whether they require a restart, and how to roll them out while clients are connected, is outside what I'm able to advise on. Apologies for proposing the lower level configuration without being clear about that up front.
2 hours ago
makes sense my ai took what you said built out so i can apply all you reccomend. Is there anything missing?
Status changed to Awaiting Railway Response Railway • about 2 hours ago
an hour ago
The values I gave you still stand, so if your AI built them out faithfully you're in reasonable shape to apply them.
Where I have to stop is reviewing the expanded version. I can see your host metrics and your Postgres logs, but not your queries or your access patterns, and a full tuning pass needs those. It's also worth being clear about what we are. Railway is an infrastructure company, not a database company. Postgres runs on us, we don't build or operate it, and our support team isn't the authority on tuning it. For something your clients depend on, you want that advice from someone who is.
Status changed to Awaiting User Response Railway • about 1 hour ago