18 days ago
Hi
Around 18:52 17/06/2026 we begin experiencing issues with one of our services.
- Project: 21e9f3a9-e3df-44e8-a3df-edea18023284
- Environment: aa01bf3c-4b26-4b81-a857-c54f68affdb7
- Service: 5d6ef8a9-745a-4aab-8907-7cd228a40f68
It is a very simple tracking service that take POST requests from the frontend services and other services can get tracking data via. another API endpoint in the service.
After doing alot of debug the bottleneck seems to be the database, so I begin to tweak the postgres server config by increasing the work_mem etc.
I also tried to improve the queries with better indexing and more.
All in all only small improvements and whenever there was more then 1-2 writes a second the database freezed for up to 100+ seconds.
The metrics for the service+database didn't seem to increase and wasn't breaking a sweat. So the issue must be something internal with Postgres + disk.
This let me to the WAL + fsync, which came out as the most likely issue based on the these logs:
2026-06-18 16:58:40 UTC sync=111.3 s (flushed ~600 KB)
2026-06-19 00:33:36 UTC sync=100.5 s
2026-06-19 05:33:10 UTC sync= 72.8 s
2026-06-18 23:52:58 UTC sync= 65.4 s
2026-06-18 15:07:49 UTC sync= 62.1 s
2026-06-18 19:02:38 UTC sync= 50.6 s
2026-06-19 02:42:36 UTC sync= 37.4 s
2026-06-18 14:42:20 UTC sync= 36.1 s
2026-06-19 04:57:27 UTC sync= 28.3 s
2026-06-18 13:27:09 UTC sync= 25.3 s
Aggregate over 211 checkpoints (window 2026-06-18 12:11 → 2026-06-19 05:42 UTC):
- pg_stat_checkpointer: num_timed=210, num_requested=1, write_time=915968 ms, sync_time=700489 ms, buffers_written=15831
- Per-checkpoint fsync: p50 = 0.17 s, p90 = 2.1 s, max = 111.3 s — ~3.3 s average to flush ~600 KB
- 11 checkpoints exceeded 10 s of fsync; 8 exceeded 30 s
Representative app impact: a ~41 s full database stall at 2026-06-19 04:43:50–04:44:31 UTC (client statement timeouts fired mid-checkpoint).
These sync times seems VERY high compared to the small size of the actual load + database.
My solution has been to spin up a new DB and migrated the data to that database, which seems to have fixed all issues.
Luckly this is not a very important service for us and most likely we lost some tracking data without much value.
But more importantly I hope you can check 2 things for me:
- For my own sanity can you please validate that this seems like the actual cause of the issue. This is the database:
- Project: 21e9f3a9-e3df-44e8-a3df-edea18023284
- Environment: aa01bf3c-4b26-4b81-a857-c54f68affdb7
- Service (Postgres): c74627a5-51a3-477c-a003-f865f1cffdd9
- Deployment: 51e18657-3466-43b6-9cb1-30c7bd5c5b23
- Replica: 7c60c4dd-53b4-4e6d-aeef-507f340302ac
I have left it for you to research
- Check if this can give issues with other services that uses the same disk.
2 Replies
Status changed to Awaiting Railway Response Railway • 18 days ago
17 days ago
We looked into this, and your read on it is right: this was storage-layer latency on the host your database was running on, not your Postgres configuration or your workload, the fsync times are far beyond what your data volume would explain. Moving to the new database resolved it because it landed on healthy storage. Our team is checking the affected host now to make sure your other services in this region aren't impacted and to get it remediated, and we'll follow up here. You're safe to keep running on the new database in the meantime.
Status changed to Awaiting User Response Railway • 17 days ago
Status changed to Awaiting Railway Response angelo-railway • 17 days ago
17 days ago
Your analysis is sound. Checkpoint fsync hitting 100+ seconds to flush only ~600 KB, while the service's own CPU/memory metrics stay flat, is not a Postgres-config or query problem, it's a disk-level write stall: the WAL/data fsync waiting on the underlying volume. Tuning work_mem or indexes wouldn't move that, and the fact that migrating to a fresh DB (landing your data on different backing storage) cleared it confirms the root cause was the volume's backing disk, not your workload. So yes, your read is correct. On your second question: a degraded disk can absolutely affect other services whose volumes sit on the same host, so we're checking the host that volume was on for broader impact and will pull it from rotation if it's unhealthy. Thanks for the detailed write-up, it made this easy to confirm. — Angelo
Status changed to Awaiting User Response Railway • 17 days ago
Status changed to Solved madsmhm • 15 days ago