2 days ago
I'm seeing a sudden performance degradation on a Railway-hosted PostgreSQL 18.6 database.
The application began experiencing 10–30 second database-backed requests today. PostgreSQL checkpoint logs show a clear change beginning around 13:01 UTC:
- Before 13:01 UTC: checkpoint sync/flush times were typically 5–40 ms, with whole checkpoints under ~1.3 seconds.
- Starting around 13:01 UTC: sync/flush times increased to roughly 0.2–5.5 seconds, with some checkpoints taking up to ~24 seconds.
The application and database were running normally for hours before this change, with no deployment corresponding to the onset.
Railway metrics currently show very low PostgreSQL CPU usage, approximately 75–100 MB memory usage, and plenty of unused volume capacity. There have been no service restarts, deadlocks, database errors, or 5xx responses.
At idle, queries are currently completing in approximately 2–7 ms and there are no waiting locks or stuck transactions. During the degradation periods, however, normally fast requests queue for many seconds waiting for database connections/writes to complete.
This appears consistent with intermittent high write/fsync latency on the underlying PostgreSQL volume rather than CPU, memory, database locking, or application load.
Is there a way to check the underlying volume/storage I/O health for this PostgreSQL service, or determine whether the service is experiencing degraded storage performance on its host?
I found an existing Railway thread describing very similar intermittent WAL/fsync latency where migration to another region was suggested as a diagnostic step. Before moving a production database, I'd prefer to determine whether the underlying volume/host for this service is currently experiencing degraded I/O.
3 Replies
Status changed to Awaiting Railway Response Railway • 2 days ago
2 days ago
We checked the storage behind your Postgres volume. The host it runs on has had sharply elevated disk write latency since about 13:00 UTC today, with average write wait times rising several-fold from their earlier baseline. That lines up with the checkpoint sync and flush times you measured.
Your volume itself is healthy and well under its capacity and I/O limits, and nothing in your database or application configuration is causing this. The slowdown comes from the shared disk on that host, not from your workload.
Status changed to Awaiting User Response Railway • 2 days ago
2 days ago
Thanks, Nico. That lines up exactly with what we're seeing.
It appeared to recover for several hours, but the severe latency has now returned. In the most recent period, Postgres checkpoints have taken roughly 13–26 seconds, with disk flush/sync times around 0.6–7 seconds. We're again seeing the impact in normal production use, including several-second database-backed operations on a very light workload.
Since you've confirmed the issue is the shared disk on the host rather than our volume or workload, is there anything you can do on your side to move the service away from the affected host, or is there a migration/redeploy step you recommend we take?
I'd prefer not to move the production database unnecessarily if the underlying host issue can be resolved on your side.
Status changed to Awaiting Railway Response Railway • 1 day ago
2 days ago
Confirming that we're seeing this on our side, have called an incident here and are looking into this: https://status.railway.com/incident/A0O39CGE
Redeploying or restarting will not help here, we suggest holding off on both for now
Our infrastructure team is looking into it, and we'll update the incident as soon as we have more
Status changed to Awaiting User Response Railway • 1 day ago