sync took 85s on a 70KB checkpoint while DB is essentially idle; sustained fsync latency >10s is making writes unusable.
flyingminion10
PROOP

18 days ago

  1. If this still doesn't work, the storage is fundamentally degraded. You'd need to contact Railway support or migrate to a different region/plan.
Solved

4 Replies

flyingminion10
PROOP

18 days ago

Here are the logs

Attachments


flyingminion10
PROOP

18 days ago

Same PUT transactions on railway's DB is taking +25000 ms; I temporary migrated to NEON database and same operation 623 ms. I really want to stay with railway, cause all my infra is here, but I can't wait too much


flyingminion10
PROOP

18 days ago

Thanks railway team, you suck, all day debugging my app, at the end a damaged disc volume.

Attachments


Status changed to Awaiting Railway Response brody 11 days ago


codydearkland
EMPLOYEE

8 days ago

Hi — I'm so sorry. It's been 10 days, and that's on us. You shouldn't have had to debug your app all day, and you definitely shouldn't have been left without an answer since.

I went through your logs and the host metrics for that night and want to share what I found.

Your Postgres was on a specific host where the underlying ZFS storage was under severe contention from ~02:00 UTC through ~06:00 UTC on May 5. Your Postgres logs show a checkpoint taking 56 seconds of fsync to flush 7 KB at 04:31:20 UTC. That matches what we see on the host: write throttling firing tens of thousands of times per minute, sustained iowait well above normal. Looking across the rest of the region, no other stacker came close — this was localized to that one host. Your workload wasn't the cause; it got caught in the contention.

I can see your redeployment at 05:29 UTC moved Postgres to us-east, and it has been healthy since with no recurrence of the latency.

Again, genuinely sorry for the delay and for what the night looked like on your side. If anything still feels off on the new region, reply here and I'll stay on it directly.


Status changed to Awaiting User Response Railway 8 days ago


Status changed to Solved codydearkland 8 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...