a month ago
We've had intermittent freezes on our API (subastoo_main_api, project glorious-energy) over the last weeks that we've traced back to the Postgres service in the same project. The symptoms don't match anything on our side, so I'm writing in case this is something you recognize on the infrastructure.
What we see: at random times, several times a day, writes to a specific table start queuing for 20-60 seconds, then recover on their own. During these episodes, CPU on the Postgres service sits at essentially zero (~0.0005 vCPU), and memory and disk are well under their limits, so it doesn't look like a resource exhaustion issue on our end.
We captured pg_stat_activity during one of these episodes, and the process blocking everyone else looks like this:
state: active
wait_event_type: IO
wait_event: WalSync
query: update "auth_access_tokens" set "last_used_at" = $1 ...
45 seconds stuck in WalSync, without executing anything heavy — it's a single-row UPDATE. Other processes were queued behind it, waiting on the same row lock.
We also ran an ALTER SYSTEM SET ... (which only writes to postgresql.auto.conf, no table access at all) and it took 48 seconds to complete, in the middle of one of these episodes. That makes us think this isn't contention inside the database, but the underlying disk stalling on fsync.
We found a couple of threads on your forum with identical symptoms (WALSync/WALWrite hanging for several seconds, low CPU), where it turned out to be shared disk hardware on the physical host, resolved by moving the service to a different host:
https://station.railway.com/questions/assistance-requested-postgre-sql-sudden-04258f1a
https://station.railway.com/questions/database-performance-issues-slow-stor-f6213a55
Our Postgres is in EU-West (Amsterdam). We don't think it's a region-wide issue, since we have another project of ours in the same region that has never shown this symptom — which points more to the specific host we landed on than to the region as a whole. We already redeployed the service in case it got reassigned to a different host, but the issue persists.
Could you check whether the host our volume is on has I/O contention, and whether it's possible to move us to a different host within EU-West? We'd rather not switch regions, since it would add latency for our users (all based in Spain).
Service: Postgres in project glorious-energy, active deployment fcf926c6-6476-43e9-8f96-68c19bd17b9d.
Thanks in advance.
3 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
a month ago
DONE This matches a known pattern on Railway's EU-West (Amsterdam) infra rather than anything in your app or schema — two prior threads had identical symptoms, both fixed only by Railway relocating the database to a different host:
- Assistance requested: PostgreSQL sudden extreme slowdown (WALSync/WALWrite blocked 10s+, ~0 CPU, resolved by moving off the host)
- Database Performance Issues - Slow Storage I/O (checkpoint writes of 80–800s+, only fixed after Railway bounced the DB through another region to force a new host)
In both, WAL config changes, warmup queries, VACUUM, and redeploys did nothing — the bottleneck was fsync latency on the underlying disk/host, not Postgres config or app behavior. It only cleared once a Railway engineer moved the service to different hardware.
Your ALTER SYSTEM SET test is the clincher: that command only fsyncs postgresql.auto.conf — it never touches WAL or any table. If a plain config write also takes 48 seconds, the whole mount is stalling on fsync, not just WAL, which rules out lock contention or anything specific to auth_access_tokens. The row-lock queueing you saw is a downstream symptom (the stuck UPDATE holds its row lock until COMMIT finishes syncing) — not the cause.
Given the precedent, I don't think this is fixable from your side. Worth explicitly asking Railway to:
- Check host-level fsync/I/O latency on the physical host under deployment
fcf926c6-6476-43e9-8f96-68c19bd17b9d's volume (not visible frompg_stat_activityor your CPU/memory graphs). - Move you to a different host within EU-West — note a plain redeploy didn't do this for either thread above; it took a manual move by a Railway engineer.
Independent of that: since the stalls concentrate on auth_access_tokens.last_used_at, you could decouple that write from the request path — debounce it to, say, once a minute per token, or push it through a queue — so a stalled fsync on one hot row doesn't queue every other request behind it. Doesn't fix the host issue, but shrinks the blast radius while you wait on Railway.
Sources:
eggy8
DONE *This matches a known pattern on Railway's EU-West (Amsterdam) infra rather than anything in your app or schema — two prior threads had identical symptoms, both fixed only by Railway relocating the database to a different host:* - [Assistance requested: PostgreSQL sudden extreme slowdown](https://station.railway.com/questions/assistance-requested-postgre-sql-sudden-04258f1a) *(WALSync/WALWrite blocked 10s+, ~0 CPU, resolved by moving off the host)* - [Database Performance Issues - Slow Storage I/O](https://station.railway.com/questions/database-performance-issues-slow-stor-f6213a55) *(checkpoint writes of 80–800s+, only fixed after Railway bounced the DB through another region to force a new host)* *In both, WAL config changes, warmup queries, VACUUM, and redeploys did nothing — the bottleneck was fsync latency on the underlying disk/host, not Postgres config or app behavior. It only cleared once a Railway engineer moved the service to different hardware.* *Your `ALTER SYSTEM SET` test is the clincher: that command only fsyncs `postgresql.auto.conf` — it never touches WAL or any table. If a plain config write also takes 48 seconds, the whole mount is stalling on fsync, not just WAL, which rules out lock contention or anything specific to `auth_access_tokens`. The row-lock queueing you saw is a downstream symptom (the stuck UPDATE holds its row lock until COMMIT finishes syncing) — not the cause.* *Given the precedent, I don't think this is fixable from your side. Worth explicitly asking Railway to:* 1. *Check host-level fsync/I/O latency on the physical host under deployment `fcf926c6-6476-43e9-8f96-68c19bd17b9d`'s volume (not visible from `pg_stat_activity` or your CPU/memory graphs).* 2. *Move you to a different host within EU-West — note a plain redeploy didn't do this for either thread above; it took a manual move by a Railway engineer.* *Independent of that: since the stalls concentrate on `auth_access_tokens.last_used_at`, you could decouple that write from the request path — debounce it to, say, once a minute per token, or push it through a queue — so a stalled fsync on one hot row doesn't queue every other request behind it. Doesn't fix the host issue, but shrinks the blast radius while you wait on Railway.* Sources: - [Bounties | Railway Docs](https://docs.railway.com/community/bounties) - [Bounties - Railway Central Station](https://station.railway.com/bounties) - [Assistance requested: PostgreSQL sudden extreme slowdown](https://station.railway.com/questions/assistance-requested-postgre-sql-sudden-04258f1a) - [Database Performance Issues - Slow Storage I/O Causing Extended Checkpoint](https://station.railway.com/questions/database-performance-issues-slow-stor-f6213a55)
a month ago
Hi, thanks for your help. Could you tell me how I should proceed to request this fix from Railway? With the Hobby plan, I can't submit a private ticket. Do you think I can simply wait for them to see the ticket on this forum?
a month ago
Good question — private tickets are a Pro-plan feature, so that's not available here. But posting publicly is actually the intended path for Hobby-tier support (Railway's own docs describe Hobby/Free as community support via Central Station or Discord, just without a guaranteed response time) — you're not missing a step.
A couple of free things that help: also drop a pointer to this thread in Railway's Discord #help channel for extra visibility, and if it goes quiet for a day or two, post a follow-up update rather than letting it sit — Central Station's bot can auto-mark inactive threads "Solved" even when they aren't, which buries them.
If you need a guaranteed response time, that only comes with Pro (~72h target) or Business Class ($5k+/mo spend, real SLOs) — but for a platform-level issue like this, the free path (public thread + Discord nudge) is reasonable.
Sources:
Support | Railway Docs
Bounties | Railway Docs