Intermittent PostgreSQL/WAL storage stall causing 59s API request
sinegovsky-ivan
HOBBYOP

18 days ago

Hello Railway Support,

We are experiencing intermittent PostgreSQL latency in production.

Project: Bober Travel

Environment: production

Region: europe-west4-drams3a

Services:

  • Backend: 790333bb-3999-42bc-8ec7-628b6c537ee6
  • PostgreSQL: c0c3ced9-ac84-4f07-9739-6aef32e13816

Affected request:

  • Time window: 2026-06-19 12:07:35–12:08:34 UTC
  • Method/path: POST /route
  • Status: 201
  • Railway request ID: vEq6_lXBTZKmYccv63j4Ww
  • Backend deployment: 18b78625-e39d-46a5-8518-5d80ed7c7f6b
  • Deployment instance: 11b3b600-bf9b-4617-a08b-9e87e6430a61
  • Sentry trace ID: 4c0c27d79d7b49d5a5a30429535c3d05

Timing:

  • Railway totalDuration: 59,152 ms
  • Railway upstreamRqDuration: 59,148 ms
  • Application duration: 59,122 ms
  • Prisma/DB duration: 59,103 ms across 6 sequential queries

The same endpoint normally completes in 38–189 ms.

The services were not under meaningful CPU or memory load. PostgreSQL had 4 connections out of max_connections=100 when inspected, no recorded

deadlocks, and the volume was approximately 320 MB / 500 MB.

PostgreSQL logs also show unusually long checkpoint/fsync operations. For example:

2026-06-19 10:51:30.008 UTC

checkpoint complete: wrote 6 buffers; write=0.536 s, sync=110.643 s, total=114.366 s

Other checkpoints today took approximately 30–105 seconds despite writing only 5–22 buffers.

This appears consistent with intermittent volume/WAL fsync latency rather than application CPU load or connection-pool exhaustion. No checkpoint

was logged during the exact affected request, so we cannot prove that its specific wait was an fsync; however, 59.103 seconds were spent inside

the database client and the storage stalls are independently visible in PostgreSQL logs.

Could you please:

  1. Inspect storage/volume latency for the PostgreSQL service during 2026-06-19 12:07:35–12:08:34 UTC.
  2. Check whether the underlying host or volume experienced an I/O incident.
  3. Advise whether the database should be migrated/reprovisioned onto a new volume or host.
  4. Confirm whether similar latency is visible in Railway’s internal telemetry.

Thank you.

Solved

6 Replies

Status changed to Awaiting Railway Response Railway 18 days ago


Hi Ivan, thanks for the unusually thorough report, it made this quick to confirm. We can independently see the storage stalls in our telemetry for your Postgres service: checkpoints are completing with sub-second write and sync phases but total durations of 30-105 seconds (one this morning took 104.7s total with only 0.22s of actual sync). That gap is time the database spends blocked at the volume/storage layer, not CPU, connections, or your application, so your diagnosis is correct. This is consistent with a degraded storage host rather than anything in your project. Our infrastructure team is locating the underlying host for your Postgres volume in europe-west4-drams3a to migrate it onto healthy storage, and we'll update this thread once that's underway. No action needed on your side, and a redeploy on its own won't reliably move you off the affected host.


Status changed to Awaiting User Response Railway 17 days ago


Status changed to Awaiting Railway Response angelo-railway 17 days ago


Quick status update: this is still being handled as a storage-host issue, not an application or connection-pool issue on your side.

Your report and the telemetry both point at the volume/storage layer, and a normal redeploy is not expected to reliably fix it. The pending action is still to identify the underlying host and move the database volume off degraded storage. We will update this thread once that work is complete or if we need anything from you.


Status changed to Awaiting User Response Railway 14 days ago


angelo-railway

Quick status update: this is still being handled as a storage-host issue, not an application or connection-pool issue on your side. Your report and the telemetry both point at the volume/storage layer, and a normal redeploy is not expected to reliably fix it. The pending action is still to identify the underlying host and move the database volume off degraded storage. We will update this thread once that work is complete or if we need anything from you.

sinegovsky-ivan
HOBBYOP

10 days ago

Hi! Thank you! Will wait for update


Status changed to Awaiting Railway Response Railway 10 days ago


8 days ago

Hey!

Are you still seeing this high latency?


Status changed to Awaiting User Response Railway 8 days ago


noahd

Hey! Are you still seeing this high latency?

sinegovsky-ivan
HOBBYOP

8 days ago

Hi! Didn't noticed it for a few days, everything should be resolved?


Status changed to Awaiting Railway Response Railway 8 days ago


8 days ago

It should be resolved yes! If you run into issues like this again can you please let me know? Thanks!


Status changed to Awaiting User Response Railway 8 days ago


Railway
BOT

15 hours ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway about 15 hours ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...