a month ago
Hi, my service's volume I/O appears to be severely degraded since yesterday.
-
Project: quant-trading (id: d179eccd-153a-4673-9818-0718416b0d8d)
-
Service: quant-trading (id: 1cfe2377-b67e-4045-bf6c-22b712a294fa)
-
Environment: app-product
-
Volume: quant-trading-volume (id: a4f40714-ff56-4961-8a22-7ffb56dc164f),
mounted at /app/data_cache, region: sfo
Timeline (KST, UTC+9):
-
Until July 12 night: our nightly DuckDB batch job (sequential reads/writes
on the volume) completed normally (~4h).
-
July 13 morning: several container OOM restarts (our own app bug, since fixed).
-
July 13 ~19:55: volume was resized from 500MB to 5GB via the dashboard.
-
Since then: any sustained read/write workload on the volume stalls.
Evidence:
-
"railway volume files download" of a ~280MB file ran at ~170KB/s,
then timed out at 243MB.
-
Our batch job that previously completed in ~4h now cannot process even
its first items after 15+ minutes (DuckDB scans on a 280MB file).
-
Light random reads still work (API responses 0.6-2.3s) — only
throughput-heavy I/O is unusable.
-
The same job with the same data completes in ~8 minutes on a local
machine, so it is not an application-side issue.
Could you check the health of this volume / its underlying storage, and
migrate or repair it if degraded? Happy to provide more details. Thanks!
1 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
a month ago
There is nothing wrong with the volume on the Railway side; but has a 3K IOPS limit. Locally will have no IOPS limit, thus it will be faster.
Regarding volume download speed, you seem to be located far away from where the volume is (SFO), implying a 170kbps “speed limit” because of the round trip that the 32kb chunks have to take.