a day ago
Hello, from the past 2 - 3 days the servers has been really slow. Giving backend response times of 12 - 20 secs, I believe because of database slowness too for really simple queries
2 Replies
a day ago
Apologies for the trouble. Your database runs in US West, where we had two incidents over the last couple of days: one made services and databases unreachable for some users on October 3, and one slowed attached storage, which made queries take longer than expected. Both have since been resolved.
You can see the full timelines here: slow attached storage and services and databases unavailable.
Status changed to Awaiting User Response Railway • 1 day ago
a day ago
Thanks for the update. The two incidents clearly hit us, and things are much better since around 06:30 UTC today (Oct 4). But we're still seeing short storage/latency stalls this morning that don't match our load, so I'd like to check
whether the attached-storage issue is fully resolved for our volume.
Setup: MySQL 9.4 service with a volume (US West), one FastAPI service in the same project/region.
What we measured (all times UTC, Oct 4):
- Our load is very low: ~40 HTTP req/min, ~1 commit/s, 6 open connections, no row-lock waits, and the buffer pool serves everything from memory (no disk reads).
- 10:48-11:34: single-row primary-key UPDATEs (~12 KB) took 2-9 s, with no lock time. We've temporarily set innodb_flush_log_at_trx_commit=2, so commits don't even fsync, and these writes are still slow.
- 12:07-12:08: for about a minute every API route, across different users, took 8-33 s, while MySQL's slow query log (threshold 1 s) shows nothing in that window. That points at network or host latency between the API and the database, not
at query execution.
- For comparison, on Oct 3 between 13:00 and 00:00 UTC we logged 300-1,000 slow queries per hour, mostly plain COMMITs averaging 3-5 s. Since 06:30 UTC today that's down to 1-30 per hour.
Could you check whether our MySQL volume/host and the API service are still affected by the storage or network degradation? Happy to share request IDs or exact timestamps.
Status changed to Awaiting Railway Response Railway • 1 day ago