a month ago
I used Claude to investigate some details on database errors, and gathered some information for review:
SYMPTOM
Our MySQL connections intermittently stall for ~45 seconds. The database is
effectively idle when this happens. On 2026-07-13 (UTC) our app logged two
episodes via HikariCP leak detection (threshold 30s):
02:18:36 -> 02:19:21 ~45s three separate connections, ALL released in the
same millisecond (02:19:21.103)13:35:03 -> 13:35:48 ~45s single connection
Separately, a single-row INSERT into a table containing one row, run from a
client via the TCP proxy (gondola.proxy.rlwy.net:38846), took ~10 seconds.
WHAT WE HAVE ALREADY RULED OUT
This is not load, resources, or a slow query:
- MySQL CPU: 0 vCPU (0%) of 32 vCPU
- MySQL memory: 707 MB of 32 GB
- Volume: 1.29 GB of 48.8 GB
- App traffic: ~3 HTTP requests/hour at the time
- Innodb_log_waits = 0 (redo log not a bottleneck)
- Innodb_data_pending_fsyncs = 0 (storage is NOT stalling)
- Threads_connected = 31 (Max_used_connections 62 of max 151)
- Aborted_clients = 26 (clients dying without a clean close)
Config is the stock Railway MySQL template, untuned.
WHY WE THINK IT IS THE NETWORK / HOST, NOT MYSQL
-
Zero pending fsyncs and zero log waits, at 0% CPU, rules out disk and CPU.
-
Both connection paths are affected: the private network
(mysql.railway.internal) AND the public TCP proxy. Those are different
routes; the common factor is the MySQL instance's host/network.
-
Three connections unblocking in the SAME MILLISECOND is characteristic of a
connection-path stall clearing, not of a database working through a queue.
-
Aborted_clients climbing is consistent with sockets being dropped.
IMPACT
Certain Inserts take much longer than expected.
WHAT WE ARE ASKING
-
Were there host events — migration, throttling, noisy neighbour, network
maintenance — affecting this MySQL instance during those two windows?
-
Are there known private-network / TCP-proxy issues in US West around
2026-07-13?
-
Is there anything on your side we can monitor or alarm on so we see this
before a customer does?
Happy to provide raw logs or run any diagnostics you need.
1 Replies
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
-
The 45s stalls are pointing toward network related delays. Reverse DNS lookups, when db server tries resolving client IP address to hostname, can hang if DNS server is slow or not in reach.
-
Check for connection timeouts or misconfigured connection Pooler settings (like HikariCP) which might be exhausting the pool.
To check for DNS errors, share relative server logs here