Seeing some MySQL lag ~45s on an idle database (production)
bgsesr-code
PROOP

a month ago

I used Claude to investigate some details on database errors, and gathered some information for review:

SYMPTOM

Our MySQL connections intermittently stall for ~45 seconds. The database is

effectively idle when this happens. On 2026-07-13 (UTC) our app logged two

episodes via HikariCP leak detection (threshold 30s):

02:18:36 -> 02:19:21 ~45s three separate connections, ALL released in the

                            same millisecond (02:19:21.103)

13:35:03 -> 13:35:48 ~45s single connection

Separately, a single-row INSERT into a table containing one row, run from a

client via the TCP proxy (gondola.proxy.rlwy.net:38846), took ~10 seconds.

WHAT WE HAVE ALREADY RULED OUT

This is not load, resources, or a slow query:

  • MySQL CPU: 0 vCPU (0%) of 32 vCPU
  • MySQL memory: 707 MB of 32 GB
  • Volume: 1.29 GB of 48.8 GB
  • App traffic: ~3 HTTP requests/hour at the time
  • Innodb_log_waits = 0 (redo log not a bottleneck)
  • Innodb_data_pending_fsyncs = 0 (storage is NOT stalling)
  • Threads_connected = 31 (Max_used_connections 62 of max 151)
  • Aborted_clients = 26 (clients dying without a clean close)

Config is the stock Railway MySQL template, untuned.

WHY WE THINK IT IS THE NETWORK / HOST, NOT MYSQL

  • Zero pending fsyncs and zero log waits, at 0% CPU, rules out disk and CPU.

  • Both connection paths are affected: the private network

    (mysql.railway.internal) AND the public TCP proxy. Those are different

    routes; the common factor is the MySQL instance's host/network.

  • Three connections unblocking in the SAME MILLISECOND is characteristic of a

    connection-path stall clearing, not of a database working through a queue.

  • Aborted_clients climbing is consistent with sockets being dropped.

IMPACT

Certain Inserts take much longer than expected.

WHAT WE ARE ASKING

  1. Were there host events — migration, throttling, noisy neighbour, network

    maintenance — affecting this MySQL instance during those two windows?

  2. Are there known private-network / TCP-proxy issues in US West around

    2026-07-13?

  3. Is there anything on your side we can monitor or alarm on so we see this

    before a customer does?

Happy to provide raw logs or run any diagnostics you need.

$20 Bounty

1 Replies

Railway
BOT

a month ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway about 1 month ago


a month ago

  1. The 45s stalls are pointing toward network related delays. Reverse DNS lookups, when db server tries resolving client IP address to hostname, can hang if DNS server is slow or not in reach.

  2. Check for connection timeouts or misconfigured connection Pooler settings (like HikariCP) which might be exhausting the pool.

To check for DNS errors, share relative server logs here


Welcome!

Sign in to your Railway account to join the conversation.

Loading...