[EMERGENCY] mysql not working singapore
youngsuk-kim
PROOP

16 days ago

image.png

Status: api-server is DOWN — 100% HTTP error rate (production).

Symptoms

  • HTTP error rate: 100% (200/200 sampled requests returning 4xx/5xx)
  • HikariCP pool fully drained: total=0, active=0, idle=0, waiting=5 — cannot create a single connection
  • MySQL CPU: 0% sustained (07:56–08:00 UTC) — process alive but receiving zero queries
  • Errors ongoing for 6+ minutes with no self-recovery (last error 17:00:18 KST)

Error chain (api-server)

java.net.SocketTimeoutException: Connect timed out

→ com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure

→ java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available,

  request timed out after 3000ms (total=0, active=0, idle=0, waiting=5)

→ org.springframework.transaction.CannotCreateTransactionException:

  Could not open JPA EntityManager for transaction

Timeline (UTC)

  • 07:54:18 — MySQL received graceful SIGTERM (Received SHUTDOWN from user ) — Railway platform-initiated restart,

not a crash

  • 07:54:21 — MySQL back up, logged "ready for connections"
  • 07:58 onward — api-server flooding Connect timed out / Communications link failure
  • 08:00:18 — still failing; Hikari pool empty, requests queueing

Root cause

MySQL container restarted in place (no redeploy — last MySQL deploy was 2026-05-22). After restart, the MySQL service is alive but

CPU 0% = no queries reaching it. api-server fails at the socket connect layer (Connect timed out, before auth), meaning it cannot

reach MySQL over Railway private networking — almost certainly connecting to a stale internal address from before the restart.

HikariCP retries the dead address forever, so total=0 never recovers on its own.

"Forcing close of thread" warnings = NOT the cause

Those [MY-010909] Forcing close of thread warnings are normal MySQL shutdown noise. Same pattern appeared on the 2026-05-22 restart.

Benign.

Attachments

0 Replies

Welcome!

Sign in to your Railway account to join the conversation.

Loading...