16 days ago
Status: api-server is DOWN — 100% HTTP error rate (production).
Symptoms
- HTTP error rate: 100% (200/200 sampled requests returning 4xx/5xx)
- HikariCP pool fully drained: total=0, active=0, idle=0, waiting=5 — cannot create a single connection
- MySQL CPU: 0% sustained (07:56–08:00 UTC) — process alive but receiving zero queries
- Errors ongoing for 6+ minutes with no self-recovery (last error 17:00:18 KST)
Error chain (api-server)
java.net.SocketTimeoutException: Connect timed out
→ com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
→ java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available,
request timed out after 3000ms (total=0, active=0, idle=0, waiting=5)
→ org.springframework.transaction.CannotCreateTransactionException:
Could not open JPA EntityManager for transactionTimeline (UTC)
- 07:54:18 — MySQL received graceful SIGTERM (Received SHUTDOWN from user ) — Railway platform-initiated restart,
not a crash
- 07:54:21 — MySQL back up, logged "ready for connections"
- 07:58 onward — api-server flooding Connect timed out / Communications link failure
- 08:00:18 — still failing; Hikari pool empty, requests queueing
Root cause
MySQL container restarted in place (no redeploy — last MySQL deploy was 2026-05-22). After restart, the MySQL service is alive but
CPU 0% = no queries reaching it. api-server fails at the socket connect layer (Connect timed out, before auth), meaning it cannot
reach MySQL over Railway private networking — almost certainly connecting to a stale internal address from before the restart.
HikariCP retries the dead address forever, so total=0 never recovers on its own.
"Forcing close of thread" warnings = NOT the cause
Those [MY-010909] Forcing close of thread warnings are normal MySQL shutdown noise. Same pattern appeared on the 2026-05-22 restart.
Benign.
Attachments
0 Replies