My MySQL service has been unreachable since today's US West incident (RL8FRJE6). The status page now shows "Fully Operational", but my database never recovered, so I don't think this project came back with the rest of the region.

PROJECT DETAILS

Project: sincere-expression (c361a60d-b21a-4aa7-adb7-64175498f54e)

Environment: production (0f8441b6-6e77-4350-8f36-155bbedf45e9)

MySQL service: 2ca0cd14-7071-4981-926a-807cf9b29b24 (mysql:9.4, US West), deployment 374a0749

Volume: mysql-volume -> vol_4pi6943khquovy5v

App service: 7fe4e22c-0160-4dad-be1f-4d25b40723b5

TIMELINE (times GMT+8)

1. 11:33 - InnoDB started reporting long semaphore waits. Multiple threads blocked 305-886 seconds on the same LOG_WRITER mutex:

--Thread 139943866226240 has waited at log0write.cc line 971 for 886 seconds the semaphore:

Mutex at 0x7f47b7c2e6c0, Mutex LOG_WRITER created log0log.cc:625, locked by 139943867283008

This looks like log writes to the volume stopped completing.

1. 11:34:38 - InnoDB's long-semaphore-wait watchdog aborted the server:

InnoDB: about forcing recovery.

2026-08-10T03:34:38Z UTC - mysqld got signal 6 ;

Thread pointer: 0x0

BuildID[sha1]=c4e0cd59af2a4ab7eb21afe2a4c2c6c7e745a603

1. 12:20:29 - stack trace written, followed by the standard MySQL crash notice.

2. 12:22:14 - container restarted and logged:

Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/a228d6ae-2d8b-41a5-ac14-690c0b289cbd/vol_4pi6943khquovy5v

After this line there is NO further log output at all. mysqld never prints its usual startup or InnoDB recovery messages, so it appears to block immediately after mounting the volume.

CURRENT SYMPTOMS

• Service shows "Online" in the dashboard, but no MySQL output since 12:22:14.
• • Railway's own Database > Data tab cannot connect either: "We are unable to connect to the database via SSH - Connection lost: The server closed the connection."
• • My app connects over private networking (mysql.railway.internal:3306) and every query fails with ETIMEDOUT. Requests consistently take ~10.2s, matching the mysql2 client's 10s connect timeout, i.e. nothing is listening on 3306.
• • I restarted the APP service once (no effect, as expected). I have deliberately NOT restarted the MySQL service, to avoid interrupting any InnoDB crash recovery in progress.

WHAT I THINK IS WRONG

Volume vol_4pi6943khquovy5v appears to be unresponsive at the storage layer. The original crash was InnoDB protecting itself after writes stopped completing, and the restarted container now hangs at exactly the same point - right after mounting that volume.

WHAT I'M ASKING FOR

1. Could you check the health of volume vol_4pi6943khquovy5v and its underlying host? I suspect it is still hung even though the regional incident is marked resolved.

2. If the volume is healthy and the data needs InnoDB recovery, please advise the safest path. I would rather not set innodb_force_recovery blindly.

3. IMPORTANT: this is a production system for a small manufacturing company and there are no backups (backups/PITR require the Pro plan). Please prioritise data preservation over restoring service quickly - please do not reinitialise or wipe the volume.

Happy to provide more logs or run diagnostics. Thanks for your help.