6 days ago
Let me be upfront that I'm a junior developer and I'm personally struggling to understand the issue. On your dashboard, I see the database constantly attempting to connect and failing. I did not realize the Hobby plan does not come with any backup support and I don't think I have any. I dumped the logs I could see into Claude and it has produced everything you see below:
My MySQL service has been unreachable for five days and has not recovered on its own. I'm requesting a host-level restart or internal recovery of the mysqld process.
Production data must be preserved. Please do not recreate, reset, or delete the service or its volume without talking to me first.
Project: MCHREC
Project ID: 31b9ee92-06cc-44c1-b7ac-f0e1ce21e2f1
Environment: production (0dcd0713-97d2-4d1c-ba87-0130dc18a9ae)
Service: MySQL (d8f00d52-9a11-411a-90c4-a6a668743307)
Deployment: 4aab8565-4b27-4f0c-b3f7-753d7e2bce18
Replica: 7271c7aa-7854-4f53-916c-455d1104ae31
Volume: vol_jmhdfjlkevtlow48
CURRENT STATE
The deployment shows Online and Required Variables passes, but the Database tab's own connection check fails with "Connection lost: The server closed the connection." My application gets the same result — PROTOCOL_CONNECTION_LOST on every query. Since Railway's own in-dashboard check fails using credentials Railway generated, I don't believe this is a client-side or configuration problem.
WHAT HAPPENED
At approximately 03:11 UTC on 2026-08-10, the InnoDB main thread (139862422304512, state=enforcing dict cache limit) blocked waiting on the FIL_SHARD mutex at fil0fil.cc:854. The mutex was held by thread 139862391842560, which never released it and appears nowhere else in the InnoDB monitor output. The wait counter climbed continuously for about 16 minutes, then:
2026-08-10T03:27:33.619520Z 0 [ERROR] [MY-012872] [InnoDB] [FATAL] Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.
2026-08-10T03:27:33.619988Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: srv0srv.cc:1878:ib::fatal triggered thread 139862439089920
2026-08-10T03:27:33Z UTC - mysqld got signal 6 ;
WHY I BELIEVE THIS IS INFRASTRUCTURE-SIDE
- Not memory: buffer pool 8192 pages with 1501 free, "Total large memory allocated 0", and signal 6 (SIGABRT — InnoDB's deliberate abort) rather than SIGKILL from an OOM kill.
- Not I/O saturation: pending normal aio reads and writes all zero, pending fsyncs zero, 0.00 reads/s and 0.00 writes/s throughout the entire window.
- Not application load: 0 queries inside InnoDB, 0 in queue, 0 read views open, history list length 0.
- The stuck thread held an InnoDB file-layer mutex with no InnoDB-visible work attributed to it, which is consistent with a blocked syscall against the mounted volume rather than anything happening inside the database.
- The crash backtrace lines carry a timestamp of 03:27:33 but did not reach the log pipeline until 04:21:28 — a 54-minute gap, suggesting the container itself was wedged rather than just mysqld.
- A volume remount was logged at 04:22:38 ("Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/7271c7aa-7854-4f53-916c-455d1104ae31/vol_jmhdfjlkevtlow48"), but the service still does not accept connections.
DATA STATE AT CRASH
Log sequence number, last checkpoint, and pages-flushed-up-to were all identical (441398914) with "Modified db pages 0", so everything was durably written before the abort. I expect the data to be intact with nothing to replay on recovery.
[BACKUP STATUS — fill this in from the Backups tab before sending]
WHAT I'M ASKING FOR
- Restart or recover the mysqld process at the host level, preserving the existing volume and data.
- If the volume or filesystem is in a bad state, please tell me what you're seeing rather than recreating the service.
- This service is running MySQL 8.3, which is an end-of-life innovation release. Once it's back up, I'd like guidance on moving to 8.4 LTS.
1 Replies
6 days ago
Your MySQL service's volume is attached with data intact (~478 MB used), and the deployment status is SUCCESS, meaning the container is running but the mysqld process inside is not accepting connections after the InnoDB semaphore abort. You can restart it yourself from the deployment menu on your service page: click the three dots on the current deployment and select Restart, which restarts the process without a rebuild and should trigger InnoDB crash recovery against your existing data. If that does not bring it back, open the command palette (Cmd+K or Ctrl+K) and select "Redeploy source image" to re-pull a fresh MySQL image. Your volume and its data are preserved through both operations. Regarding MySQL 8.3 to 8.4 LTS and volume backups (which require Pro), those are good questions for a follow-up once the service is back online.
Status changed to Awaiting User Response Railway • 6 days ago