Production MySQL down on Railway. Service restart hung 10+ min on "Mounting volume". DB unreachable.
vonsa
PROOP

18 days ago

Hi,

ProjectId: d9ac9779-d4c6-45b2-83e6-ee4a5c70b424

Service: MySQL 8.4.3, Pro plan, 50GB volume

Started: ~11:30 UTC, 4 May 2026

We started experiencing issues with the DB today:

All INSERT/UPDATE queries hung 30s–1min+ ("waiting for handler commit")

GET/SELECT queries worked normally

SHOW ENGINE INNODB STATUS showed only 0.66 fsyncs/sec and 5 writes/sec

Current state (I attempted restart twice through the UI):

MySQL shutdown completed at 11:49:05 UTC

After first restart, nothing mounting. After second restart, deploy log stuck on "Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/..." for 10+ minutes

DB unreachable, production app down

Email support 48h isn't viable for this prod outage

Changes I made today (in order):

SET PERSIST max_connections = 400 (was 151)

SET PERSIST innodb_buffer_pool_size = 4GB

SET GLOBAL sync_binlog = 100 (was 1)

SET GLOBAL innodb_flush_log_at_trx_commit = 2 (was 1)

Triggered service Restart at 11:49 UTC

8 Replies

vonsa
PROOP

18 days ago

We have to resolve this ASAP. I'm already looking into migrating to an alternative DB solution. Not sure why 48 hour response time for a prod issue? I don't think that's viable for any professional usage.


laurentcompania
PRO

18 days ago

Same exact issue here, started 05:18 UTC today on a MySQL service in europe-west4 — waiting for handler commit blocking all UPDATEs, SELECTs fine. My app went down too.

Restart got stuck on Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/... for ~30 min before unblocking on its own (probably Railway intervened on the host). Then InnoDB crash recovery took another 12 min before MySQL accepted connections again. So total ~45 min of downtime, but it did come back without data loss.

Two services with identical symptoms on the same day → this looks like a host-level storage incident, not anything we did wrong. Hang in there, and definitely push Railway to confirm and migrate you to a healthy host.


lonasozo
PRO

18 days ago

Yea, same here, unable to mount the mysql database!


marcellov7
PRO

18 days ago

We're having the same problem here. He's been trying to mount a volume for 3 hours without success.

No response from support regarding something this serious!

We don't know what to do, and we're under pressure from our customers.


iyimiyiz-admin
PRO

18 days ago

same here.


vonsa
PROOP

18 days ago

Hi all,

This worked for me:

1. Create a new backup

2. Restore the backup

After creating a new backup, I also created a new volume, not sure if that matters. I didn't use it.

And as it seems to be a bit broken, I hope all data was backed up correctly.


mativm02
PRO

18 days ago

I'm also having some issues with two different services that connect to two different Redis instances (getting some timeouts, started 4 hours ago)


18 days ago

Team has been made aware and incident has been called: https://status.railway.com/incident/QK1OHYXB


Welcome!

Sign in to your Railway account to join the conversation.

Loading...