**Title: MySQL 9 security patch caused InnoDB startup failure** Hello Railway team, My MySQL service started crashing after Railway automatically upgraded it to the latest MySQL 9 release as part of the scheduled security patch. **Project:** sparkling-education **Environment:** production **Service:** MySQL I did not make any changes to the database or application for about a month before this issue occurred. Railway notified me that MySQL was scheduled to be patched for multiple CVEs and moved to the latest MySQL 9 release. After the upgrade, the MySQL deployment started crashing. The current MySQL version shown in the logs is: ```text MySQL 9.7.2 ``` The deployment repeatedly fails with the following InnoDB error: ```text [ERROR] [MY-012888] [InnoDB] Cannot resize redo log file ./#innodb_redo/#ib_redo106_tmp to 3 MB (Failed to set size) [Warning] [MY-012638] [InnoDB] Retry attempts for writing partial data failed. ``` This error repeats many times and then MySQL shuts down: ```text [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 9.7.2) MySQL Community Server - GPL. ``` I tried restarting the deployment, but it crashes again with the same error. The attached MySQL volume is currently at approximately **77% capacity**, so it is not at 100% disk usage. Railway's security patch notification also stated that: * A volume snapshot would be taken before patching. * The snapshot would be kept for 30 days. * The MySQL service would be moved to the latest MySQL 9 release. * The move cannot be undone by simply redeploying. * The snapshot can be restored to roll back. Could you please investigate whether there is an issue with the MySQL 9 upgrade, the volume/filesystem, or the InnoDB redo log initialization? **Most importantly, please confirm that my existing data on the volume is safe and advise whether I should restore the pre-patch volume snapshot.** I do not want to delete, recreate, or reinitialize the MySQL volume because the database contains production data. Thank you.

Restarting the deployment produces the same failure.

m97161

HOBBYOP

a day ago

Title: MySQL 9 security patch caused InnoDB startup failure

Hello Railway team,

My MySQL service started crashing after Railway automatically upgraded it to the latest MySQL 9 release as part of the scheduled security patch.

Project: sparkling-education

Environment: production

Service: MySQL

I did not make any changes to the database or application for about a month before this issue occurred.

Railway notified me that MySQL was scheduled to be patched for multiple CVEs and moved to the latest MySQL 9 release.

After the upgrade, the MySQL deployment started crashing.

The current MySQL version shown in the logs is:

MySQL 9.7.2

The deployment repeatedly fails with the following InnoDB error:

[ERROR] [MY-012888] [InnoDB] Cannot resize redo log file
./#innodb_redo/#ib_redo106_tmp to 3 MB (Failed to set size)

[Warning] [MY-012638] [InnoDB] Retry attempts for writing partial data failed.

This error repeats many times and then MySQL shuts down:

[System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete
(mysqld 9.7.2) MySQL Community Server - GPL.

I tried restarting the deployment, but it crashes again with the same error.

The attached MySQL volume is currently at approximately 77% capacity, so it is not at 100% disk usage.

Railway's security patch notification also stated that:

A volume snapshot would be taken before patching.
The snapshot would be kept for 30 days.
The MySQL service would be moved to the latest MySQL 9 release.
The move cannot be undone by simply redeploying.
The snapshot can be restored to roll back.

Could you please investigate whether there is an issue with the MySQL 9 upgrade, the volume/filesystem, or the InnoDB redo log initialization?

Most importantly, please confirm that my existing data on the volume is safe and advise whether I should restore the pre-patch volume snapshot.

I do not want to delete, recreate, or reinitialize the MySQL volume because the database contains production data.

Thank you.

Awaiting User Response

3 Replies

Status changed to Awaiting Railway Response Railway • 1 day ago

brody

EMPLOYEE

a day ago

Your MySQL volume is attached and holds your production data, so nothing has been lost. The security patch moved your service from a MySQL 9.x Innovation release to MySQL 9.7.2 LTS, and a pre-patch volume snapshot was taken before the redeploy, held in the service's Backups tab for 30 days. Restoring that snapshot is the designed rollback path and would return your volume to the exact state before the patch, but on the Hobby plan the restore is gated behind Pro ($20/mo). Upgrading to Pro unlocks self-serve restore from the Backups tab.

Status changed to Awaiting User Response brody • 1 day ago

brody

m97161

HOBBYOP

a day ago

Thank you for the detailed explanation.

I understand that my production data is still safely stored on the attached volume and that a pre-patch snapshot is available for 30 days.

This is not an urgent issue for me, so I would prefer to wait for further investigation rather than upgrade to Pro just to restore the snapshot at this time.

For now, I will not delete or reinitialize the volume.

Please let me know if there is any recommended action or if the MySQL 9.7.2 startup issue can be resolved without restoring the snapshot.

Thank you for your help.

Status changed to Awaiting Railway Response Railway • 1 day ago

angelo-railway

EMPLOYEE

3 hours ago

Your volume is in a ready state with 382 MB of production data at /var/lib/mysql, so your data remains intact. The pre-patch volume snapshot taken before the redeploy is visible in the service's Backups tab for 30 days, and restoring it is the rollback path for this. On Hobby, restoring requires upgrading to Pro ($20/mo), which is pro-rated against your current plan payment.

Status changed to Awaiting User Response Railway • about 4 hours ago

Welcome!