a month ago
Hello Railway Support,
I am experiencing recurring outages on a managed MySQL service and would like help investigating a possible storage-related issue.
Symptoms
My application works normally for a period of time, but eventually MySQL becomes unavailable and application workers start failing with:
SQLSTATE[HY000] [2002] Connection refused
The failures occur when attempting normal reads/writes against the database.
Database size
I inspected all application tables using information_schema and the total size is only approximately 45 MB:
- pixels: ~30 MB
-
- bot_faction_plan_pixels: ~14.6 MB
-
- all remaining tables combined: ~1 MB
The largest tables are therefore relatively small.
Railway volume usage
Despite the small amount of actual table data, the Railway volume usage repeatedly reaches around 92-97% of a 1 GB volume.
When the volume approaches this level, MySQL becomes unstable and application workers start failing.
MySQL logs
The MySQL logs contain repeated InnoDB errors such as:
- Cannot resize redo log file
-
- Failed to set size
-
- Retry attempts for writing partial data failed
Example messages:
[InnoDB] Cannot resize redo log file ./#innodb_redo/#ib_redoXXXX_tmp to 3 MB (Failed to set size)
1048576 bytes should have been written. Only 966656 bytes written. Retrying for the remaining bytes.
These errors repeat many times before MySQL becomes unavailable.
Additional information
- MySQL version: 9.4
-
- log_bin = OFF
-
- Application tables have already been cleaned aggressively:
-
- bot_events = 0 rows
- sessions = 0 rows
-
- cache = 0 rows
- The issue persists after deleting large amounts of data.
Questions
Could you please help verify:
- What is actually consuming the volume space?
- Whether InnoDB system files, redo logs, temporary files, or fragmentation are responsible for the reported usage.
- Whether the volume may be in a corrupted or unreclaimed state.
- Whether rebuilding/recreating the volume would reclaim space.
- Whether there are any known issues affecting MySQL storage volumes at the moment.
From the application side, the total database content appears far smaller than the reported volume usage, so I suspect the issue may be related to MySQL internal files or the underlying storage volume.
Thank you.
Attachments
1 Replies
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
Even though your table data is only 45MB, MySQL requires a lot more overhead that a 1GB volume isn't really enough when relying on the default configurations. At this point, rebuilding the database from a logical dump is the best way to reclaim space. But you still need to increase your volume size, so you don't run into this problem again.