18 days ago
I started experiencing MySQL container issues started May 1st.
Pattern: occasional write blocking.
I created simple probe script that measures SELECT and UPDATE time each second and dumps innodb status if problems, and here is typical pattern of what I see:
Time Read Write
13:01:38 10.5 10.2
13:01:39 8.3 10.6
13:01:40 6.2 6.9
13:01:41 5.8 6.9
13:01:43 6.0 9.5
13:01:47 8.3 3009.2 WRITE_BLOCKED → /tmp/innodb_status_20260504_130147.txt
13:01:48 8.6 7.3
13:01:49 5.4 126.0
13:01:50 6.6 5.8
13:01:51 6.5 5.5
13:01:52 7.9 5.2
Innodb status shows that everyhing is fine (attached) - MySQL itself is not overloaded.
However, there is still 3+sec hang on trivial operation.
I optimized MySQL settings a lot, increased pool size and everything, but nothing worked.
As you may see, reads are always fast. Problem is in writes. On my side, I tried everything to fix problem.
It is strange that it started all of a sudden.
So I thought if there may be some sort of IO or CPU throttling that periodically hangs my database?
Attachments
4 Replies
18 days ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • 18 days ago
18 days ago
MySQL start command:
docker-entrypoint.sh mysqld --innodb-use-native-aio=0 --disable-log-bin --performance_schema=0 --innodb-buffer-pool-size=10G --innodb-buffer-pool-instances=8 --innodb-flush-log-at-trx-commit=2 --innodb-redo-log-capacity=4294967296 --innodb-write-io-threads=16 --innodb-page-cleaners=8 --innodb-adaptive-hash-index=OFF
10 days ago
Anyone?
4 days ago
try doing this , see if this works , the redo log are huge like 4 gb , sometimes this take time to flush dirty pages
--innodb-io-capacity=2000
--innodb-io-capacity-max=10000
4 days ago
Since the issue is isolated to intermittent write stalls and InnoDB shows no internal contention, the most effective mitigation is to reduce synchronous disk flush pressure and isolate the database from noisy host contention.
Try setting innodb_flush_log_at_trx_commit=2, lowering excessive fsync frequency, moving to a dedicated/higher-memory instance, reducing bursty concurrent writes, and if possible migrating the MySQL volume/container to a fresh Railway service/region since the behavior began suddenly and points to underlying IO/runtime degradation rather than schema or query design.