My Prod is down: MySQL volume ran out of disk space
ricardomacario
PROOP

2 years ago

Hi Railway team,

I'm in full mode crisis. My MySQL volume is full, and it doesn't let me increase the field is set at 50GB.

The project is: f5925531-2de0-4da9-8a6d-15b5ba712ebe

I need to increase it now! I'm down in prod.

Other than that, it's weird that it got that big because my MySQL DB was 2-3GB tops. So maybe it's logs getting accumulated?

Please help.

12 Replies

ricardomacario
PROOP

2 years ago

Ok I was able to fix this by purging MySQL binary logs. They were taking 30+ GB.

I recommend adding alerts when the volume is going to be full.

brody
EMPLOYEE

2 years ago

going forward let's disable binary logging, set this as your start command for the mysql database -

docker-entrypoint.sh mysqld --innodb-use-native-aio=0 --disable-log-bin --performance_schema=0
ricardomacario
PROOP

2 years ago

@Brody where can I set that start command?

ricardomacario
PROOP

2 years ago

1228754853887738000

ricardomacario
PROOP

2 years ago

there right? Will there be any side effects of turning off binary logging?

ricardomacario
PROOP

2 years ago

Also, now that I have you here! Do you have any tips for lowering my MySQL memory consumption?

ricardomacario
PROOP

2 years ago

1228755184310685700

ricardomacario
PROOP

2 years ago

I have about 2GB worth of data on my DB but it takes 4-7 GB of ram.

brody
EMPLOYEE

2 years ago

Will there be any side effects of turning off binary logging?

brody
EMPLOYEE

2 years ago

Will there be any side effects of turning off binary logging?
none that i know of, all newly deployed mysql databases come with it disabled.

Do you have any tips for lowering my MySQL memory consumption?
--performance_schema=0 will cut down on a tiny amount of the memory

ricardomacario
PROOP

2 years ago

perfect thank you so much. Super appreciated!

The situation was tense because:

  1. I noticed I had ran out of memory because prod was down

  2. I went to increase the size but its capped at 50gb

  3. Then I had to figure out what was taking space and clear it

brody
EMPLOYEE

2 years ago

I understand, and if you hadn't already solved it by the time I woke up I would have gotten a team member involved immediately to bump that limit since you're pro

Loading...