Hey list

 

We keep adding devices to Observium, and it seems we’re getting close to some tipping point for MySQL performance.

 

Looking at output from “top” we frequently see high iowait:

 

 

Looking at iotop, it seems that MySQL is the aggressor here:

 

 

We figured syslog was the culprit at first, and tried disabling that for a while to see if stuff died down. It did not.

 

Been playing a little with mytop, and even though we have a QPS around 780, it’s not possible for me to nail down what queries are hogging the IO on the server. I googled around and did some changes to the InnoDB, and though I saw improvements, they were not as massive as I had hoped. This is what I currently have changed from defaults:

 

[mysqld]

# This is a little too much for the installed 8G in the machine and will be adjusted at a later time. Just trying stuff out now.

innodb_buffer_pool_size=6G

 

# We have fast SSD drives on vSAN

innodb_io_capacity=1000

 

# Don't flush file write so often, be more relaxed.

innodb-flush-method=O_DIRECT_NO_FSYNC

 

So I’m kind of stumped here. It’s not killing Observium totally just yet, but we have experienced some outages this week and will probably see more as we keep adding devices.

 

Any recommendations will be much appreciated.

 

 

Lars