How big is the install? What is the hardware?
I usually recommend MySQL/Web/rrdcached is placed on a system with higher individual core speed for just this reason, mysql scales better with faster cores than more cores.
Perhaps it would be useful if we tracked MySQL time per-module.
I suspect a lot of the load might be updating very heavy indexes, but I'm not sure how to profile that.
Adam.
From: observium observium-bounces@observium.org On Behalf Of Lars Joergensen via observium Sent: 14 July 2022 14:27 To: Observium observium@observium.org Cc: Lars Joergensen DKLARJ@chr-hansen.com Subject: [Observium] MySQL IO Wait
Hey list
We keep adding devices to Observium, and it seems we're getting close to some tipping point for MySQL performance.
Looking at output from "top" we frequently see high iowait:
Looking at iotop, it seems that MySQL is the aggressor here:
We figured syslog was the culprit at first, and tried disabling that for a while to see if stuff died down. It did not.
Been playing a little with mytop, and even though we have a QPS around 780, it's not possible for me to nail down what queries are hogging the IO on the server. I googled around and did some changes to the InnoDB, and though I saw improvements, they were not as massive as I had hoped. This is what I currently have changed from defaults:
[mysqld]
# This is a little too much for the installed 8G in the machine and will be adjusted at a later time. Just trying stuff out now.
innodb_buffer_pool_size=6G
# We have fast SSD drives on vSAN
innodb_io_capacity=1000
# Don't flush file write so often, be more relaxed.
innodb-flush-method=O_DIRECT_NO_FSYNC
So I'm kind of stumped here. It's not killing Observium totally just yet, but we have experienced some outages this week and will probably see more as we keep adding devices.
Any recommendations will be much appreciated.
Lars
_____
Disclaimer: This e-mail, including any attachments, is for the intended recipient only. If you have received this e-mail by mistake please notify the sender immediately by return e-mail and delete this e-mail and any attachments, without opening the attachments, from your system. Access, disclosure, copying, distribution or reliance on any part of this e-mail by anyone else is prohibited. This e-mail is confidential and may be legally privileged. Chr. Hansen does not represent and/or warrant that the information sent and/or received by or with this e-mail is correct and does not accept any liability for damages related thereto. https://www.chr-hansen.com/en/legal-notice
