That's a pretty small I/O load especially for SSD's. When doing an iostat on the lun what are you showing for iops? Normally when I see large i/o wait it's the back-end subsystem. Since you're indicating a san here do you have/can you get any data from your san/storage admin on your luns?
Not knowing your installation size, we also had flash based san storage here but I moved our observium deployment to local ssd's (raid 10) which I clipped to 50% of capacity for better garbage collection instead due to over saturation of the san's i/o channels for that tier of storage. This was for ~600 devices ~200,000+ ports. It just wasn't cost effective to use the san for the iops needs.
On 7/14/22 08:27, Lars Joergensen via observium wrote:
Hey list
We keep adding devices to Observium, and it seems we’re getting close to some tipping point for MySQL performance.
Looking at output from “top” we frequently see high iowait:
Looking at iotop, it seems that MySQL is the aggressor here:
We figured syslog was the culprit at first, and tried disabling that for a while to see if stuff died down. It did not.
Been playing a little with mytop, and even though we have a QPS around 780, it’s not possible for me to nail down what queries are hogging the IO on the server. I googled around and did some changes to the InnoDB, and though I saw improvements, they were not as massive as I had hoped. This is what I currently have changed from defaults:
[mysqld]
# This is a little too much for the installed 8G in the machine and will be adjusted at a later time. Just trying stuff out now.
innodb_buffer_pool_size=6G
# We have fast SSD drives on vSAN
innodb_io_capacity=1000
# Don't flush file write so often, be more relaxed.
innodb-flush-method=O_DIRECT_NO_FSYNC
So I’m kind of stumped here. It’s not killing Observium totally just yet, but we have experienced some outages this week and will probably see more as we keep adding devices.
Any recommendations will be much appreciated.
Lars
Disclaimer: This e-mail, including any attachments, is for the intended recipient only. If you have received this e-mail by mistake please notify the sender immediately by return e-mail and delete this e-mail and any attachments, without opening the attachments, from your system. Access, disclosure, copying, distribution or reliance on any part of this e-mail by anyone else is prohibited. This e-mail is confidential and may be legally privileged. Chr. Hansen does not represent and/or warrant that the information sent and/or received by or with this e-mail is correct and does not accept any liability for damages related thereto. https://www.chr-hansen.com/en/legal-notice
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
