Thanks Adam. Our pollerlog pages are below. Should we still try to disable fdb-table as suggested? If so, any docs we can follow to disable it?
Any other thoughts or suggestions? Thanks.
- Gordon
[cid:image001.png@01D46C97.0B070D20]
[cid:image002.png@01D46C97.0B070D20] [cid:image003.png@01D46C97.0B070D20]
From: observium observium-bounces@observium.org on behalf of Adam Armstrong via observium observium@observium.org Reply-To: Observium observium@observium.org Date: Tuesday, October 23, 2018 at 12:50 PM To: Chris Neam via observium observium@observium.org Cc: Adam Armstrong adama@memetic.org Subject: Re: [Observium] Performance Issue - High CPU with 'mysqld'?
Hi,
You probably (definitely) need more cores and/or more I/O throughput capacity.
It's likely that you have multiple overlapping poller_wrapper processes running because they can't finish quickly enough.
what do the graphs on this page look like?
http://dev.observium.org/pollerlog/
and what are the slowest devices on this :
http://dev.observium.org/pollerlog/view=devices/
You may be able to disable a module or two you don't care about that's taking time :
http://dev.observium.org/pollerlog/view=modules/
fdb-table is usually a good candidate.
adam.
On 2018-10-23 19:54:32, Gordon Cheng (gocheng) via observium observium@observium.org wrote: Hi all,
I have a new Observium setup (verion 18.9.9428 (9th September 2018)) with the following VM spec:
Cores 4 Memory 64GB HD 500GB CentOS 7 PHP 7.0.31 MySQL 5.5.60-MariaDB
and with the following network devices to be monitored:
Devices - 634 (total), 597 (up), 37 (down) Ports - 58,032 (total), 27,664 (up), 19,967 (down) Sensors - 47,639 (total), 43,243 (ok), 4,396 (down) Statuses - 5,880 (total), 5,854 (ok), 10 (alert)
---
Issue is that we are frequently seeing the following error when connecting on GUI/CLI, with slow response time on GUI/CLI:
DB Error 1040: Too many connections
And we are seeing gaps (broken graphs) like the following:
[cid:image001.png@01D46AC5.2F7A6900]
We have tried to bump up the number of 'max_connections' from original 151 to 500 and then 200, but it didn't help:
sjc-observium-1:/etc# grep max_conn my.cnf max_connections=200 sjc-observium-1:/etc#
And it's observed that the CPU load is heavy with 'mysqld':
top - 09:36:17 up 35 days, 18:54, 2 users, load average: 74.18, 89.49, 82.67
Tasks: 665 total, 79 running, 586 sleeping, 0 stopped, 0 zombie
%Cpu(s): 62.9 us, 36.9 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem : 65810376 total, 2511836 free, 5989292 used, 57309248 buff/cache
KiB Swap: 4193020 total, 4141016 free, 52004 used. 59172312 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5152 mysql 20 0 2749132 664216 9164 S 20.7 1.0 4058:17 mysqld
31879 root 20 0 419676 27564 8264 R 4.2 0.0 0:00.13 php
31769 root 20 0 417212 25532 8336 S 3.6 0.0 0:00.12 php
31933 root 20 0 416868 25116 8148 S 3.2 0.0 0:00.10 php
31935 root 20 0 416736 24700 7832 S 3.2 0.0 0:00.10 php
<snip>
My 'config.php' is pretty much with default values.
---
I'm new to Observium. It'd be greatly appreciated if someone can shed some light how it can be resolved.
Thanks.
- Gordon
