Hi Paul,
My ibdata1 file is 1,948 MB.
Below are my settings in my.cnf:
key_buffer = 16M max_allowed_packet = 16M thread_stack = 192K thread_cache_size = 8 myisam-recover = BACKUP table_cache = 166 tmp_table_size = 64M max_heap_table_size = 64M innodb_buffer_pool_size = 350M query_cache_limit = 4M query_cache_size = 48M
Average response times:
Front page – 1.455s (Summary, Map, Syslog, Event log) Ports page – 2.323s (32362 ports in one query) Devices page – 1.739s (822 devices)
Kind regards,
Matt Harper Network Analyst | ICT Operations Information Technology Branch | Department of Transport and Main Roads
Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006 PO Box 673 | Fortitude Valley Qld 4006 P: (07) 30661125 | F: (07) 30662044 E: matthew.a.harper@tmr.qld.gov.aumailto:matthew.a.harper@tmr.qld.gov.au W: www.tmr.qld.gov.auhttp://www.tmr.qld.gov.au/
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Paul Meys Sent: Tuesday, 17 June 2014 6:16 PM To: Observium Network Observation System Subject: Re: [Observium] high CPU / slow response
Hi All,
thanks already for the help.
The stuff I already did:
- apparently it was running on a 32bit kernel, move it to a new 64bit
- changed to 4 cores .. no real result
- disabled the unused-polling
It seems that the cpu-usage of mysqld goes really high when logging in to show to startpage with the map ..
when going directly to the device the response-time is "okay" but not great.
When running mysqltuner:
innodb_buffer_pool_size should be greater than 23G ?
How big is your ibdata1 ?
Thanks,
Have a good day ..
Paul
my top: top - 10:08:10 up 13:46, 2 users, load average: 1.28, 0.90, 0.56 Tasks: 169 total, 3 running, 166 sleeping, 0 stopped, 0 zombie %Cpu(s): 30.4 us, 1.8 sy, 0.0 ni, 67.5 id, 0.1 wa, 0.0 hi, 0.3 si, 0.0 st KiB Mem: 16474156 total, 16241936 used, 232220 free, 225928 buffers KiB Swap: 10644476 total, 25912 used, 10618564 free, 11002996 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2755 mysql 20 0 5641m 4.3g 7888 S 114.8 27.1 23:33.94 mysqld
________________________________ Van: observium <observium-bounces@observium.orgmailto:observium-bounces@observium.org> namens Matthew A Harper <Matthew.A.Harper@tmr.qld.gov.aumailto:Matthew.A.Harper@tmr.qld.gov.au> Verzonden: dinsdag 17 juni 2014 0:03 Aan: Observium Network Observation System Onderwerp: Re: [Observium] high CPU / slow response
Hi Paul,
Mark has raised some good points. Check your TOP output as suggested but based on my specs I’d say you need more CPU power.
To give you an idea of scaling and more specifically CPU usage - I’m running the following specs and it runs fine:
Debian 7 on ESX 8x E7-4870 cores @ 84% (observium seems to benefit from faster clock speeds) 8GB ram (only using 2GB) SAN Storage (using 82GB of 250GB)
820 devices, 33000 ports, 5500 sensors, 10 polling instances, 6 alert checkers.
With the number of ports you’re running I’d say you need more CPU power to handle all the port data. You could try as Martin suggested and disable any un-used polling. Specifically FDB, ARP and CEF (if used).
Further to this you should look into tuning MySQL to perform better with the types of queries its dealing with. This script is very helpful: https://github.com/major/MySQLTuner-perl/blob/master/mysqltuner.pl just follow the instructions in the output.
Hope this helps.
Kind regards,
Matt Harper Network Analyst | ICT Operations Information Technology Branch | Department of Transport and Main Roads
Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006 PO Box 673 | Fortitude Valley Qld 4006 P: (07) 30661125 | F: (07) 30662044 E: matthew.a.harper@tmr.qld.gov.aumailto:matthew.a.harper@tmr.qld.gov.au W: www.tmr.qld.gov.auhttp://www.tmr.qld.gov.au/
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Martin Smith Sent: Tuesday, 17 June 2014 12:43 AM To: observium@observium.orgmailto:observium@observium.org Subject: Re: [Observium] high CPU / slow response
Hello,
We are running on a VM as well with 5 cores, 6GB of RAM and SAN storage. We currently have ~250 devices with 7100 ports and the system works fine.
First of all check “top” check to see if your database is waiting on slow disks with %wa. Cpu(s): 14.9%us, 6.9%sy, 0.0%ni, 76.5%id, 1.2%wa, 0.0%hi, 0.6%si, 0.0%st
We have also disabled a couple features that seem to cause a much longer polling time. /opt/observium/config.php
// Disable EtherLike-MIB $config['enable_ports_etherlike'] = 0; # Enable Polling EtherLike-MIB (doubles interface processing time)
// Disable FDB table polling $config['poller_modules']['fdb-table'] = 0;
// Disable ARP table polling $config['discovery_modules']['arp-table'] = 0;
Finally we use 5 threads of the poller-wrapper /etc/cron.d/observium */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Hopefully this helps,
Martin Smith | Network Analyst | Netgain 720 West Saint Germain Street | St. Cloud | MN | 56301 Phone: 320-257-6607 | 877.797.4700 x170
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Paul Meys Sent: Monday, June 16, 2014 5:07 AM To: observium@observium.orgmailto:observium@observium.org Subject: [Observium] high CPU / slow response
Hi,
I'm running the latest version, but it is extremely slow.
Even to load the login-page, it takes a very long time.
I always see mysqld using almost all CPU-power.
the server is running in vmware, os = debian.
Currently I have one v-core assigned with 16G of ram. ( already tried with 4 cores.. )
In Observium I have 75 devices with a total of 7500 ports.
Please advice ;-)
Thank you,
Have a good day,
Paul
*********************************************************************** WARNING: This email (including any attachments) may contain legally privileged, confidential or private information and may be protected by copyright. You may only use it if you are the person(s) it was intended to be sent to and if you use it in an authorised way. No one is allowed to use, review, alter, transmit, disclose, distribute, print or copy this email without appropriate authority.
If this email was not intended for you and was sent to you by mistake, please telephone or email me immediately, destroy any hardcopies of this email and delete it and any copies of it from your computer system. Any right which the sender may have under copyright law, and any legal privilege and confidentiality attached to this email is not waived or destroyed by that mistake.
It is your responsibility to ensure that this email does not contain and is not affected by computer viruses, defects or interference by third parties or replication problems (including incompatibility with your computer system).
Opinions contained in this email do not necessarily reflect the opinions of the Department of Transport and Main Roads, or endorsed organisations utilising the same infrastructure. ***********************************************************************