Re: [Observium] high CPU / slow response

I had same issue with syslog... events. Maybe a an auto-clean withing a configured delay in the config.php should be done ?

----- Mail original -----

De: "Paul Meys" paul@milla-it.be À: "Observium Network Observation System" observium@observium.org Envoyé: Mardi 15 Juillet 2014 10:28:58 Objet: Re: [Observium] high CPU / slow response

​Hi Guys,

to give an update,

I tried to tune the machine, no result. added cpu, ram, ...

What really helped: I cleaned the event-log ( more the 8 000 000 items in there .. )

Now the ​site is really responsive ..

So, I guess some of you have more than 8 000 000 events ?

Regards,

Paul

Van: observium observium-bounces@observium.org namens Matthew A Harper Matthew.A.Harper@tmr.qld.gov.au Verzonden: woensdag 18 juni 2014 1:07 Aan: Observium Network Observation System Onderwerp: Re: [Observium] high CPU / slow response

Hi Paul,

My ibdata1 file is 1,948 MB.

Below are my settings in my.cnf:

key_buffer = 16M

max_allowed_packet = 16M

thread_stack = 192K

thread_cache_size = 8

myisam-recover = BACKUP

table_cache = 166

tmp_table_size = 64M

max_heap_table_size = 64M

innodb_buffer_pool_size = 350M

query_cache_limit = 4M

query_cache_size = 48M

Average response times:

Front page – 1.455s (Summary, Map, Syslog, Event log)

Ports page – 2.323s (32362 ports in one query)

Devices page – 1.739s (822 devices)

Kind regards,

Matt Harper Network Analyst | ICT Operations Information Technology Branch | Department of Transport and Main Roads

Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006 PO Box 673 | Fortitude Valley Qld 4006 P: (07) 30661125 | F: (07) 30662044 E: matthew.a.harper@tmr.qld.gov.au W: www.tmr.qld.gov.au

From: observium [mailto:observium-bounces@observium.org] On Behalf Of Paul Meys Sent: Tuesday, 17 June 2014 6:16 PM To: Observium Network Observation System Subject: Re: [Observium] high CPU / slow response

Hi All,

thanks already for the help.

The stuff I already did:

• apparently it was running on a 32bit kernel, move it to a new 64bit

• changed to 4 cores .. no real result

• disabled the unused-polling

It seems that the cpu-usage of mysqld goes really high when logging in to show to startpage with the map ..

when going directly to the device the response-time is "okay" but not great.

When running mysqltuner:

innodb_buffer_pool_size​ should be greater than 23G ?

How big is your ibdata1 ?

Thanks,

Have a good day ..

Paul

my top:

top - 10:08:10 up 13:46, 2 users, load average: 1.28, 0.90, 0.56

Tasks: 169 total, 3 running, 166 sleeping, 0 stopped, 0 zombie

%Cpu(s): 30.4 us, 1.8 sy, 0.0 ni, 67.5 id, 0.1 wa, 0.0 hi, 0.3 si, 0.0 st

KiB Mem: 16474156 total, 16241936 used, 232220 free, 225928 buffers

KiB Swap: 10644476 total, 25912 used, 10618564 free, 11002996 cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

2755 mysql 20 0 5641m 4.3g 7888 S 114.8 27.1 23:33.94 mysqld

Van: observium < observium-bounces@observium.org > namens Matthew A Harper < Matthew.A.Harper@tmr.qld.gov.au > Verzonden: dinsdag 17 juni 2014 0:03 Aan: Observium Network Observation System Onderwerp: Re: [Observium] high CPU / slow response

Hi Paul,

Mark has raised some good points. Check your TOP output as suggested but based on my specs I’d say you need more CPU power.

To give you an idea of scaling and more specifically CPU usage - I’m running the following specs and it runs fine:

Debian 7 on ESX

8x E7-4870 cores @ 84% (observium seems to benefit from faster clock speeds)

8GB ram (only using 2GB)

SAN Storage (using 82GB of 250GB)

820 devices, 33000 ports, 5500 sensors, 10 polling instances, 6 alert checkers.

With the number of ports you’re running I’d say you need more CPU power to handle all the port data. You could try as Martin suggested and disable any un-used polling. Specifically FDB, ARP and CEF (if used).

Further to this you should look into tuning MySQL to perform better with the types of queries its dealing with. This script is very helpful: https://github.com/major/MySQLTuner-perl/blob/master/mysqltuner.pl just follow the instructions in the output.

Hope this helps.

Kind regards,

Matt Harper Network Analyst | ICT Operations Information Technology Branch | Department of Transport and Main Roads

Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006 PO Box 673 | Fortitude Valley Qld 4006 P: (07) 30661125 | F: (07) 30662044 E: matthew.a.harper@tmr.qld.gov.au W: www.tmr.qld.gov.au

From: observium [ mailto:observium-bounces@observium.org ] On Behalf Of Martin Smith Sent: Tuesday, 17 June 2014 12:43 AM To: observium@observium.org Subject: Re: [Observium] high CPU / slow response

Hello,

We are running on a VM as well with 5 cores, 6GB of RAM and SAN storage. We currently have ~250 devices with 7100 ports and the system works fine.

First of all check “top” check to see if your database is waiting on slow disks with %wa.

Cpu(s): 14.9%us, 6.9%sy, 0.0%ni, 76.5%id, 1.2%wa , 0.0%hi, 0.6%si, 0.0%st

We have also disabled a couple features that seem to cause a much longer polling time.

/opt/observium/config.php

// Disable EtherLike-MIB

$config['enable_ports_etherlike'] = 0; # Enable Polling EtherLike-MIB (doubles interface processing time)

// Disable FDB table polling

$config['poller_modules']['fdb-table'] = 0;

// Disable ARP table polling

$config['discovery_modules']['arp-table'] = 0;

Finally we use 5 threads of the poller-wrapper

/etc/cron.d/observium

*/5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1

Hopefully this helps,

Martin Smith | Network Analyst | Netgain 720 West Saint Germain Street | St. Cloud | MN | 56301

Phone: 320-257-6607 | 877.797.4700 x170

From: observium [ mailto:observium-bounces@observium.org ] On Behalf Of Paul Meys Sent: Monday, June 16, 2014 5:07 AM To: observium@observium.org Subject: [Observium] high CPU / slow response

Hi,

I'm running the latest version, but it is extremely slow.

Even to load the login-page, it takes a very long time.

I always see mysqld using almost all CPU-power.

the server is running in vmware, os = debian.

Currently I have one v-core assigned with 16G of ram. ( already tried with 4 cores.. )

In Observium I have 75 devices with a total of 7500 ports.

Please advice ;-)

Thank you,

Have a good day,

Paul

---

WARNING: This email (including any attachments) may contain legally privileged, confidential or private information and may be protected by copyright. You may only use it if you are the person(s) it was intended to be sent to and if you use it in an authorised way. No one is allowed to use, review, alter, transmit, disclose, distribute, print or copy this email without appropriate authority.

If this email was not intended for you and was sent to you by mistake, please telephone or email me immediately, destroy any hardcopies of this email and delete it and any copies of it from your computer system. Any right which the sender may have under copyright law, and any legal privilege and confidentiality attached to this email is not waived or destroyed by that mistake.

It is your responsibility to ensure that this email does not contain and is not affected by computer viruses, defects or interference by third parties or replication problems (including incompatibility with your computer system).

Opinions contained in this email do not necessarily reflect the opinions of the Department of Transport and Main Roads, or endorsed organisations utilising the same infrastructure.

---

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium