I had same issue with syslog... events.
Maybe a an auto-clean withing a configured delay in the config.php should be done ?

De: "Paul Meys" <paul@milla-it.be>
À: "Observium Network Observation System" <observium@observium.org>
Envoyé: Mardi 15 Juillet 2014 10:28:58
Objet: Re: [Observium] high CPU / slow response

Hi Guys,

to give an update,

I tried to tune the machine, no result. added cpu, ram, ...

What really helped: I cleaned the event-log ( more the 8 000 000 items in there .. )

Now the site is really responsive ..

So, I guess some of you have more than 8 000 000 events ?

Regards,

Paul

Van: observium <observium-bounces@observium.org> namens Matthew A Harper <Matthew.A.Harper@tmr.qld.gov.au>
Verzonden: woensdag 18 juni 2014 1:07
Aan: Observium Network Observation System
Onderwerp: Re: [Observium] high CPU / slow response

Hi Paul,

My ibdata1 file is 1,948 MB.

Below are my settings in my.cnf:

key_buffer              = 16M

max_allowed_packet      = 16M

thread_stack            = 192K

thread_cache_size       = 8

myisam-recover         = BACKUP

table_cache            = 166

tmp_table_size          = 64M

max_heap_table_size     = 64M

innodb_buffer_pool_size = 350M

query_cache_limit       = 4M

query_cache_size        = 48M

Average response times:

Front page – 1.455s (Summary, Map, Syslog, Event log)

Ports page – 2.323s (32362 ports in one query)

Devices page – 1.739s (822 devices)

Kind regards,

Matt Harper
Network Analyst | ICT Operations
Information Technology Branch | Department of Transport and Main Roads

Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006
PO Box 673 | Fortitude Valley Qld 4006
P: (07) 30661125 | F: (07) 30662044
E: matthew.a.harper@tmr.qld.gov.au
W: www.tmr.qld.gov.au

From: observium [mailto:observium-bounces@observium.org] On Behalf Of Paul Meys
Sent: Tuesday, 17 June 2014 6:16 PM
To: Observium Network Observation System
Subject: Re: [Observium] high CPU / slow response

Hi All,

thanks already for the help.

The stuff I already did:

- apparently it was running on a 32bit kernel, move it to a new 64bit

- changed to 4 cores .. no real result

- disabled the unused-polling

It seems that the cpu-usage of mysqld goes really high when logging in to show to startpage with the map ..

when going directly to the device the response-time is "okay" but not great.

When running mysqltuner:

innodb_buffer_pool_size should be greater than 23G ?

How big is your ibdata1 ?

Thanks,

Have a good day ..

Paul

my top:

top - 10:08:10 up 13:46, 2 users, load average: 1.28, 0.90, 0.56

Tasks: 169 total, 3 running, 166 sleeping, 0 stopped, 0 zombie

%Cpu(s): 30.4 us, 1.8 sy, 0.0 ni, 67.5 id, 0.1 wa, 0.0 hi, 0.3 si, 0.0 st

KiB Mem: 16474156 total, 16241936 used, 232220 free, 225928 buffers

KiB Swap: 10644476 total, 25912 used, 10618564 free, 11002996 cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

2755 mysql 20 0 5641m 4.3g 7888 S 114.8 27.1 23:33.94 mysqld

Van: observium <observium-bounces@observium.org> namens Matthew A Harper <Matthew.A.Harper@tmr.qld.gov.au>
Verzonden: dinsdag 17 juni 2014 0:03
Aan: Observium Network Observation System
Onderwerp: Re: [Observium] high CPU / slow response

Hi Paul,

Mark has raised some good points. Check your TOP output as suggested but based on my specs I’d say you need more CPU power.

To give you an idea of scaling and more specifically CPU usage - I’m running the following specs and it runs fine:

Debian 7 on ESX

8x E7-4870 cores @ 84% (observium seems to benefit from faster clock speeds)

8GB ram (only using 2GB)

SAN Storage (using 82GB of 250GB)

820 devices, 33000 ports, 5500 sensors, 10 polling instances, 6 alert checkers.

With the number of ports you’re running I’d say you need more CPU power to handle all the port data. You could try as Martin suggested and disable any un-used polling. Specifically FDB, ARP and CEF (if used).

Further to this you should look into tuning MySQL to perform better with the types of queries its dealing with. This script is very helpful: https://github.com/major/MySQLTuner-perl/blob/master/mysqltuner.pl just follow the instructions in the output.

Hope this helps.

Kind regards,

Matt Harper
Network Analyst | ICT Operations
Information Technology Branch | Department of Transport and Main Roads

Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006
PO Box 673 | Fortitude Valley Qld 4006
P: (07) 30661125 | F: (07) 30662044
E: matthew.a.harper@tmr.qld.gov.au
W: www.tmr.qld.gov.au

From: observium [mailto:observium-bounces@observium.org] On Behalf Of Martin Smith
Sent: Tuesday, 17 June 2014 12:43 AM
To: observium@observium.org
Subject: Re: [Observium] high CPU / slow response

Hello,

We are running on a VM as well with 5 cores, 6GB of RAM and SAN storage. We currently have ~250 devices with 7100 ports and the system works fine.

First of all check “top” check to see if your database is waiting on slow disks with %wa.

Cpu(s): 14.9%us, 6.9%sy, 0.0%ni, 76.5%id, 1.2%wa, 0.0%hi, 0.6%si, 0.0%st

We have also disabled a couple features that seem to cause a much longer polling time.

/opt/observium/config.php

// Disable EtherLike-MIB

$config['enable_ports_etherlike']       = 0;     # Enable Polling EtherLike-MIB (doubles interface processing time)

// Disable FDB table polling

$config['poller_modules']['fdb-table']                   = 0;

// Disable ARP table polling

$config['discovery_modules']['arp-table']                    = 0;

Finally we use 5 threads of the poller-wrapper

/etc/cron.d/observium

*/5 *     * * *   root    /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1

Hopefully this helps,

Martin Smith | Network Analyst | Netgain
720 West Saint Germain Street | St. Cloud | MN | 56301

Phone: 320-257-6607 | 877.797.4700 x170

From: observium [mailto:observium-bounces@observium.org] On Behalf Of Paul Meys
Sent: Monday, June 16, 2014 5:07 AM
To: observium@observium.org
Subject: [Observium] high CPU / slow response

Hi,

I'm running the latest version, but it is extremely slow.

Even to load the login-page, it takes a very long time.

I always see mysqld using almost all CPU-power.

the server is running in vmware, os = debian.

Currently I have one v-core assigned with 16G of ram. ( already tried with 4 cores.. )

In Observium I have 75 devices with a total of 7500 ports.

Please advice ;-)

Thank you,

Have a good day,

Paul

***********************************************************************
WARNING: This email (including any attachments) may contain legally
privileged, confidential or private information and may be protected by
copyright. You may only use it if you are the person(s) it was
intended to be sent to and if you use it in an authorised way. No one
is allowed to use, review, alter, transmit, disclose, distribute, print
or copy this email without appropriate authority.

If this email was not intended for you and was sent to you by mistake,
please telephone or email me immediately, destroy any hardcopies of
this email and delete it and any copies of it from your computer
system. Any right which the sender may have under copyright law, and
any legal privilege and confidentiality attached to this email is not
waived or destroyed by that mistake.

It is your responsibility to ensure that this email does not contain
and is not affected by computer viruses, defects or interference by
third parties or replication problems (including incompatibility with
your computer system).

Opinions contained in this email do not necessarily reflect the
opinions of the Department of Transport and Main Roads,
or endorsed organisations utilising the same infrastructure.
***********************************************************************

_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

Xavier Beaudouin - Senior Network and System Administrator
Infrastructure and network director
Horizon Software - http://www.hsoftware.com/
13 rue La Fayette - 75009 PARIS - France
Phone: +33 (0)1 4260 9490 Fax: +33 (0)1 44 56 97 01
Why Top posting is bad : http://ck.wikia.com/wiki/TopPosting