high CPU / slow response
Hi,
I'm running the latest version, but it is extremely slow.
Even to load the login-page, it takes a very long time.
I always see mysqld using almost all CPU-power.
the server is running in vmware, os = debian.
Currently I have one v-core assigned with 16G of ram. ( already tried with 4 cores.. )
In Observium I have 75 devices with a total of 7500 ports.
Please advice ;-)
Thank you,
Have a good day,
Paul
Hello,
We are running on a VM as well with 5 cores, 6GB of RAM and SAN storage. We currently have ~250 devices with 7100 ports and the system works fine.
First of all check "top" check to see if your database is waiting on slow disks with %wa. Cpu(s): 14.9%us, 6.9%sy, 0.0%ni, 76.5%id, 1.2%wa, 0.0%hi, 0.6%si, 0.0%st
We have also disabled a couple features that seem to cause a much longer polling time. /opt/observium/config.php
// Disable EtherLike-MIB $config['enable_ports_etherlike'] = 0; # Enable Polling EtherLike-MIB (doubles interface processing time)
// Disable FDB table polling $config['poller_modules']['fdb-table'] = 0;
// Disable ARP table polling $config['discovery_modules']['arp-table'] = 0;
Finally we use 5 threads of the poller-wrapper /etc/cron.d/observium */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Hopefully this helps,
Martin Smith | Network Analyst | Netgain 720 West Saint Germain Street | St. Cloud | MN | 56301 Phone: 320-257-6607 | 877.797.4700 x170
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Paul Meys Sent: Monday, June 16, 2014 5:07 AM To: observium@observium.org Subject: [Observium] high CPU / slow response
Hi,
I'm running the latest version, but it is extremely slow.
Even to load the login-page, it takes a very long time.
I always see mysqld using almost all CPU-power.
the server is running in vmware, os = debian.
Currently I have one v-core assigned with 16G of ram. ( already tried with 4 cores.. )
In Observium I have 75 devices with a total of 7500 ports.
Please advice ;-)
Thank you,
Have a good day,
Paul
Hi Paul,
Mark has raised some good points. Check your TOP output as suggested but based on my specs I'd say you need more CPU power.
To give you an idea of scaling and more specifically CPU usage - I'm running the following specs and it runs fine:
Debian 7 on ESX 8x E7-4870 cores @ 84% (observium seems to benefit from faster clock speeds) 8GB ram (only using 2GB) SAN Storage (using 82GB of 250GB)
820 devices, 33000 ports, 5500 sensors, 10 polling instances, 6 alert checkers.
With the number of ports you're running I'd say you need more CPU power to handle all the port data. You could try as Martin suggested and disable any un-used polling. Specifically FDB, ARP and CEF (if used).
Further to this you should look into tuning MySQL to perform better with the types of queries its dealing with. This script is very helpful: https://github.com/major/MySQLTuner-perl/blob/master/mysqltuner.pl just follow the instructions in the output.
Hope this helps.
Kind regards,
Matt Harper Network Analyst | ICT Operations Information Technology Branch | Department of Transport and Main Roads
Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006 PO Box 673 | Fortitude Valley Qld 4006 P: (07) 30661125 | F: (07) 30662044 E: matthew.a.harper@tmr.qld.gov.aumailto:matthew.a.harper@tmr.qld.gov.au W: www.tmr.qld.gov.auhttp://www.tmr.qld.gov.au/
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Martin Smith Sent: Tuesday, 17 June 2014 12:43 AM To: observium@observium.org Subject: Re: [Observium] high CPU / slow response
Hello,
We are running on a VM as well with 5 cores, 6GB of RAM and SAN storage. We currently have ~250 devices with 7100 ports and the system works fine.
First of all check "top" check to see if your database is waiting on slow disks with %wa. Cpu(s): 14.9%us, 6.9%sy, 0.0%ni, 76.5%id, 1.2%wa, 0.0%hi, 0.6%si, 0.0%st
We have also disabled a couple features that seem to cause a much longer polling time. /opt/observium/config.php
// Disable EtherLike-MIB $config['enable_ports_etherlike'] = 0; # Enable Polling EtherLike-MIB (doubles interface processing time)
// Disable FDB table polling $config['poller_modules']['fdb-table'] = 0;
// Disable ARP table polling $config['discovery_modules']['arp-table'] = 0;
Finally we use 5 threads of the poller-wrapper /etc/cron.d/observium */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Hopefully this helps,
Martin Smith | Network Analyst | Netgain 720 West Saint Germain Street | St. Cloud | MN | 56301 Phone: 320-257-6607 | 877.797.4700 x170
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Paul Meys Sent: Monday, June 16, 2014 5:07 AM To: observium@observium.orgmailto:observium@observium.org Subject: [Observium] high CPU / slow response
Hi,
I'm running the latest version, but it is extremely slow.
Even to load the login-page, it takes a very long time.
I always see mysqld using almost all CPU-power.
the server is running in vmware, os = debian.
Currently I have one v-core assigned with 16G of ram. ( already tried with 4 cores.. )
In Observium I have 75 devices with a total of 7500 ports.
Please advice ;-)
Thank you,
Have a good day,
Paul
*********************************************************************** WARNING: This email (including any attachments) may contain legally privileged, confidential or private information and may be protected by copyright. You may only use it if you are the person(s) it was intended to be sent to and if you use it in an authorised way. No one is allowed to use, review, alter, transmit, disclose, distribute, print or copy this email without appropriate authority.
If this email was not intended for you and was sent to you by mistake, please telephone or email me immediately, destroy any hardcopies of this email and delete it and any copies of it from your computer system. Any right which the sender may have under copyright law, and any legal privilege and confidentiality attached to this email is not waived or destroyed by that mistake.
It is your responsibility to ensure that this email does not contain and is not affected by computer viruses, defects or interference by third parties or replication problems (including incompatibility with your computer system).
Opinions contained in this email do not necessarily reflect the opinions of the Department of Transport and Main Roads, or endorsed organisations utilising the same infrastructure. ***********************************************************************
Hi All,
thanks already for the help.
The stuff I already did:
- apparently it was running on a 32bit kernel, move it to a new 64bit
- changed to 4 cores .. no real result
- disabled the unused-polling
It seems that the cpu-usage of mysqld goes really high when logging in to show to startpage with the map ..
when going directly to the device the response-time is "okay" but not great.
When running mysqltuner:
innodb_buffer_pool_size should be greater than 23G ?
How big is your ibdata1 ?
Thanks,
Have a good day ..
Paul
my top:
top - 10:08:10 up 13:46, 2 users, load average: 1.28, 0.90, 0.56 Tasks: 169 total, 3 running, 166 sleeping, 0 stopped, 0 zombie %Cpu(s): 30.4 us, 1.8 sy, 0.0 ni, 67.5 id, 0.1 wa, 0.0 hi, 0.3 si, 0.0 st KiB Mem: 16474156 total, 16241936 used, 232220 free, 225928 buffers KiB Swap: 10644476 total, 25912 used, 10618564 free, 11002996 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2755 mysql 20 0 5641m 4.3g 7888 S 114.8 27.1 23:33.94 mysqld
________________________________ Van: observium observium-bounces@observium.org namens Matthew A Harper Matthew.A.Harper@tmr.qld.gov.au Verzonden: dinsdag 17 juni 2014 0:03 Aan: Observium Network Observation System Onderwerp: Re: [Observium] high CPU / slow response
Hi Paul,
Mark has raised some good points. Check your TOP output as suggested but based on my specs I’d say you need more CPU power.
To give you an idea of scaling and more specifically CPU usage - I’m running the following specs and it runs fine:
Debian 7 on ESX 8x E7-4870 cores @ 84% (observium seems to benefit from faster clock speeds) 8GB ram (only using 2GB) SAN Storage (using 82GB of 250GB)
820 devices, 33000 ports, 5500 sensors, 10 polling instances, 6 alert checkers.
With the number of ports you’re running I’d say you need more CPU power to handle all the port data. You could try as Martin suggested and disable any un-used polling. Specifically FDB, ARP and CEF (if used).
Further to this you should look into tuning MySQL to perform better with the types of queries its dealing with. This script is very helpful: https://github.com/major/MySQLTuner-perl/blob/master/mysqltuner.pl just follow the instructions in the output.
Hope this helps.
Kind regards,
Matt Harper Network Analyst | ICT Operations Information Technology Branch | Department of Transport and Main Roads
Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006 PO Box 673 | Fortitude Valley Qld 4006 P: (07) 30661125 | F: (07) 30662044 E: matthew.a.harper@tmr.qld.gov.aumailto:matthew.a.harper@tmr.qld.gov.au W: www.tmr.qld.gov.auhttp://www.tmr.qld.gov.au/
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Martin Smith Sent: Tuesday, 17 June 2014 12:43 AM To: observium@observium.org Subject: Re: [Observium] high CPU / slow response
Hello,
We are running on a VM as well with 5 cores, 6GB of RAM and SAN storage. We currently have ~250 devices with 7100 ports and the system works fine.
First of all check “top” check to see if your database is waiting on slow disks with %wa. Cpu(s): 14.9%us, 6.9%sy, 0.0%ni, 76.5%id, 1.2%wa, 0.0%hi, 0.6%si, 0.0%st
We have also disabled a couple features that seem to cause a much longer polling time. /opt/observium/config.php
// Disable EtherLike-MIB $config['enable_ports_etherlike'] = 0; # Enable Polling EtherLike-MIB (doubles interface processing time)
// Disable FDB table polling $config['poller_modules']['fdb-table'] = 0;
// Disable ARP table polling $config['discovery_modules']['arp-table'] = 0;
Finally we use 5 threads of the poller-wrapper /etc/cron.d/observium */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Hopefully this helps,
Martin Smith | Network Analyst | Netgain 720 West Saint Germain Street | St. Cloud | MN | 56301 Phone: 320-257-6607 | 877.797.4700 x170
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Paul Meys Sent: Monday, June 16, 2014 5:07 AM To: observium@observium.orgmailto:observium@observium.org Subject: [Observium] high CPU / slow response
Hi,
I'm running the latest version, but it is extremely slow.
Even to load the login-page, it takes a very long time.
I always see mysqld using almost all CPU-power.
the server is running in vmware, os = debian.
Currently I have one v-core assigned with 16G of ram. ( already tried with 4 cores.. )
In Observium I have 75 devices with a total of 7500 ports.
Please advice ;-)
Thank you,
Have a good day,
Paul
*********************************************************************** WARNING: This email (including any attachments) may contain legally privileged, confidential or private information and may be protected by copyright. You may only use it if you are the person(s) it was intended to be sent to and if you use it in an authorised way. No one is allowed to use, review, alter, transmit, disclose, distribute, print or copy this email without appropriate authority.
If this email was not intended for you and was sent to you by mistake, please telephone or email me immediately, destroy any hardcopies of this email and delete it and any copies of it from your computer system. Any right which the sender may have under copyright law, and any legal privilege and confidentiality attached to this email is not waived or destroyed by that mistake.
It is your responsibility to ensure that this email does not contain and is not affected by computer viruses, defects or interference by third parties or replication problems (including incompatibility with your computer system).
Opinions contained in this email do not necessarily reflect the opinions of the Department of Transport and Main Roads, or endorsed organisations utilising the same infrastructure. ***********************************************************************
Hi Paul
Have no issues like this (using community edition BTW), with about 142 devices polled.
The stuff I already did:
- apparently it was running on a 32bit kernel, move it to a new 64bit
- changed to 4 cores .. no real result
- disabled the unused-polling
It seems that the cpu-usage of mysqld goes really high when logging in to show to startpage with the map ..
when going directly to the device the response-time is "okay" but not great.
When running mysqltuner:
innodb_buffer_pool_size should be greater than 23G ?
How big is your ibdata1 ?
Some optimizations :
innodb_file_per_table=1 innodb_buffer_pool_size=1024M max_connections = 300
And use XFS with option noatime uppon options of FSTAB.
Also use elevator=deadline or elevator=noop (on a VM) is the best way to have more IO than the shitty CFQ ...
Xavier
Hi Paul,
My ibdata1 file is 1,948 MB.
Below are my settings in my.cnf:
key_buffer = 16M max_allowed_packet = 16M thread_stack = 192K thread_cache_size = 8 myisam-recover = BACKUP table_cache = 166 tmp_table_size = 64M max_heap_table_size = 64M innodb_buffer_pool_size = 350M query_cache_limit = 4M query_cache_size = 48M
Average response times:
Front page – 1.455s (Summary, Map, Syslog, Event log) Ports page – 2.323s (32362 ports in one query) Devices page – 1.739s (822 devices)
Kind regards,
Matt Harper Network Analyst | ICT Operations Information Technology Branch | Department of Transport and Main Roads
Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006 PO Box 673 | Fortitude Valley Qld 4006 P: (07) 30661125 | F: (07) 30662044 E: matthew.a.harper@tmr.qld.gov.aumailto:matthew.a.harper@tmr.qld.gov.au W: www.tmr.qld.gov.auhttp://www.tmr.qld.gov.au/
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Paul Meys Sent: Tuesday, 17 June 2014 6:16 PM To: Observium Network Observation System Subject: Re: [Observium] high CPU / slow response
Hi All,
thanks already for the help.
The stuff I already did:
- apparently it was running on a 32bit kernel, move it to a new 64bit
- changed to 4 cores .. no real result
- disabled the unused-polling
It seems that the cpu-usage of mysqld goes really high when logging in to show to startpage with the map ..
when going directly to the device the response-time is "okay" but not great.
When running mysqltuner:
innodb_buffer_pool_size should be greater than 23G ?
How big is your ibdata1 ?
Thanks,
Have a good day ..
Paul
my top: top - 10:08:10 up 13:46, 2 users, load average: 1.28, 0.90, 0.56 Tasks: 169 total, 3 running, 166 sleeping, 0 stopped, 0 zombie %Cpu(s): 30.4 us, 1.8 sy, 0.0 ni, 67.5 id, 0.1 wa, 0.0 hi, 0.3 si, 0.0 st KiB Mem: 16474156 total, 16241936 used, 232220 free, 225928 buffers KiB Swap: 10644476 total, 25912 used, 10618564 free, 11002996 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2755 mysql 20 0 5641m 4.3g 7888 S 114.8 27.1 23:33.94 mysqld
________________________________ Van: observium <observium-bounces@observium.orgmailto:observium-bounces@observium.org> namens Matthew A Harper <Matthew.A.Harper@tmr.qld.gov.aumailto:Matthew.A.Harper@tmr.qld.gov.au> Verzonden: dinsdag 17 juni 2014 0:03 Aan: Observium Network Observation System Onderwerp: Re: [Observium] high CPU / slow response
Hi Paul,
Mark has raised some good points. Check your TOP output as suggested but based on my specs I’d say you need more CPU power.
To give you an idea of scaling and more specifically CPU usage - I’m running the following specs and it runs fine:
Debian 7 on ESX 8x E7-4870 cores @ 84% (observium seems to benefit from faster clock speeds) 8GB ram (only using 2GB) SAN Storage (using 82GB of 250GB)
820 devices, 33000 ports, 5500 sensors, 10 polling instances, 6 alert checkers.
With the number of ports you’re running I’d say you need more CPU power to handle all the port data. You could try as Martin suggested and disable any un-used polling. Specifically FDB, ARP and CEF (if used).
Further to this you should look into tuning MySQL to perform better with the types of queries its dealing with. This script is very helpful: https://github.com/major/MySQLTuner-perl/blob/master/mysqltuner.pl just follow the instructions in the output.
Hope this helps.
Kind regards,
Matt Harper Network Analyst | ICT Operations Information Technology Branch | Department of Transport and Main Roads
Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006 PO Box 673 | Fortitude Valley Qld 4006 P: (07) 30661125 | F: (07) 30662044 E: matthew.a.harper@tmr.qld.gov.aumailto:matthew.a.harper@tmr.qld.gov.au W: www.tmr.qld.gov.auhttp://www.tmr.qld.gov.au/
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Martin Smith Sent: Tuesday, 17 June 2014 12:43 AM To: observium@observium.orgmailto:observium@observium.org Subject: Re: [Observium] high CPU / slow response
Hello,
We are running on a VM as well with 5 cores, 6GB of RAM and SAN storage. We currently have ~250 devices with 7100 ports and the system works fine.
First of all check “top” check to see if your database is waiting on slow disks with %wa. Cpu(s): 14.9%us, 6.9%sy, 0.0%ni, 76.5%id, 1.2%wa, 0.0%hi, 0.6%si, 0.0%st
We have also disabled a couple features that seem to cause a much longer polling time. /opt/observium/config.php
// Disable EtherLike-MIB $config['enable_ports_etherlike'] = 0; # Enable Polling EtherLike-MIB (doubles interface processing time)
// Disable FDB table polling $config['poller_modules']['fdb-table'] = 0;
// Disable ARP table polling $config['discovery_modules']['arp-table'] = 0;
Finally we use 5 threads of the poller-wrapper /etc/cron.d/observium */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Hopefully this helps,
Martin Smith | Network Analyst | Netgain 720 West Saint Germain Street | St. Cloud | MN | 56301 Phone: 320-257-6607 | 877.797.4700 x170
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Paul Meys Sent: Monday, June 16, 2014 5:07 AM To: observium@observium.orgmailto:observium@observium.org Subject: [Observium] high CPU / slow response
Hi,
I'm running the latest version, but it is extremely slow.
Even to load the login-page, it takes a very long time.
I always see mysqld using almost all CPU-power.
the server is running in vmware, os = debian.
Currently I have one v-core assigned with 16G of ram. ( already tried with 4 cores.. )
In Observium I have 75 devices with a total of 7500 ports.
Please advice ;-)
Thank you,
Have a good day,
Paul
*********************************************************************** WARNING: This email (including any attachments) may contain legally privileged, confidential or private information and may be protected by copyright. You may only use it if you are the person(s) it was intended to be sent to and if you use it in an authorised way. No one is allowed to use, review, alter, transmit, disclose, distribute, print or copy this email without appropriate authority.
If this email was not intended for you and was sent to you by mistake, please telephone or email me immediately, destroy any hardcopies of this email and delete it and any copies of it from your computer system. Any right which the sender may have under copyright law, and any legal privilege and confidentiality attached to this email is not waived or destroyed by that mistake.
It is your responsibility to ensure that this email does not contain and is not affected by computer viruses, defects or interference by third parties or replication problems (including incompatibility with your computer system).
Opinions contained in this email do not necessarily reflect the opinions of the Department of Transport and Main Roads, or endorsed organisations utilising the same infrastructure. ***********************************************************************
Hi Guys,
to give an update,
I tried to tune the machine, no result. added cpu, ram, ...
What really helped: I cleaned the event-log ( more the 8 000 000 items in there .. )
Now the site is really responsive ..
So, I guess some of you have more than 8 000 000 events ?
Regards,
Paul
________________________________ Van: observium observium-bounces@observium.org namens Matthew A Harper Matthew.A.Harper@tmr.qld.gov.au Verzonden: woensdag 18 juni 2014 1:07 Aan: Observium Network Observation System Onderwerp: Re: [Observium] high CPU / slow response
Hi Paul,
My ibdata1 file is 1,948 MB.
Below are my settings in my.cnf:
key_buffer = 16M max_allowed_packet = 16M thread_stack = 192K thread_cache_size = 8 myisam-recover = BACKUP table_cache = 166 tmp_table_size = 64M max_heap_table_size = 64M innodb_buffer_pool_size = 350M query_cache_limit = 4M query_cache_size = 48M
Average response times:
Front page – 1.455s (Summary, Map, Syslog, Event log) Ports page – 2.323s (32362 ports in one query) Devices page – 1.739s (822 devices)
Kind regards,
Matt Harper Network Analyst | ICT Operations Information Technology Branch | Department of Transport and Main Roads
Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006 PO Box 673 | Fortitude Valley Qld 4006 P: (07) 30661125 | F: (07) 30662044 E: matthew.a.harper@tmr.qld.gov.aumailto:matthew.a.harper@tmr.qld.gov.au W: www.tmr.qld.gov.auhttp://www.tmr.qld.gov.au/
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Paul Meys Sent: Tuesday, 17 June 2014 6:16 PM To: Observium Network Observation System Subject: Re: [Observium] high CPU / slow response
Hi All,
thanks already for the help.
The stuff I already did:
- apparently it was running on a 32bit kernel, move it to a new 64bit
- changed to 4 cores .. no real result
- disabled the unused-polling
It seems that the cpu-usage of mysqld goes really high when logging in to show to startpage with the map ..
when going directly to the device the response-time is "okay" but not great.
When running mysqltuner:
innodb_buffer_pool_size should be greater than 23G ?
How big is your ibdata1 ?
Thanks,
Have a good day ..
Paul
my top: top - 10:08:10 up 13:46, 2 users, load average: 1.28, 0.90, 0.56 Tasks: 169 total, 3 running, 166 sleeping, 0 stopped, 0 zombie %Cpu(s): 30.4 us, 1.8 sy, 0.0 ni, 67.5 id, 0.1 wa, 0.0 hi, 0.3 si, 0.0 st KiB Mem: 16474156 total, 16241936 used, 232220 free, 225928 buffers KiB Swap: 10644476 total, 25912 used, 10618564 free, 11002996 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2755 mysql 20 0 5641m 4.3g 7888 S 114.8 27.1 23:33.94 mysqld
________________________________ Van: observium <observium-bounces@observium.orgmailto:observium-bounces@observium.org> namens Matthew A Harper <Matthew.A.Harper@tmr.qld.gov.aumailto:Matthew.A.Harper@tmr.qld.gov.au> Verzonden: dinsdag 17 juni 2014 0:03 Aan: Observium Network Observation System Onderwerp: Re: [Observium] high CPU / slow response
Hi Paul,
Mark has raised some good points. Check your TOP output as suggested but based on my specs I’d say you need more CPU power.
To give you an idea of scaling and more specifically CPU usage - I’m running the following specs and it runs fine:
Debian 7 on ESX 8x E7-4870 cores @ 84% (observium seems to benefit from faster clock speeds) 8GB ram (only using 2GB) SAN Storage (using 82GB of 250GB)
820 devices, 33000 ports, 5500 sensors, 10 polling instances, 6 alert checkers.
With the number of ports you’re running I’d say you need more CPU power to handle all the port data. You could try as Martin suggested and disable any un-used polling. Specifically FDB, ARP and CEF (if used).
Further to this you should look into tuning MySQL to perform better with the types of queries its dealing with. This script is very helpful: https://github.com/major/MySQLTuner-perl/blob/master/mysqltuner.pl just follow the instructions in the output.
Hope this helps.
Kind regards,
Matt Harper Network Analyst | ICT Operations Information Technology Branch | Department of Transport and Main Roads
Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006 PO Box 673 | Fortitude Valley Qld 4006 P: (07) 30661125 | F: (07) 30662044 E: matthew.a.harper@tmr.qld.gov.aumailto:matthew.a.harper@tmr.qld.gov.au W: www.tmr.qld.gov.auhttp://www.tmr.qld.gov.au/
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Martin Smith Sent: Tuesday, 17 June 2014 12:43 AM To: observium@observium.orgmailto:observium@observium.org Subject: Re: [Observium] high CPU / slow response
Hello,
We are running on a VM as well with 5 cores, 6GB of RAM and SAN storage. We currently have ~250 devices with 7100 ports and the system works fine.
First of all check “top” check to see if your database is waiting on slow disks with %wa. Cpu(s): 14.9%us, 6.9%sy, 0.0%ni, 76.5%id, 1.2%wa, 0.0%hi, 0.6%si, 0.0%st
We have also disabled a couple features that seem to cause a much longer polling time. /opt/observium/config.php
// Disable EtherLike-MIB $config['enable_ports_etherlike'] = 0; # Enable Polling EtherLike-MIB (doubles interface processing time)
// Disable FDB table polling $config['poller_modules']['fdb-table'] = 0;
// Disable ARP table polling $config['discovery_modules']['arp-table'] = 0;
Finally we use 5 threads of the poller-wrapper /etc/cron.d/observium */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Hopefully this helps,
Martin Smith | Network Analyst | Netgain 720 West Saint Germain Street | St. Cloud | MN | 56301 Phone: 320-257-6607 | 877.797.4700 x170
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Paul Meys Sent: Monday, June 16, 2014 5:07 AM To: observium@observium.orgmailto:observium@observium.org Subject: [Observium] high CPU / slow response
Hi,
I'm running the latest version, but it is extremely slow.
Even to load the login-page, it takes a very long time.
I always see mysqld using almost all CPU-power.
the server is running in vmware, os = debian.
Currently I have one v-core assigned with 16G of ram. ( already tried with 4 cores.. )
In Observium I have 75 devices with a total of 7500 ports.
Please advice ;-)
Thank you,
Have a good day,
Paul
*********************************************************************** WARNING: This email (including any attachments) may contain legally privileged, confidential or private information and may be protected by copyright. You may only use it if you are the person(s) it was intended to be sent to and if you use it in an authorised way. No one is allowed to use, review, alter, transmit, disclose, distribute, print or copy this email without appropriate authority.
If this email was not intended for you and was sent to you by mistake, please telephone or email me immediately, destroy any hardcopies of this email and delete it and any copies of it from your computer system. Any right which the sender may have under copyright law, and any legal privilege and confidentiality attached to this email is not waived or destroyed by that mistake.
It is your responsibility to ensure that this email does not contain and is not affected by computer viruses, defects or interference by third parties or replication problems (including incompatibility with your computer system).
Opinions contained in this email do not necessarily reflect the opinions of the Department of Transport and Main Roads, or endorsed organisations utilising the same infrastructure. ***********************************************************************
I had same issue with syslog... events. Maybe a an auto-clean withing a configured delay in the config.php should be done ?
----- Mail original -----
De: "Paul Meys" paul@milla-it.be À: "Observium Network Observation System" observium@observium.org Envoyé: Mardi 15 Juillet 2014 10:28:58 Objet: Re: [Observium] high CPU / slow response
Hi Guys,
to give an update,
I tried to tune the machine, no result. added cpu, ram, ...
What really helped: I cleaned the event-log ( more the 8 000 000 items in there .. )
Now the site is really responsive ..
So, I guess some of you have more than 8 000 000 events ?
Regards,
Paul
Van: observium observium-bounces@observium.org namens Matthew A Harper Matthew.A.Harper@tmr.qld.gov.au Verzonden: woensdag 18 juni 2014 1:07 Aan: Observium Network Observation System Onderwerp: Re: [Observium] high CPU / slow response
Hi Paul,
My ibdata1 file is 1,948 MB.
Below are my settings in my.cnf:
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
myisam-recover = BACKUP
table_cache = 166
tmp_table_size = 64M
max_heap_table_size = 64M
innodb_buffer_pool_size = 350M
query_cache_limit = 4M
query_cache_size = 48M
Average response times:
Front page – 1.455s (Summary, Map, Syslog, Event log)
Ports page – 2.323s (32362 ports in one query)
Devices page – 1.739s (822 devices)
Kind regards,
Matt Harper Network Analyst | ICT Operations Information Technology Branch | Department of Transport and Main Roads
Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006 PO Box 673 | Fortitude Valley Qld 4006 P: (07) 30661125 | F: (07) 30662044 E: matthew.a.harper@tmr.qld.gov.au W: www.tmr.qld.gov.au
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Paul Meys Sent: Tuesday, 17 June 2014 6:16 PM To: Observium Network Observation System Subject: Re: [Observium] high CPU / slow response
Hi All,
thanks already for the help.
The stuff I already did:
- apparently it was running on a 32bit kernel, move it to a new 64bit
- changed to 4 cores .. no real result
- disabled the unused-polling
It seems that the cpu-usage of mysqld goes really high when logging in to show to startpage with the map ..
when going directly to the device the response-time is "okay" but not great.
When running mysqltuner:
innodb_buffer_pool_size should be greater than 23G ?
How big is your ibdata1 ?
Thanks,
Have a good day ..
Paul
my top:
top - 10:08:10 up 13:46, 2 users, load average: 1.28, 0.90, 0.56
Tasks: 169 total, 3 running, 166 sleeping, 0 stopped, 0 zombie
%Cpu(s): 30.4 us, 1.8 sy, 0.0 ni, 67.5 id, 0.1 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem: 16474156 total, 16241936 used, 232220 free, 225928 buffers
KiB Swap: 10644476 total, 25912 used, 10618564 free, 11002996 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2755 mysql 20 0 5641m 4.3g 7888 S 114.8 27.1 23:33.94 mysqld
Van: observium < observium-bounces@observium.org > namens Matthew A Harper < Matthew.A.Harper@tmr.qld.gov.au > Verzonden: dinsdag 17 juni 2014 0:03 Aan: Observium Network Observation System Onderwerp: Re: [Observium] high CPU / slow response
Hi Paul,
Mark has raised some good points. Check your TOP output as suggested but based on my specs I’d say you need more CPU power.
To give you an idea of scaling and more specifically CPU usage - I’m running the following specs and it runs fine:
Debian 7 on ESX
8x E7-4870 cores @ 84% (observium seems to benefit from faster clock speeds)
8GB ram (only using 2GB)
SAN Storage (using 82GB of 250GB)
820 devices, 33000 ports, 5500 sensors, 10 polling instances, 6 alert checkers.
With the number of ports you’re running I’d say you need more CPU power to handle all the port data. You could try as Martin suggested and disable any un-used polling. Specifically FDB, ARP and CEF (if used).
Further to this you should look into tuning MySQL to perform better with the types of queries its dealing with. This script is very helpful: https://github.com/major/MySQLTuner-perl/blob/master/mysqltuner.pl just follow the instructions in the output.
Hope this helps.
Kind regards,
Matt Harper Network Analyst | ICT Operations Information Technology Branch | Department of Transport and Main Roads
Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006 PO Box 673 | Fortitude Valley Qld 4006 P: (07) 30661125 | F: (07) 30662044 E: matthew.a.harper@tmr.qld.gov.au W: www.tmr.qld.gov.au
From: observium [ mailto:observium-bounces@observium.org ] On Behalf Of Martin Smith Sent: Tuesday, 17 June 2014 12:43 AM To: observium@observium.org Subject: Re: [Observium] high CPU / slow response
Hello,
We are running on a VM as well with 5 cores, 6GB of RAM and SAN storage. We currently have ~250 devices with 7100 ports and the system works fine.
First of all check “top” check to see if your database is waiting on slow disks with %wa.
Cpu(s): 14.9%us, 6.9%sy, 0.0%ni, 76.5%id, 1.2%wa , 0.0%hi, 0.6%si, 0.0%st
We have also disabled a couple features that seem to cause a much longer polling time.
/opt/observium/config.php
// Disable EtherLike-MIB
$config['enable_ports_etherlike'] = 0; # Enable Polling EtherLike-MIB (doubles interface processing time)
// Disable FDB table polling
$config['poller_modules']['fdb-table'] = 0;
// Disable ARP table polling
$config['discovery_modules']['arp-table'] = 0;
Finally we use 5 threads of the poller-wrapper
/etc/cron.d/observium
*/5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Hopefully this helps,
Martin Smith | Network Analyst | Netgain 720 West Saint Germain Street | St. Cloud | MN | 56301
Phone: 320-257-6607 | 877.797.4700 x170
From: observium [ mailto:observium-bounces@observium.org ] On Behalf Of Paul Meys Sent: Monday, June 16, 2014 5:07 AM To: observium@observium.org Subject: [Observium] high CPU / slow response
Hi,
I'm running the latest version, but it is extremely slow.
Even to load the login-page, it takes a very long time.
I always see mysqld using almost all CPU-power.
the server is running in vmware, os = debian.
Currently I have one v-core assigned with 16G of ram. ( already tried with 4 cores.. )
In Observium I have 75 devices with a total of 7500 ports.
Please advice ;-)
Thank you,
Have a good day,
Paul
WARNING: This email (including any attachments) may contain legally privileged, confidential or private information and may be protected by copyright. You may only use it if you are the person(s) it was intended to be sent to and if you use it in an authorised way. No one is allowed to use, review, alter, transmit, disclose, distribute, print or copy this email without appropriate authority.
If this email was not intended for you and was sent to you by mistake, please telephone or email me immediately, destroy any hardcopies of this email and delete it and any copies of it from your computer system. Any right which the sender may have under copyright law, and any legal privilege and confidentiality attached to this email is not waived or destroyed by that mistake.
It is your responsibility to ensure that this email does not contain and is not affected by computer viruses, defects or interference by third parties or replication problems (including incompatibility with your computer system).
Opinions contained in this email do not necessarily reflect the opinions of the Department of Transport and Main Roads, or endorsed organisations utilising the same infrastructure.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
I'm working on a housekeeping script as we speak.
Combination of 2 scripts people already submitted to the bug tracker and some own ideas to clean things up. :)
Tom
On 07/15/2014 10:43 AM, Xavier Beaudouin wrote:
I had same issue with syslog... events. Maybe a an auto-clean withing a configured delay in the config.php should be done ?
*De: *"Paul Meys" <paul@milla-it.be> *À: *"Observium Network Observation System" <observium@observium.org> *Envoyé: *Mardi 15 Juillet 2014 10:28:58 *Objet: *Re: [Observium] high CPU / slow response Hi Guys, to give an update, I tried to tune the machine, no result. added cpu, ram, ... What really helped: I cleaned the event-log ( more the 8 000 000 items in there .. ) Now the site is really responsive .. So, I guess some of you have more than 8 000 000 events ? Regards, Paul ------------------------------------------------------------------------ *Van:* observium <observium-bounces@observium.org> namens Matthew A Harper <Matthew.A.Harper@tmr.qld.gov.au> *Verzonden:* woensdag 18 juni 2014 1:07 *Aan:* Observium Network Observation System *Onderwerp:* Re: [Observium] high CPU / slow response Hi Paul, My ibdata1 file is 1,948 MB. Below are my settings in my.cnf: key_buffer = 16M max_allowed_packet = 16M thread_stack = 192K thread_cache_size = 8 myisam-recover = BACKUP table_cache = 166 tmp_table_size = 64M max_heap_table_size = 64M innodb_buffer_pool_size = 350M query_cache_limit = 4M query_cache_size = 48M Average response times: Front page -- 1.455s (Summary, Map, Syslog, Event log) Ports page -- 2.323s (32362 ports in one query) Devices page -- 1.739s (822 devices) Kind regards, * Matt Harper* Network Analyst| ICT Operations *Information Technology Branch* | Department of Transport and Main Roads Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006 PO Box 673 | Fortitude Valley Qld 4006 P: (07) 30661125 | F: (07) 30662044 E: matthew.a.harper@tmr.qld.gov.au <mailto:matthew.a.harper@tmr.qld.gov.au> W: www.tmr.qld.gov.au <http://www.tmr.qld.gov.au/> *From:*observium [mailto:observium-bounces@observium.org] *On Behalf Of *Paul Meys *Sent:* Tuesday, 17 June 2014 6:16 PM *To:* Observium Network Observation System *Subject:* Re: [Observium] high CPU / slow response Hi All, thanks already for the help. The stuff I already did: - apparently it was running on a 32bit kernel, move it to a new 64bit - changed to 4 cores .. no real result - disabled the unused-polling It seems that the cpu-usage of mysqld goes really high when logging in to show to startpage with the map .. when going directly to the device the response-time is "okay" but not great. When running mysqltuner: innodb_buffer_pool_size should be greater than 23G ? How big is your ibdata1 ? Thanks, Have a good day .. Paul my top: top - 10:08:10 up 13:46, 2 users, load average: 1.28, 0.90, 0.56 Tasks: 169 total, 3 running, 166 sleeping, 0 stopped, 0 zombie %Cpu(s): 30.4 us, 1.8 sy, 0.0 ni, 67.5 id, 0.1 wa, 0.0 hi, 0.3 si, 0.0 st KiB Mem: 16474156 total, 16241936 used, 232220 free, 225928 buffers KiB Swap: 10644476 total, 25912 used, 10618564 free, 11002996 cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2755 mysql 20 0 5641m 4.3g 7888 S 114.8 27.1 23:33.94 mysqld ------------------------------------------------------------------------ *Van:*observium <observium-bounces@observium.org <mailto:observium-bounces@observium.org>> namens Matthew A Harper <Matthew.A.Harper@tmr.qld.gov.au <mailto:Matthew.A.Harper@tmr.qld.gov.au>> *Verzonden:* dinsdag 17 juni 2014 0:03 *Aan:* Observium Network Observation System *Onderwerp:* Re: [Observium] high CPU / slow response Hi Paul, Mark has raised some good points. Check your TOP output as suggested but based on my specs I'd say you need more CPU power. To give you an idea of scaling and more specifically CPU usage - I'm running the following specs and it runs fine: Debian 7 on ESX 8x E7-4870 cores @ 84% (observium seems to benefit from faster clock speeds) 8GB ram (only using 2GB) SAN Storage (using 82GB of 250GB) 820 devices, 33000 ports, 5500 sensors, 10 polling instances, 6 alert checkers. With the number of ports you're running I'd say you need more CPU power to handle all the port data. You could try as Martin suggested and disable any un-used polling. Specifically FDB, ARP and CEF (if used). Further to this you should look into tuning MySQL to perform better with the types of queries its dealing with. This script is very helpful: https://github.com/major/MySQLTuner-perl/blob/master/mysqltuner.pl just follow the instructions in the output. Hope this helps. Kind regards, * Matt Harper* Network Analyst| ICT Operations *Information Technology Branch* | Department of Transport and Main Roads Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006 PO Box 673 | Fortitude Valley Qld 4006 P: (07) 30661125 | F: (07) 30662044 E: matthew.a.harper@tmr.qld.gov.au <mailto:matthew.a.harper@tmr.qld.gov.au> W: www.tmr.qld.gov.au <http://www.tmr.qld.gov.au/> *From:*observium [mailto:observium-bounces@observium.org] *On Behalf Of *Martin Smith *Sent:* Tuesday, 17 June 2014 12:43 AM *To:* observium@observium.org <mailto:observium@observium.org> *Subject:* Re: [Observium] high CPU / slow response Hello, We are running on a VM as well with 5 cores, 6GB of RAM and SAN storage. We currently have ~250 devices with 7100 ports and the system works fine. First of all check "top" check to see if your database is waiting on slow disks with %wa. Cpu(s): 14.9%us, 6.9%sy, 0.0%ni, 76.5%id, *1.2%wa*, 0.0%hi, 0.6%si, 0.0%st We have also disabled a couple features that seem to cause a much longer polling time. /opt/observium/config.php // Disable EtherLike-MIB $config['enable_ports_etherlike'] = 0; # Enable Polling EtherLike-MIB (doubles interface processing time) // Disable FDB table polling $config['poller_modules']['fdb-table'] = 0; // Disable ARP table polling $config['discovery_modules']['arp-table'] = 0; Finally we use 5 threads of the poller-wrapper /etc/cron.d/observium */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1 Hopefully this helps, *Martin Smith **|****Network Analyst **|****Netgain *720 West Saint Germain Street | St. Cloud | MN | 56301 Phone: 320-257-6607 *|* 877.797.4700 x170 *From:*observium [mailto:observium-bounces@observium.org] *On Behalf Of *Paul Meys *Sent:* Monday, June 16, 2014 5:07 AM *To:* observium@observium.org <mailto:observium@observium.org> *Subject:* [Observium] high CPU / slow response Hi, I'm running the latest version, but it is extremely slow. Even to load the login-page, it takes a very long time. I always see mysqld using almost all CPU-power. the server is running in vmware, os = debian. Currently I have one v-core assigned with 16G of ram. ( already tried with 4 cores.. ) In Observium I have 75 devices with a total of 7500 ports. Please advice ;-) Thank you, Have a good day, Paul *********************************************************************** WARNING: This email (including any attachments) may contain legally privileged, confidential or private information and may be protected by copyright. You may only use it if you are the person(s) it was intended to be sent to and if you use it in an authorised way. No one is allowed to use, review, alter, transmit, disclose, distribute, print or copy this email without appropriate authority. If this email was not intended for you and was sent to you by mistake, please telephone or email me immediately, destroy any hardcopies of this email and delete it and any copies of it from your computer system. Any right which the sender may have under copyright law, and any legal privilege and confidentiality attached to this email is not waived or destroyed by that mistake. It is your responsibility to ensure that this email does not contain and is not affected by computer viruses, defects or interference by third parties or replication problems (including incompatibility with your computer system). Opinions contained in this email do not necessarily reflect the opinions of the Department of Transport and Main Roads, or endorsed organisations utilising the same infrastructure. *********************************************************************** _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium-- Xavier Beaudouin - Senior Network and System Administrator Infrastructure and network director Horizon Software - http://www.hsoftware.com/ 13 rue La Fayette - 75009 PARIS - France Phone: +33 (0)1 4260 9490 Fax: +33 (0)1 44 56 97 01 Why Top posting is bad : http://ck.wikia.com/wiki/TopPosting
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Thanks Tom :D Xavier
----- Mail original -----
De: "Tom Laermans" tom.laermans@powersource.cx À: "Observium Network Observation System" observium@observium.org Envoyé: Mardi 15 Juillet 2014 11:22:28 Objet: Re: [Observium] high CPU / slow response
I'm working on a housekeeping script as we speak.
Combination of 2 scripts people already submitted to the bug tracker and some own ideas to clean things up. :)
Tom
On 07/15/2014 10:43 AM, Xavier Beaudouin wrote:
I had same issue with syslog... events.
Maybe a an auto-clean withing a configured delay in the config.php should be done ?
----- Mail original -----
De: "Paul Meys" paul@milla-it.be
À: "Observium Network Observation System" observium@observium.org
Envoyé: Mardi 15 Juillet 2014 10:28:58
Objet: Re: [Observium] high CPU / slow response
Hi Guys,
to give an update,
I tried to tune the machine, no result. added cpu, ram, ...
What really helped: I cleaned the event-log ( more the 8 000 000 items in there .. )
Now the site is really responsive ..
So, I guess some of you have more than 8 000 000 events ?
Regards,
Paul
Van: observium observium-bounces@observium.org namens Matthew A Harper Matthew.A.Harper@tmr.qld.gov.au
Verzonden: woensdag 18 juni 2014 1:07
Aan: Observium Network Observation System
Onderwerp: Re: [Observium] high CPU / slow response
Hi Paul,
My ibdata1 file is 1,948 MB.
Below are my settings in my.cnf:
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
myisam-recover = BACKUP
table_cache = 166
tmp_table_size = 64M
max_heap_table_size = 64M
innodb_buffer_pool_size = 350M
query_cache_limit = 4M
query_cache_size = 48M
Average response times:
Front page – 1.455s (Summary, Map, Syslog, Event log)
Ports page – 2.323s (32362 ports in one query)
Devices page – 1.739s (822 devices)
Kind regards,
Matt Harper
Network Analyst | ICT Operations
Information Technology Branch | Department of Transport and Main Roads
Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006
PO Box 673 | Fortitude Valley Qld 4006
P: (07) 30661125 | F: (07) 30662044
E: matthew.a.harper@tmr.qld.gov.au
W: www.tmr.qld.gov.au
From: observium [ mailto:observium-bounces@observium.org ] On Behalf Of Paul Meys
Sent: Tuesday, 17 June 2014 6:16 PM
To: Observium Network Observation System
Subject: Re: [Observium] high CPU / slow response
Hi All,
thanks already for the help.
The stuff I already did:
- apparently it was running on a 32bit kernel, move it to a new 64bit
- changed to 4 cores .. no real result
- disabled the unused-polling
It seems that the cpu-usage of mysqld goes really high when logging in to show to startpage with the map ..
when going directly to the device the response-time is "okay" but not great.
When running mysqltuner:
innodb_buffer_pool_size should be greater than 23G ?
How big is your ibdata1 ?
Thanks,
Have a good day ..
Paul
my top:
top - 10:08:10 up 13:46, 2 users, load average: 1.28, 0.90, 0.56
Tasks: 169 total, 3 running, 166 sleeping, 0 stopped, 0 zombie
%Cpu(s): 30.4 us, 1.8 sy, 0.0 ni, 67.5 id, 0.1 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem: 16474156 total, 16241936 used, 232220 free, 225928 buffers
KiB Swap: 10644476 total, 25912 used, 10618564 free, 11002996 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2755 mysql 20 0 5641m 4.3g 7888 S 114.8 27.1 23:33.94 mysqld
Van: observium < observium-bounces@observium.org > namens Matthew A Harper < Matthew.A.Harper@tmr.qld.gov.au >
Verzonden: dinsdag 17 juni 2014 0:03
Aan: Observium Network Observation System
Onderwerp: Re: [Observium] high CPU / slow response
Hi Paul,
Mark has raised some good points. Check your TOP output as suggested but based on my specs I’d say you need more CPU power.
To give you an idea of scaling and more specifically CPU usage - I’m running the following specs and it runs fine:
Debian 7 on ESX
8x E7-4870 cores @ 84% (observium seems to benefit from faster clock speeds)
8GB ram (only using 2GB)
SAN Storage (using 82GB of 250GB)
820 devices, 33000 ports, 5500 sensors, 10 polling instances, 6 alert checkers.
With the number of ports you’re running I’d say you need more CPU power to handle all the port data. You could try as Martin suggested and disable any un-used polling. Specifically FDB, ARP and CEF (if used).
Further to this you should look into tuning MySQL to perform better with the types of queries its dealing with. This script is very helpful: https://github.com/major/MySQLTuner-perl/blob/master/mysqltuner.pl just follow the instructions in the output.
Hope this helps.
Kind regards,
Matt Harper
Network Analyst | ICT Operations
Information Technology Branch | Department of Transport and Main Roads
Floor 4 | Transport House | 230 Brunswick Street | Fortitude Valley Qld 4006
PO Box 673 | Fortitude Valley Qld 4006
P: (07) 30661125 | F: (07) 30662044
E: matthew.a.harper@tmr.qld.gov.au
W: www.tmr.qld.gov.au
From: observium [ mailto:observium-bounces@observium.org ] On Behalf Of Martin Smith
Sent: Tuesday, 17 June 2014 12:43 AM
To: observium@observium.org
Subject: Re: [Observium] high CPU / slow response
Hello,
We are running on a VM as well with 5 cores, 6GB of RAM and SAN storage. We currently have ~250 devices with 7100 ports and the system works fine.
First of all check “top” check to see if your database is waiting on slow disks with %wa.
Cpu(s): 14.9%us, 6.9%sy, 0.0%ni, 76.5%id, 1.2%wa , 0.0%hi, 0.6%si, 0.0%st
We have also disabled a couple features that seem to cause a much longer polling time.
/opt/observium/config.php
// Disable EtherLike-MIB
$config['enable_ports_etherlike'] = 0; # Enable Polling EtherLike-MIB (doubles interface processing time)
// Disable FDB table polling
$config['poller_modules']['fdb-table'] = 0;
// Disable ARP table polling
$config['discovery_modules']['arp-table'] = 0;
Finally we use 5 threads of the poller-wrapper
/etc/cron.d/observium
*/5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Hopefully this helps,
Martin Smith | Network Analyst | Netgain
720 West Saint Germain Street | St. Cloud | MN | 56301
Phone: 320-257-6607 | 877.797.4700 x170
From: observium [ mailto:observium-bounces@observium.org ] On Behalf Of Paul Meys
Sent: Monday, June 16, 2014 5:07 AM
To: observium@observium.org
Subject: [Observium] high CPU / slow response
Hi,
I'm running the latest version, but it is extremely slow.
Even to load the login-page, it takes a very long time.
I always see mysqld using almost all CPU-power.
the server is running in vmware, os = debian.
Currently I have one v-core assigned with 16G of ram. ( already tried with 4 cores.. )
In Observium I have 75 devices with a total of 7500 ports.
Please advice ;-)
Thank you,
Have a good day,
Paul
WARNING: This email (including any attachments) may contain legally
privileged, confidential or private information and may be protected by
copyright. You may only use it if you are the person(s) it was
intended to be sent to and if you use it in an authorised way. No one
is allowed to use, review, alter, transmit, disclose, distribute, print
or copy this email without appropriate authority.
If this email was not intended for you and was sent to you by mistake,
please telephone or email me immediately, destroy any hardcopies of
this email and delete it and any copies of it from your computer
system. Any right which the sender may have under copyright law, and
any legal privilege and confidentiality attached to this email is not
waived or destroyed by that mistake.
It is your responsibility to ensure that this email does not contain
and is not affected by computer viruses, defects or interference by
third parties or replication problems (including incompatibility with
your computer system).
Opinions contained in this email do not necessarily reflect the
opinions of the Department of Transport and Main Roads,
or endorsed organisations utilising the same infrastructure.
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
--
Xavier Beaudouin - Senior Network and System Administrator
Infrastructure and network director
Horizon Software - http://www.hsoftware.com/
13 rue La Fayette - 75009 PARIS - France
Phone: +33 (0)1 4260 9490 Fax: +33 (0)1 44 56 97 01
Why Top posting is bad : http://ck.wikia.com/wiki/TopPosting
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (5)
-
Martin Smith -
Matthew A Harper -
Paul Meys -
Tom Laermans -
Xavier Beaudouin