Committed as r6969 for those on the Professional Edition. Adam says it doesn’t improve much for him, but it was significantly improved for me. Anyone else see
reduced CPU usage of the poller after this commit?
--
Cameron Moore
Manager of Systems & Networks
Hardin-Simmons University, Technology Services
Ph: (325) 670-1506 Fx: (325) 670-1570
From: observium [mailto:observium-bounces@observium.org]
On Behalf Of Moore, Cameron
Sent: Monday, September 07, 2015 3:39 PM
To: Observium Network Observation System <observium@observium.org>
Subject: Re: [Observium] poller.php burns all available CPU in the last community edition
I’ve submitted my solution to the poller’s high CPU usage:
http://jira.observium.org/browse/OBSERVIUM-1439
--
Cameron Moore
Manager of Systems & Networks
Hardin-Simmons University, Technology Services
Ph: (325) 670-1506 Fx: (325) 670-1570
From: observium [mailto:observium-bounces@observium.org]
On Behalf Of Moore, Cameron
Sent: Saturday, September 05, 2015 12:15 PM
To: Observium Network Observation System <observium@observium.org>
Subject: Re: [Observium] poller.php burns all available CPU in the last community edition
That patch did not work for me. It looked promising at first with the poller CPU usage cut by 30%. However, I let it run overnight and came in to a server with
load avg of 300 with tons of poller processes still running (cron spawns a new poller every 5 mins).
--
Cameron Moore
Manager of Systems & Networks
Hardin-Simmons University, Technology Services
Ph: (325) 670-1506 Fx: (325) 670-1570
From: observium [mailto:observium-bounces@observium.org]
On Behalf Of Adam Armstrong
Sent: Saturday, September 05, 2015 5:32 AM
To: observium@observium.org
Subject: Re: [Observium] poller.php burns all available CPU in the last community edition
I see no appreciable decrease in poller usage after applying this modification, so perhaps it was fixing something that had already been fixed since the last CE
release.
adam.
Sent from
Mailbird
On 05/09/2015 00:47:51, Moore, Cameron <cmoore@hsutx.edu> wrote:
I noticed the high CPU usage a few weeks ago, but I haven't had time to track it down. Thanks for the hard work! I've incorporated your patch and will let the poller run a while to see how it goes.
--
Cameron Moore
-----Original Message-----
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Mark Martinec
Sent: Friday, September 04, 2015 5:20 PM
To: observium@observium.org
Subject: [Observium] poller.php burns all available CPU in the last community edition
After upgrading a somewhat outdated version Observium CE 0.13.10.4585 to CE 0.15.6.6430 (under PHP 5.5.28), the poller.php started burning excessive CPU: eight running pollers on an 8-CPU machine started burning 100% of available CPU time instead of a few percents as before.
They somehow still managed to get their job done, but just barely so in five minutes, and often missing the time slot.
While watching the behaviour for a day and learning PHP profiling along the way, I now have an explanation and a solution. Apology if this has been reported before, couldn't find any such report.
The problem is in a function external_exec() in includes/common(.inc).php, which has been rewritten some time between these two versions, now switching to a common loop waiting with a stream_select() on two pipes (stdout and stderr) from a spawned snmp*walk process.
The documentation on stream_select helps understanding the issue:
http://php.net/manual/en/function.stream-select.php
read
The streams listed in the read array will be watched to see if
characters become available for reading (more precisely, to see
if a read will not block - in particular, a stream resource is
also ready on end-of-file, in which case an fread() will return
a zero length string).
So what happens is that when a spawned snmp* command closes its stderr but not yet its stdout, the external_exec() goes in a spin:
feof($pipes[1]) is false but stream is not yet ready for reading, however the feof($pipes[2]) is true and hence the stream_select() returns right away and reports $pipes[2] as ready. Then the
fread($pipes[2]) is called, returns an empty string (because stderr is at eof), and the whole loop busily goes around, until the time the snmp command happens to terminate.
Here is a fix:
--- includes/common.inc.php.ori 2015-09-04 20:29:39.561244285 +0200
+++ includes/common.inc.php 2015-09-04 20:30:07.617242602 +0200
@@ -618,9 +618,13 @@
{
$start = microtime(TRUE);
- //while ($status['running'] !== FALSE)
- while (feof($pipes[1]) === FALSE || feof($pipes[2]) === FALSE)
+ while (TRUE)
{
+ $read = array();
+ if (!feof($pipes[1])) { $read[] = $pipes[1]; }
+ if (!feof($pipes[2])) { $read[] = $pipes[2]; }
+ if (empty($read)) { break; }
+
stream_select(
- $read = array($pipes[1], $pipes[2]),
+ $read,
$write = NULL,
$except = NULL,
Regards
Mark
_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium