After upgrading a somewhat outdated version Observium CE 0.13.10.4585 to CE 0.15.6.6430 (under PHP 5.5.28), the poller.php started burning excessive CPU: eight running pollers on an 8-CPU machine started burning 100% of available CPU time instead of a few percents as before. They somehow still managed to get their job done, but just barely so in five minutes, and often missing the time slot.
While watching the behaviour for a day and learning PHP profiling along the way, I now have an explanation and a solution. Apology if this has been reported before, couldn't find any such report.
The problem is in a function external_exec() in includes/common(.inc).php, which has been rewritten some time between these two versions, now switching to a common loop waiting with a stream_select() on two pipes (stdout and stderr) from a spawned snmp*walk process.
The documentation on stream_select helps understanding the issue:
http://php.net/manual/en/function.stream-select.php
read The streams listed in the read array will be watched to see if characters become available for reading (more precisely, to see if a read will not block - in particular, a stream resource is also ready on end-of-file, in which case an fread() will return a zero length string).
So what happens is that when a spawned snmp* command closes its stderr but not yet its stdout, the external_exec() goes in a spin:
feof($pipes[1]) is false but stream is not yet ready for reading, however the feof($pipes[2]) is true and hence the stream_select() returns right away and reports $pipes[2] as ready. Then the fread($pipes[2]) is called, returns an empty string (because stderr is at eof), and the whole loop busily goes around, until the time the snmp command happens to terminate.
Here is a fix:
--- includes/common.inc.php.ori 2015-09-04 20:29:39.561244285 +0200 +++ includes/common.inc.php 2015-09-04 20:30:07.617242602 +0200 @@ -618,9 +618,13 @@ { $start = microtime(TRUE); - //while ($status['running'] !== FALSE) - while (feof($pipes[1]) === FALSE || feof($pipes[2]) === FALSE) + while (TRUE) { + $read = array(); + if (!feof($pipes[1])) { $read[] = $pipes[1]; } + if (!feof($pipes[2])) { $read[] = $pipes[2]; } + if (empty($read)) { break; } + stream_select( - $read = array($pipes[1], $pipes[2]), + $read, $write = NULL, $except = NULL,
Regards Mark