I seem to remember we fairly quickly released a new CE due to some rrd process issue? Mike! Where are you! :)
On 05/09/2015 12:32, Adam Armstrong wrote:
I see no appreciable decrease in poller usage after applying this modification, so perhaps it was fixing something that had already been fixed since the last CE release.
adam.
Sent from Mailbird http://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm_campaign=sent-from-mailbird
On 05/09/2015 00:47:51, Moore, Cameron cmoore@hsutx.edu wrote:
I noticed the high CPU usage a few weeks ago, but I haven't had time to track it down. Thanks for the hard work! I've incorporated your patch and will let the poller run a while to see how it goes. -- Cameron Moore
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Mark Martinec Sent: Friday, September 04, 2015 5:20 PM To: observium@observium.org Subject: [Observium] poller.php burns all available CPU in the last community edition
After upgrading a somewhat outdated version Observium CE 0.13.10.4585 to CE 0.15.6.6430 (under PHP 5.5.28), the poller.php started burning excessive CPU: eight running pollers on an 8-CPU machine started burning 100% of available CPU time instead of a few percents as before. They somehow still managed to get their job done, but just barely so in five minutes, and often missing the time slot.
While watching the behaviour for a day and learning PHP profiling along the way, I now have an explanation and a solution. Apology if this has been reported before, couldn't find any such report.
The problem is in a function external_exec() in includes/common(.inc).php, which has been rewritten some time between these two versions, now switching to a common loop waiting with a stream_select() on two pipes (stdout and stderr) from a spawned snmp*walk process.
The documentation on stream_select helps understanding the issue:
http://php.net/manual/en/function.stream-select.php
read The streams listed in the read array will be watched to see if characters become available for reading (more precisely, to see if a read will not block - in particular, a stream resource is also ready on end-of-file, in which case an fread() will return a zero length string).
So what happens is that when a spawned snmp* command closes its stderr but not yet its stdout, the external_exec() goes in a spin:
feof($pipes[1]) is false but stream is not yet ready for reading, however the feof($pipes[2]) is true and hence the stream_select() returns right away and reports $pipes[2] as ready. Then the fread($pipes[2]) is called, returns an empty string (because stderr is at eof), and the whole loop busily goes around, until the time the snmp command happens to terminate.
Here is a fix:
--- includes/common.inc.php.ori 2015-09-04 20:29:39.561244285 +0200 +++ includes/common.inc.php 2015-09-04 20:30:07.617242602 +0200 @@ -618,9 +618,13 @@ { $start = microtime(TRUE);
- //while ($status['running'] !== FALSE)
- while (feof($pipes[1]) === FALSE || feof($pipes[2]) === FALSE)
- while (TRUE)
{
- $read = array();
- if (!feof($pipes[1])) { $read[] = $pipes[1]; }
- if (!feof($pipes[2])) { $read[] = $pipes[2]; }
- if (empty($read)) { break; }
stream_select(
- $read = array($pipes[1], $pipes[2]),
- $read,
$write = NULL, $except = NULL,
Regards Mark _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium