Hi!
So related to this, I enabled server-status on Apache and here's what I see on the webserver when it locks up.
Based upon my past experience it seems it's somehow related to the script /ajax/entity_popup.php as I've seen it lock up several times when calling that script.
Any clues ?
Apache Server Status for 1.2.3.4 (via 1.2.3.4)Server Version: Apache/2.4.6 (CentOS) PHP/5.4.16Server MPM: preforkServer Built: Jul 18 2016 15:30:14 ------------------------------ Current Time: Sunday, 06-Nov-2016 12:51:50 PSTRestart Time: Friday, 04-Nov-2016 21:28:13 PDTParent Server Config. Generation: 1Parent Server MPM Generation: 0Server uptime: 1 day 16 hours 23 minutes 37 secondsServer load: 2.27 2.89 3.05Total accesses: 6840 - Total Traffic: 165.9 MBCPU Usage: u954.14 s117.84 cu292.79 cs133.22 - 1.03% CPU load.047 requests/sec - 1196 B/second - 24.8 kB/request9 requests currently being processed, 20 idle workers
______W_W__W_W_WW___W___WW___................................... ................................................................ ................................................................ ........
Scoreboard Key: "*_*" Waiting for Connection, "*S*" Starting up, "*R*" Reading Request, "*W*" Sending Reply, "*K*" Keepalive (read), "*D*" DNS Lookup, "*C*" Closing connection, "*L*" Logging, "*G*" Gracefully finishing, "*I*" Idle cleanup of worker, "*.*" Open slot with no current process
SrvPIDAccMCPUSSReqConnChildSlotClientVHostRequest *0-0* 14261 0/257/257 _ 53.97 371 178 0.0 5.66 5.66 10.31.212.162 observium.xyz.com:80 NULL *1-0* 14262 0/286/286 _ 59.05 284 253 0.0 6.46 6.46 10.31.212.162 observium.xyz.com:80 NULL *2-0* 14263 0/256/256 _ 51.48 284 263 0.0 6.23 6.23 10.31.212.162 observium.xyz.com:80 NULL *3-0* 14266 0/235/235 _ 54.04 371 218 0.0 6.63 6.63 10.31.212.162 observium.xyz.com:80 NULL *4-0* 14268 0/226/226 _ 51.20 24 517 0.0 6.71 6.71 10.31.212.162 *5-0* 14270 0/229/229 _ 56.39 284 420 0.0 6.45 6.45 10.31.212.162 observium.xyz.com:80 NULL *6-0* 14272 8/305/305 *W* 69.47 139 0 75.7 7.03 7.03 10.31.212.162 observium.xyz.com:80 POST /ajax/entity_popup.php HTTP/1.1 *7-0* 14273 0/212/212 _ 48.27 420 951 0.0 5.82 5.82 10.31.212.162 observium.xyz.com:80 NULL *8-0* 14276 7/228/228 *W* 53.04 140 0 14.0 7.03 7.03 10.31.212.162 observium.xyz.com:80 GET /graph.php?type=port_bits&legend=yes&height=100&width=275&t *9-0* 14278 0/235/235 _ 52.99 24 449 0.0 6.66 6.66 10.31.212.162 *10-0* 15279 0/230/230 _ 51.95 11 0 0.0 6.19 6.19 10.31.212.162 observium.xyz.com:80 NULL *11-0* 16648 9/244/244 *W* 60.07 140 0 32.9 7.38 7.38 10.31.212.162 observium.xyz.com:80 GET /graph.php?type=port_bits&legend=yes&height=100&width=275&t *12-0* 16649 0/225/225 _ 48.25 9 194 0.0 5.43 5.43 10.31.212.162 *13-0* 17132 6/225/225 *W* 47.62 140 0 14.7 5.66 5.66 10.31.212.162 observium.xyz.com:80 POST /ajax/entity_popup.php HTTP/1.1 *14-0* 17133 0/278/278 _ 55.62 24 1 0.0 5.50 5.50 10.31.212.162 *15-0* 17135 0/269/269 *W* 58.24 120 0 0.0 6.39 6.39 10.31.212.162 observium.xyz.com:80 GET /addhost/ HTTP/1.1 *16-0* 17136 0/353/353 *W* 69.36 97 0 0.0 5.48 5.48 10.31.212.162 observium.xyz.com:80 POST /ajax/entity_popup.php HTTP/1.1 *17-0* 18015 0/195/195 _ 47.28 420 515 0.0 5.30 5.30 10.31.212.162 observium.xyz.com:80 NULL *18-0* 18016 0/352/352 _ 72.91 133 244 0.0 7.34 7.34 10.31.212.162 observium.xyz.com:80 NULL *19-0* 18017 0/241/241 _ 65.89 284 305 0.0 7.79 7.79 10.31.212.162 observium.xyz.com:80 NULL *20-0* 18018 0/352/352 *W* 72.88 105 0 0.0 5.98 5.98 10.31.212.162 observium.xyz.com:80 GET /overview/ HTTP/1.1 *21-0* 18019 0/222/222 _ 52.68 284 297 0.0 5.99 5.99 10.31.212.162 observium.xyz.com:80 NULL *22-0* 18020 0/270/270 _ 54.70 371 225 0.0 5.52 5.52 10.31.212.162 observium.xyz.com:80 NULL *23-0* 18021 0/220/220 _ 49.63 9 155 0.0 6.18 6.18 10.31.212.162 *24-0* 18022 5/266/266 *W* 59.03 0 0 49.3 7.15 7.15 10.31.212.162 observium.xyz.com:80 GET /server-status HTTP/1.1 *25-0* 16178 0/324/324 *W* 62.23 35 0 0.0 6.18 6.18 10.31.212.162 observium.xyz.com:80 GET / HTTP/1.1 *26-0* 8624 0/61/61 _ 11.56 135 189 0.0 1.09 1.09 10.31.212.162 observium.xyz.com:80 NULL *27-0* 11067 0/44/44 _ 8.19 284 240 0.0 0.70 0.70 10.31.212.162 observium.xyz.com:80 NULL
------------------------------ Srv Child Server number - generation PID OS process ID Acc Number of accesses this connection / this child / this slot M Mode of operation CPU CPU usage, number of seconds SS Seconds since beginning of most recent request Req Milliseconds required to process most recent request Conn Kilobytes transferred this connection Child Megabytes transferred this child Slot Total megabytes transferred this slot
On Thu, Oct 27, 2016 at 10:07 AM Jesper Frank Nemholt jfn@dassic.com wrote:
Hi!
The server is fairly idle and good on memory and CPU. Apache on this server (it's a QEMU KVM VM) is only serving Observium.
/Jesper
On Thu, Oct 27, 2016 at 10:04 AM Derek dandenoth@gmail.com wrote:
How's the server itself doing when it happens? If you run a top command, is Apache maxing out CPU or hogging memory? I had something similar happen when I was trying to configure SSL/TLS on my Apache server, which was running multiple virtual hosts for Observium, a SVN frontend for RANCID, NIPAP, and Smokeping.
Derek
On Thu, Oct 27, 2016 at 8:46 AM, Jesper Frank Nemholt jfn@dassic.com wrote:
Hi!
I've started to see that on specific actions from within the Observium GUI Apache will lock up. It seems random what action it is, but all Apache workers stop responding and it takes forever to restart Apache afterwards (5-10 min to do a service httpd restart).
Anyone had the same issue ?
I see no errors in the Apache logs and no errors in Observium logs, nor in other logs on the LInux server (CentOS 7.2). Observium polling still run as usual, so it's just the web interface stopping.
/Jesper
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium