Hi!

No answer from anyone so far, but I did get a bit closer to the issue today. The lockup happens exactly at the time I hover the mouse over the interface name/IP in the picture attached.

Hovering above will normally create a popup with the traffic information etc., but instead it locks up the entire http session.

In Chrome when this happens the status bar at the bottom of Chrome says "Waiting for socket". I tried to open a parallel browser (Safari & Firefox) and these can initially connect to Observium (new session) but if I go back to the same interface and hover the mouse above it, also they lock up, and eventually no sessions are left on the webserver and it will not respond to anything.

I saw elsewhere in a forum that the "Waiting for socket" can also relate to Chrome running out of sockets, but since I get the same error on two other browsers (with no other pages open), and since Apache is clearly locking up on something, I guess the browser itself is not the issue.

So it appears the issue is clearly related to the Observium entity_popup.php script.

Funny thing though is that in most cases this script does work, but it seems that sometimes it doesn't.

/Jesper

On Sun, Nov 6, 2016 at 1:18 PM Jesper Frank Nemholt <jfn@dassic.com> wrote:

Just as an add-on, an strace on the Apache processes give this :

[root@observium-1-vm ~]# strace -p 9221

Process 9221 attached

flock(13, LOCK_EX

[root@observium-1-vm ~]# strace -p 9157

Process 9157 attached

select(17, [14 16], [], [], NULL

/Jesper

On Sun, Nov 6, 2016 at 12:56 PM Jesper Frank Nemholt <jfn@dassic.com> wrote:

Hi!

So related to this, I enabled server-status on Apache and here's what I see on the webserver when it locks up.

Based upon my past experience it seems it's somehow related to the script /ajax/entity_popup.php as I've seen it lock up several times when calling that script.

Any clues ?

Apache Server Status for 1.2.3.4 (via 1.2.3.4)

Server Version: Apache/2.4.6 (CentOS) PHP/5.4.16

Server MPM: prefork

Server Built: Jul 18 2016 15:30:14

---

Current Time: Sunday, 06-Nov-2016 12:51:50 PST

Restart Time: Friday, 04-Nov-2016 21:28:13 PDT

Parent Server Config. Generation: 1

Parent Server MPM Generation: 0

Server uptime: 1 day 16 hours 23 minutes 37 seconds

Server load: 2.27 2.89 3.05

Total accesses: 6840 - Total Traffic: 165.9 MB

CPU Usage: u954.14 s117.84 cu292.79 cs133.22 - 1.03% CPU load

.047 requests/sec - 1196 B/second - 24.8 kB/request

9 requests currently being processed, 20 idle workers

______W_W__W_W_WW___W___WW___...................................
................................................................
................................................................
........

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv	PID	Acc	M	CPU	SS	Req	Conn	Child	Slot	Client	VHost	Request
0-0	14261	0/257/257	_	53.97	371	178	0.0	5.66	5.66	10.31.212.162	observium.xyz.com:80	NULL
1-0	14262	0/286/286	_	59.05	284	253	0.0	6.46	6.46	10.31.212.162	observium.xyz.com:80	NULL
2-0	14263	0/256/256	_	51.48	284	263	0.0	6.23	6.23	10.31.212.162	observium.xyz.com:80	NULL
3-0	14266	0/235/235	_	54.04	371	218	0.0	6.63	6.63	10.31.212.162	observium.xyz.com:80	NULL
4-0	14268	0/226/226	_	51.20	24	517	0.0	6.71	6.71	10.31.212.162
5-0	14270	0/229/229	_	56.39	284	420	0.0	6.45	6.45	10.31.212.162	observium.xyz.com:80	NULL
6-0	14272	8/305/305	W	69.47	139	0	75.7	7.03	7.03	10.31.212.162	observium.xyz.com:80	POST /ajax/entity_popup.php HTTP/1.1
7-0	14273	0/212/212	_	48.27	420	951	0.0	5.82	5.82	10.31.212.162	observium.xyz.com:80	NULL
8-0	14276	7/228/228	W	53.04	140	0	14.0	7.03	7.03	10.31.212.162	observium.xyz.com:80	GET /graph.php?type=port_bits&legend=yes&height=100&width=275&t
9-0	14278	0/235/235	_	52.99	24	449	0.0	6.66	6.66	10.31.212.162
10-0	15279	0/230/230	_	51.95	11	0	0.0	6.19	6.19	10.31.212.162	observium.xyz.com:80	NULL
11-0	16648	9/244/244	W	60.07	140	0	32.9	7.38	7.38	10.31.212.162	observium.xyz.com:80	GET /graph.php?type=port_bits&legend=yes&height=100&width=275&t
12-0	16649	0/225/225	_	48.25	9	194	0.0	5.43	5.43	10.31.212.162
13-0	17132	6/225/225	W	47.62	140	0	14.7	5.66	5.66	10.31.212.162	observium.xyz.com:80	POST /ajax/entity_popup.php HTTP/1.1
14-0	17133	0/278/278	_	55.62	24	1	0.0	5.50	5.50	10.31.212.162
15-0	17135	0/269/269	W	58.24	120	0	0.0	6.39	6.39	10.31.212.162	observium.xyz.com:80	GET /addhost/ HTTP/1.1
16-0	17136	0/353/353	W	69.36	97	0	0.0	5.48	5.48	10.31.212.162	observium.xyz.com:80	POST /ajax/entity_popup.php HTTP/1.1
17-0	18015	0/195/195	_	47.28	420	515	0.0	5.30	5.30	10.31.212.162	observium.xyz.com:80	NULL
18-0	18016	0/352/352	_	72.91	133	244	0.0	7.34	7.34	10.31.212.162	observium.xyz.com:80	NULL
19-0	18017	0/241/241	_	65.89	284	305	0.0	7.79	7.79	10.31.212.162	observium.xyz.com:80	NULL
20-0	18018	0/352/352	W	72.88	105	0	0.0	5.98	5.98	10.31.212.162	observium.xyz.com:80	GET /overview/ HTTP/1.1
21-0	18019	0/222/222	_	52.68	284	297	0.0	5.99	5.99	10.31.212.162	observium.xyz.com:80	NULL
22-0	18020	0/270/270	_	54.70	371	225	0.0	5.52	5.52	10.31.212.162	observium.xyz.com:80	NULL
23-0	18021	0/220/220	_	49.63	9	155	0.0	6.18	6.18	10.31.212.162
24-0	18022	5/266/266	W	59.03	0	0	49.3	7.15	7.15	10.31.212.162	observium.xyz.com:80	GET /server-status HTTP/1.1
25-0	16178	0/324/324	W	62.23	35	0	0.0	6.18	6.18	10.31.212.162	observium.xyz.com:80	GET / HTTP/1.1
26-0	8624	0/61/61	_	11.56	135	189	0.0	1.09	1.09	10.31.212.162	observium.xyz.com:80	NULL
27-0	11067	0/44/44	_	8.19	284	240	0.0	0.70	0.70	10.31.212.162	observium.xyz.com:80	NULL

---

Srv	Child Server number - generation
PID	OS process ID
Acc	Number of accesses this connection / this child / this slot
M	Mode of operation
CPU	CPU usage, number of seconds
SS	Seconds since beginning of most recent request
Req	Milliseconds required to process most recent request
Conn	Kilobytes transferred this connection
Child	Megabytes transferred this child
Slot	Total megabytes transferred this slot

On Thu, Oct 27, 2016 at 10:07 AM Jesper Frank Nemholt <jfn@dassic.com> wrote:

Hi!

The server is fairly idle and good on memory and CPU. Apache on this server (it's a QEMU KVM VM) is only serving Observium.

/Jesper

On Thu, Oct 27, 2016 at 10:04 AM Derek <dandenoth@gmail.com> wrote:

How's the server itself doing when it happens? If you run a top command, is Apache maxing out CPU or hogging memory? I had something similar happen when I was trying to configure SSL/TLS on my Apache server, which was running multiple virtual hosts for Observium, a SVN frontend for RANCID, NIPAP, and Smokeping.

Derek

On Thu, Oct 27, 2016 at 8:46 AM, Jesper Frank Nemholt <jfn@dassic.com> wrote:

Hi!

I've started to see that on specific actions from within the Observium GUI Apache will lock up. It seems random what action it is, but all Apache workers stop responding and it takes forever to restart Apache afterwards (5-10 min to do a service httpd restart).

Anyone had the same issue ?

I see no errors in the Apache logs and no errors in Observium logs, nor in other logs on the LInux server (CentOS 7.2). Observium polling still run as usual, so it's just the web interface stopping.

/Jesper

_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium