Observium stops displaying stats for all monitored hosts for a period of time
Hello Team, It seems Observium stops displaying stats for all (most) monitored hosts for a period of time and then back to normal.!!! I've attached a few screenshots. Has this behavior been documented in the past? If not what is the best way to debug this.
Observium ver 24.8.13606 Running on a dedicated server. 12 Core Xeon Proc 64 GB RAM Raided SSD drives.
Thank you
Kevin Riahi
Was your poller still running during that same time period? Click on the gear and select polling information. You should see graphs of polling statistics.
-- Tim
On Tue, Aug 20, 2024 at 12:21 PM Kevin Riahi via observium < observium@lists.observium.org> wrote:
Hello Team, It seems Observium stops displaying stats for all (most) monitored hosts for a period of time and then back to normal.!!! I've attached a few screenshots. Has this behavior been documented in the past? If not what is the best way to debug this.
Observium ver 24.8.13606 Running on a dedicated server. 12 Core Xeon Proc 64 GB RAM Raided SSD drives.
Thank youKevin Riahi
observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
Tim, Yes it seems so. A closer look also shows other shorter periods when this happens. Please see attached. The large gap seems to happen between 06:45 and 08:10 and I don't see that here.
Thank you
Kevin Riahi
From: "Timothy Stoddard" testoddard@ualr.edu To: "Observium" observium@lists.observium.org Cc: "Kevin Riahi" kevin.riahi@envoi.com Sent: Tuesday, August 20, 2024 1:38:24 PM Subject: Re: [Observium] Observium stops displaying stats for all monitored hosts for a period of time
Was your poller still running during that same time period? Click on the gear and select polling information. You should see graphs of polling statistics.
a) if you're polling through a firewall, it's probably the firewall being useless b) check the cpu/memory/etc usage of the observium server during these periods
Does the large gap coincide with when devices are being discovered?
24 poller threads is not a lot, but you don't say how many devices you have or how long polling usually takes.
In general the most common things that cause this are:
a) firewall rate limits blocking communication b) mysql connection limits being hit preventing poller from running c) excessive pollers being executed preventing anything from completing for a period of time
c) will be visible on cpu/memory/poller graphs b) will be visible on mysql connection graphs and probably in logs
Are the devices being marked as down? For what reason? ping or snmp?
adam.
Kevin Riahi via observium wrote on 2024-08-20 18:58:
Tim, Yes it seems so. A closer look also shows other shorter periods when this happens. Please see attached. The large gap seems to happen between 06:45 and 08:10 and I don't see that here.
Thank you
Kevin Riahi
*From: *"Timothy Stoddard" testoddard@ualr.edu *To: *"Observium" observium@lists.observium.org *Cc: *"Kevin Riahi" kevin.riahi@envoi.com *Sent: *Tuesday, August 20, 2024 1:38:24 PM *Subject: *Re: [Observium] Observium stops displaying stats for all monitored hosts for a period of time
Was your poller still running during that same time period? Click on the gear and select polling information. You should see graphs of polling statistics.
-- Tim
On Tue, Aug 20, 2024 at 12:21 PM Kevin Riahi via observium <observium@lists.observium.org mailto:observium@lists.observium.org> wrote:
Hello Team, It seems Observium stops displaying stats for all (most) monitored hosts for a period of time and then back to normal.!!! I've attached a few screenshots. Has this behavior been documented in the past? If not what is the best way to debug this. Observium ver 24.8.13606 Running on a dedicated server. 12 Core Xeon Proc 64 GB RAM Raided SSD drives. Thank you Kevin Riahi _______________________________________________ observium mailing list -- observium@lists.observium.org <mailto:observium@lists.observium.org> To unsubscribe send an email to observium-leave@lists.observium.org <mailto:observium-leave@lists.observium.org>
observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
participants (3)
-
Adam Armstrong -
Kevin Riahi -
Timothy Stoddard