list,
I have been working all angles to scale my observium CE instance running well but seem to be failing. By failing I am referring to the classic gaps in graphs for all my devices. I am running on esxi with 16 cores @ 2.40GHz, 16G of ram, SSD SAS Raid and continue to have gaps. Most recently I have pushed sql to its own instance in hopes of working around potential IO issues but that didnt help.
What I do see is that my ram use is very high, I am guessing from the discovery process twice a day. When my ram use is lower throughout the day, I have no gaps. I am monitoring 6000ports on 190 devices. Should I be using north of 16G of ram in my env? I am happy to add more but all my reading indicates that observium is not RAM intense. My last attempt will be to move the RRDs to a ram disk but my RRDs are almost 16G at this time which I am guessing is part of the problem. Maybe it is time to clean house.
Any insight would be appreciated.
db
I have 22840 ports monitored on 540 devices and my poller completes in under 300 seconds. Make sure you are using rrdcached as it greatly helps with the handling of the rrd files.
Thanks, -- Tim
On Tue, Sep 6, 2022 at 1:14 PM Daniel Benson via observium < observium@lists.observium.org> wrote:
list,
I have been working all angles to scale my observium CE instance running well but seem to be failing. By failing I am referring to the classic gaps in graphs for all my devices. I am running on esxi with 16 cores @ 2.40GHz, 16G of ram, SSD SAS Raid and continue to have gaps. Most recently I have pushed sql to its own instance in hopes of working around potential IO issues but that didnt help.
What I do see is that my ram use is very high, I am guessing from the discovery process twice a day. When my ram use is lower throughout the day, I have no gaps. I am monitoring 6000ports on 190 devices. Should I be using north of 16G of ram in my env? I am happy to add more but all my reading indicates that observium is not RAM intense. My last attempt will be to move the RRDs to a ram disk but my RRDs are almost 16G at this time which I am guessing is part of the problem. Maybe it is time to clean house.
Any insight would be appreciated.
db _______________________________________________ observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
Adjust poller threads, check if unneeded modules are acitivated i.e. fdb-tables, use rrdcache, optimize mysql server. We are running Observium on a VM with 24 cores (40% used), 64gb ram (70% used) with 916 devices, 33k ports and poller is finishing still below 300 seconds 😉 Disable down devices if you have to many of them, they are blocking polling threads until timeout is reached.
Mit freundlichem Gruß wilhelm.tel GmbH
Denis Klimek Carrier Manager & Professional Network Engineer IP-Systemtechnik Tel: +49 (0) 40 / 521 04 – 1049 -> Work From Home Mobil: +49 (0) 151 / 652 219 06
dklimek@stadtwerke-norderstedt.de www.wilhelm-tel.de
______________________________________________
[cid:image004.png@01D8C29A.51AD3E90]
Postanschrift: wilhelm.tel GmbH Heidbergstraße 101-111 22846 Norderstedt
Geschäftsführer: Jens Seedorff, Theo Weirich Vorsitzender des Aufsichtsrats: Christoph Mendel Handelsregister: HRB 4216 NO, Amtsgericht Kiel Umsatzsteuer ID: DE 81 299 7663
[ig]https://www.instagram.com/azubiteam/[fb]https://www.facebook.com/wilhelmtel.norderstedt/
Von: Timothy Stoddard via observium observium@lists.observium.org Gesendet: Dienstag, 6. September 2022 20:23 An: Observium observium@lists.observium.org Cc: Daniel Benson danmbenson@gmail.com; Timothy Stoddard testoddard@ualr.edu Betreff: [Observium] Re: Graph Gaps and Scaling questions
I have 22840 ports monitored on 540 devices and my poller completes in under 300 seconds. Make sure you are using rrdcached as it greatly helps with the handling of the rrd files.
Thanks, -- Tim
On Tue, Sep 6, 2022 at 1:14 PM Daniel Benson via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> wrote: list,
I have been working all angles to scale my observium CE instance running well but seem to be failing. By failing I am referring to the classic gaps in graphs for all my devices. I am running on esxi with 16 cores @ 2.40GHz, 16G of ram, SSD SAS Raid and continue to have gaps. Most recently I have pushed sql to its own instance in hopes of working around potential IO issues but that didnt help.
What I do see is that my ram use is very high, I am guessing from the discovery process twice a day. When my ram use is lower throughout the day, I have no gaps. I am monitoring 6000ports on 190 devices. Should I be using north of 16G of ram in my env? I am happy to add more but all my reading indicates that observium is not RAM intense. My last attempt will be to move the RRDs to a ram disk but my RRDs are almost 16G at this time which I am guessing is part of the problem. Maybe it is time to clean house.
Any insight would be appreciated.
db _______________________________________________ observium mailing list -- observium@lists.observium.orgmailto:observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.orgmailto:observium-leave@lists.observium.org
Hallo
I'm trying to monitor the Disk I/O. If a disk goes into RO I noticed, that the counter is 0. This would be a good indicator.
With the checker wizard I couldn't find any options.
Is there another way?
Thank you in advanced.
Regards Walter
participants (4)
-
Daniel Benson -
Klimek, Denis -
Timothy Stoddard -
walter＠onda.ch