Hi
We have one Observium server looking at
* 476 devices (mainly switches, some firewalls, some WAN routers) * 3666 ports * 8692 sensors
The server has 6 cores (~70% idle, ~3% iowait) and 8 GB memory. We have configured 15 poller threads with max 6 allowed wrapper processes.
The server is struggling to keep the polling cycle below 300 seconds (averaging 376 secs).
Is this workload realistic for one server, and if so where do you suggest we tune? Or do we need to use a distributed setup?
Lars ________________________________ Disclaimer: This e-mail, including any attachments, is for the intended recipient only. If you have received this e-mail by mistake please notify the sender immediately by return e-mail and delete this e-mail and any attachments, without opening the attachments, from your system. Access, disclosure, copying, distribution or reliance on any part of this e-mail by anyone else is prohibited. This e-mail is confidential and may be legally privileged. Chr. Hansen does not represent and/or warrant that the information sent and/or received by or with this e-mail is correct and does not accept any liability for damages related thereto. https://www.chr-hansen.com/en/legal-notice
Just find some more/faster cores.
The per-core speed has an impact too because of MySQL.
Don't worry too much if a couple of very slow devices are sluggish and hang around over the 300 seconds. You're wanting to spread the majority of devices over the 300 seconds.
Adam.
From: observium observium-bounces@observium.org On Behalf Of Lars Joergensen via observium Sent: 18 May 2021 20:32 To: Observium observium@observium.org Cc: Lars Joergensen DKLARJ@chr-hansen.com Subject: [Observium] Performance tuning
Hi
We have one Observium server looking at
* 476 devices (mainly switches, some firewalls, some WAN routers) * 3666 ports * 8692 sensors
The server has 6 cores (~70% idle, ~3% iowait) and 8 GB memory. We have configured 15 poller threads with max 6 allowed wrapper processes.
The server is struggling to keep the polling cycle below 300 seconds (averaging 376 secs).
Is this workload realistic for one server, and if so where do you suggest we tune? Or do we need to use a distributed setup?
Lars
_____
Disclaimer: This e-mail, including any attachments, is for the intended recipient only. If you have received this e-mail by mistake please notify the sender immediately by return e-mail and delete this e-mail and any attachments, without opening the attachments, from your system. Access, disclosure, copying, distribution or reliance on any part of this e-mail by anyone else is prohibited. This e-mail is confidential and may be legally privileged. Chr. Hansen does not represent and/or warrant that the information sent and/or received by or with this e-mail is correct and does not accept any liability for damages related thereto. https://www.chr-hansen.com/en/legal-notice
Hi Lars,
try to increase the number of pollers if possible. You said that CPU is 70% idle. Another way would be checking which modules are activated. Sometimes you do not need all information i.e. FDB/ARP. On poller information site you can see which device and module takes x seconds to complete.
Mit freundlichem Gruß wilhelm.tel GmbH
Denis Klimek Carrier Manager & Professional Network Engineer IP-Systemtechnik Tel: +49 (0) 40 / 521 04 - 1049 -> Work From Home Mobil: +49 (0) 151 / 652 219 06
dklimek@stadtwerke-norderstedt.de www.wilhelm-tel.de
______________________________________________
[cid:image004.png@01D74C91.8B026D80]
Postanschrift: wilhelm.tel GmbH Heidbergstraße 101-111 22846 Norderstedt
Geschäftsführer: Jens Seedorff, Theo Weirich Vorsitzender des Aufsichtsrats: Christoph Mendel Handelsregister: HRB 4216 NO, Amtsgericht Kiel Umsatzsteuer ID: DE 81 299 7663
[ig]https://www.instagram.com/azubiteam/[fb]https://www.facebook.com/wilhelmtel.norderstedt/
Von: observium observium-bounces@observium.org Im Auftrag von Adam Armstrong via observium Gesendet: Dienstag, 18. Mai 2021 21:51 An: 'Observium' observium@observium.org Cc: Adam Armstrong adama@observium.org Betreff: Re: [Observium] Performance tuning
Just find some more/faster cores.
The per-core speed has an impact too because of MySQL.
Don't worry too much if a couple of very slow devices are sluggish and hang around over the 300 seconds. You're wanting to spread the majority of devices over the 300 seconds.
Adam.
From: observium <observium-bounces@observium.orgmailto:observium-bounces@observium.org> On Behalf Of Lars Joergensen via observium Sent: 18 May 2021 20:32 To: Observium <observium@observium.orgmailto:observium@observium.org> Cc: Lars Joergensen <DKLARJ@chr-hansen.commailto:DKLARJ@chr-hansen.com> Subject: [Observium] Performance tuning
Hi
We have one Observium server looking at
* 476 devices (mainly switches, some firewalls, some WAN routers) * 3666 ports * 8692 sensors
The server has 6 cores (~70% idle, ~3% iowait) and 8 GB memory. We have configured 15 poller threads with max 6 allowed wrapper processes.
The server is struggling to keep the polling cycle below 300 seconds (averaging 376 secs).
Is this workload realistic for one server, and if so where do you suggest we tune? Or do we need to use a distributed setup?
Lars ________________________________ Disclaimer: This e-mail, including any attachments, is for the intended recipient only. If you have received this e-mail by mistake please notify the sender immediately by return e-mail and delete this e-mail and any attachments, without opening the attachments, from your system. Access, disclosure, copying, distribution or reliance on any part of this e-mail by anyone else is prohibited. This e-mail is confidential and may be legally privileged. Chr. Hansen does not represent and/or warrant that the information sent and/or received by or with this e-mail is correct and does not accept any liability for damages related thereto. https://www.chr-hansen.com/en/legal-notice
Hi Lars, we had a similar problem but in our case we could identify that certain devices were always slow to respond. I created a group called 'Slow-Responders' to contain these devices. The poller-wrapper.py script allows you to include or exclude groups (by group-id not name). You can also play around with the number of worker threads, if you have a lot of workers waiting for a response, you might as well use up some of that idle time. Our cron job entries look like:
*/5 * * * * root /opt/observium/poller-wrapper.py -w 96 --exclude-groups 225 ... */5 * * * * root /opt/observium/poller-wrapper.py -w 32 --include-groups 225 ...
Hope this helps, Michael.
On Tue, 18 May 2021 at 20:32, Lars Joergensen via observium < observium@observium.org> wrote:
Hi
We have one Observium server looking at
- 476 devices (mainly switches, some firewalls, some WAN routers)
- 3666 ports
- 8692 sensors
The server has 6 cores (~70% idle, ~3% iowait) and 8 GB memory. We have configured 15 poller threads with max 6 allowed wrapper processes.
The server is struggling to keep the polling cycle below 300 seconds (averaging 376 secs).
Is this workload realistic for one server, and if so where do you suggest we tune? Or do we need to use a distributed setup?
Lars
Disclaimer: This e-mail, including any attachments, is for the intended recipient only. If you have received this e-mail by mistake please notify the sender immediately by return e-mail and delete this e-mail and any attachments, without opening the attachments, from your system. Access, disclosure, copying, distribution or reliance on any part of this e-mail by anyone else is prohibited. This e-mail is confidential and may be legally privileged. Chr. Hansen does not represent and/or warrant that the information sent and/or received by or with this e-mail is correct and does not accept any liability for damages related thereto. https://www.chr-hansen.com/en/legal-notice _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Michel,
Nice! So you can group devices by type (fast/not so fast/slow), is there a specific ID or we can use a random one?
Thank you
________________________________ From: observium observium-bounces@observium.org on behalf of Michael Kelly via observium observium@observium.org Sent: Wednesday, May 19, 2021 7:32 pm To: Observium Cc: Michael Kelly Subject: Re: [Observium] Performance tuning
EXTERNAL EMAIL
Hi Lars, we had a similar problem but in our case we could identify that certain devices were always slow to respond. I created a group called 'Slow-Responders' to contain these devices. The poller-wrapper.py script allows you to include or exclude groups (by group-id not name). You can also play around with the number of worker threads, if you have a lot of workers waiting for a response, you might as well use up some of that idle time. Our cron job entries look like:
*/5 * * * * root /opt/observium/poller-wrapper.py -w 96 --exclude-groups 225 ... */5 * * * * root /opt/observium/poller-wrapper.py -w 32 --include-groups 225 ...
Hope this helps, Michael.
On Tue, 18 May 2021 at 20:32, Lars Joergensen via observium <observium@observium.orgmailto:observium@observium.org> wrote: Hi
We have one Observium server looking at
* 476 devices (mainly switches, some firewalls, some WAN routers) * 3666 ports * 8692 sensors
The server has 6 cores (~70% idle, ~3% iowait) and 8 GB memory. We have configured 15 poller threads with max 6 allowed wrapper processes.
The server is struggling to keep the polling cycle below 300 seconds (averaging 376 secs).
Is this workload realistic for one server, and if so where do you suggest we tune? Or do we need to use a distributed setup?
Lars ________________________________ Disclaimer: This e-mail, including any attachments, is for the intended recipient only. If you have received this e-mail by mistake please notify the sender immediately by return e-mail and delete this e-mail and any attachments, without opening the attachments, from your system. Access, disclosure, copying, distribution or reliance on any part of this e-mail by anyone else is prohibited. This e-mail is confidential and may be legally privileged. Chr. Hansen does not represent and/or warrant that the information sent and/or received by or with this e-mail is correct and does not accept any liability for damages related thereto. https://www.chr-hansen.com/en/legal-noticehttps://urldefense.proofpoint.com/v2/url?u=https-3A__www.chr-2Dhansen.com_en_legal-2Dnotice&d=DwMFaQ&c=PzM68gSF_5r1R7BCE75oeA&r=iA-_Ejzboh_TtAmhokShXulssWr_YvG_3mK7Gpf2nkg&m=uJ7iDt58CBrIwN5sm_joqtLSv_2ZNIh0eHIDUSLnBdY&s=TiVz4X1tYkGf6xuQnbzNhJjYk-FwTIDUSpVPiKInT2c&e= _______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observiumhttps://urldefense.proofpoint.com/v2/url?u=http-3A__postman.memetic.org_cgi-2Dbin_mailman_listinfo_observium&d=DwMFaQ&c=PzM68gSF_5r1R7BCE75oeA&r=iA-_Ejzboh_TtAmhokShXulssWr_YvG_3mK7Gpf2nkg&m=uJ7iDt58CBrIwN5sm_joqtLSv_2ZNIh0eHIDUSLnBdY&s=iHb4G3Ef_nuCoYTBIpS8nJPG1v7sc9cLRymPA2FHrXg&e=
PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL. CDW Limited is a Company registered in England and Wales. This email is sent for and on behalf of CDW Finance Holdings Limited. CDW Finance Holdings Limited is the UK holding company for CDW Limited trading as CDW. The registered office for CDW Finance Holdings Limited (Company Number 05872067) and its UK group undertakings is 1 New Change, London, EC4M 9AF. VAT Registration 902 1949 39. Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than CDW or the sender of its existence.>/p> For more information about CDW please visit https://www.uk.cdw.com
You use the ID of the group you create to put the slow devices in :)
Adam.
From: observium observium-bounces@observium.org On Behalf Of Serghei Golipad via observium Sent: 19 May 2021 19:49 To: Observium observium@observium.org Cc: Serghei Golipad s.golipad@uk.cdw.com Subject: Re: [Observium] Performance tuning
Hi Michel,
Nice! So you can group devices by type (fast/not so fast/slow), is there a specific ID or we can use a random one?
Thank you
_____
From: observium <observium-bounces@observium.org mailto:observium-bounces@observium.org > on behalf of Michael Kelly via observium <observium@observium.org mailto:observium@observium.org > Sent: Wednesday, May 19, 2021 7:32 pm To: Observium Cc: Michael Kelly Subject: Re: [Observium] Performance tuning
EXTERNAL EMAIL
Hi Lars,
we had a similar problem but in our case we could identify that certain devices were always slow to respond.
I created a group called 'Slow-Responders' to contain these devices.
The poller-wrapper.py script allows you to include or exclude groups (by group-id not name).
You can also play around with the number of worker threads, if you have a lot of workers waiting for a response, you might as well use up some of that idle time.
Our cron job entries look like:
*/5 * * * * root /opt/observium/poller-wrapper.py -w 96 --exclude-groups 225 ...
*/5 * * * * root /opt/observium/poller-wrapper.py -w 32 --include-groups 225 ...
Hope this helps,
Michael.
On Tue, 18 May 2021 at 20:32, Lars Joergensen via observium <observium@observium.org mailto:observium@observium.org > wrote:
Hi
We have one Observium server looking at
* 476 devices (mainly switches, some firewalls, some WAN routers) * 3666 ports * 8692 sensors
The server has 6 cores (~70% idle, ~3% iowait) and 8 GB memory. We have configured 15 poller threads with max 6 allowed wrapper processes.
The server is struggling to keep the polling cycle below 300 seconds (averaging 376 secs).
Is this workload realistic for one server, and if so where do you suggest we tune? Or do we need to use a distributed setup?
Lars
_____
Disclaimer: This e-mail, including any attachments, is for the intended recipient only. If you have received this e-mail by mistake please notify the sender immediately by return e-mail and delete this e-mail and any attachments, without opening the attachments, from your system. Access, disclosure, copying, distribution or reliance on any part of this e-mail by anyone else is prohibited. This e-mail is confidential and may be legally privileged. Chr. Hansen does not represent and/or warrant that the information sent and/or received by or with this e-mail is correct and does not accept any liability for damages related thereto. https://www.chr-hansen.com/en/legal-notice https://urldefense.proofpoint.com/v2/url?u=https-3A__www.chr-2Dhansen.com_e n_legal-2Dnotice&d=DwMFaQ&c=PzM68gSF_5r1R7BCE75oeA&r=iA-_Ejzboh_TtAmhokShXul ssWr_YvG_3mK7Gpf2nkg&m=uJ7iDt58CBrIwN5sm_joqtLSv_2ZNIh0eHIDUSLnBdY&s=TiVz4X1 tYkGf6xuQnbzNhJjYk-FwTIDUSpVPiKInT2c&e=
_______________________________________________ observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium https://urldefense.proofpoint.com/v2/url?u=http-3A__postman.memetic.org_cgi -2Dbin_mailman_listinfo_observium&d=DwMFaQ&c=PzM68gSF_5r1R7BCE75oeA&r=iA-_Ej zboh_TtAmhokShXulssWr_YvG_3mK7Gpf2nkg&m=uJ7iDt58CBrIwN5sm_joqtLSv_2ZNIh0eHID USLnBdY&s=iHb4G3Ef_nuCoYTBIpS8nJPG1v7sc9cLRymPA2FHrXg&e=
PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL.
CDW Limited is a Company registered in England and Wales. This email is sent for and on behalf of CDW Finance Holdings Limited. CDW Finance Holdings Limited is the UK holding company for CDW Limited trading as CDW. The registered office for CDW Finance Holdings Limited (Company Number 05872067) and its UK group undertakings is 1 New Change, London, EC4M 9AF. VAT Registration 902 1949 39. Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than CDW or the sender of its existence.>/p>
For more information about CDW please visit https://www.uk.cdw.com https://www.uk.cdw.com/
Would be nice to then dedicate slow cores to those ;D
________________________________ From: observium observium-bounces@observium.org on behalf of Adam Armstrong via observium observium@observium.org Sent: Wednesday, May 19, 2021 7:49:25 PM To: 'Observium' observium@observium.org Cc: Adam Armstrong adama@observium.org Subject: Re: [Observium] Performance tuning
EXTERNAL EMAIL
You use the ID of the group you create to put the slow devices in :)
Adam.
From: observium observium-bounces@observium.org On Behalf Of Serghei Golipad via observium Sent: 19 May 2021 19:49 To: Observium observium@observium.org Cc: Serghei Golipad s.golipad@uk.cdw.com Subject: Re: [Observium] Performance tuning
Hi Michel,
Nice! So you can group devices by type (fast/not so fast/slow), is there a specific ID or we can use a random one?
Thank you
________________________________
From: observium <observium-bounces@observium.orgmailto:observium-bounces@observium.org> on behalf of Michael Kelly via observium <observium@observium.orgmailto:observium@observium.org> Sent: Wednesday, May 19, 2021 7:32 pm To: Observium Cc: Michael Kelly Subject: Re: [Observium] Performance tuning
EXTERNAL EMAIL
Hi Lars,
we had a similar problem but in our case we could identify that certain devices were always slow to respond.
I created a group called 'Slow-Responders' to contain these devices.
The poller-wrapper.py script allows you to include or exclude groups (by group-id not name).
You can also play around with the number of worker threads, if you have a lot of workers waiting for a response, you might as well use up some of that idle time.
Our cron job entries look like:
*/5 * * * * root /opt/observium/poller-wrapper.py -w 96 --exclude-groups 225 ...
*/5 * * * * root /opt/observium/poller-wrapper.py -w 32 --include-groups 225 ...
Hope this helps,
Michael.
On Tue, 18 May 2021 at 20:32, Lars Joergensen via observium <observium@observium.orgmailto:observium@observium.org> wrote:
Hi
We have one Observium server looking at
* 476 devices (mainly switches, some firewalls, some WAN routers) * 3666 ports * 8692 sensors
The server has 6 cores (~70% idle, ~3% iowait) and 8 GB memory. We have configured 15 poller threads with max 6 allowed wrapper processes.
The server is struggling to keep the polling cycle below 300 seconds (averaging 376 secs).
Is this workload realistic for one server, and if so where do you suggest we tune? Or do we need to use a distributed setup?
Lars
________________________________
Disclaimer: This e-mail, including any attachments, is for the intended recipient only. If you have received this e-mail by mistake please notify the sender immediately by return e-mail and delete this e-mail and any attachments, without opening the attachments, from your system. Access, disclosure, copying, distribution or reliance on any part of this e-mail by anyone else is prohibited. This e-mail is confidential and may be legally privileged. Chr. Hansen does not represent and/or warrant that the information sent and/or received by or with this e-mail is correct and does not accept any liability for damages related thereto. https://www.chr-hansen.com/en/legal-noticehttps://urldefense.proofpoint.com/v2/url?u=https-3A__www.chr-2Dhansen.com_en_legal-2Dnotice&d=DwMFaQ&c=PzM68gSF_5r1R7BCE75oeA&r=iA-_Ejzboh_TtAmhokShXulssWr_YvG_3mK7Gpf2nkg&m=uJ7iDt58CBrIwN5sm_joqtLSv_2ZNIh0eHIDUSLnBdY&s=TiVz4X1tYkGf6xuQnbzNhJjYk-FwTIDUSpVPiKInT2c&e=
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observiumhttps://urldefense.proofpoint.com/v2/url?u=http-3A__postman.memetic.org_cgi-2Dbin_mailman_listinfo_observium&d=DwMFaQ&c=PzM68gSF_5r1R7BCE75oeA&r=iA-_Ejzboh_TtAmhokShXulssWr_YvG_3mK7Gpf2nkg&m=uJ7iDt58CBrIwN5sm_joqtLSv_2ZNIh0eHIDUSLnBdY&s=iHb4G3Ef_nuCoYTBIpS8nJPG1v7sc9cLRymPA2FHrXg&e=
PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL.
CDW Limited is a Company registered in England and Wales. This email is sent for and on behalf of CDW Finance Holdings Limited. CDW Finance Holdings Limited is the UK holding company for CDW Limited trading as CDW. The registered office for CDW Finance Holdings Limited (Company Number 05872067) and its UK group undertakings is 1 New Change, London, EC4M 9AF. VAT Registration 902 1949 39. Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than CDW or the sender of its existence.>/p>
For more information about CDW please visit https://www.uk.cdw.comhttps://www.uk.cdw.com/
participants (5)
-
Adam Armstrong -
Klimek, Denis -
Lars Joergensen -
Michael Kelly -
Serghei Golipad