Hello all,
We're seeing holes on our graphs (example attached). This appears to be affecting all graphs in our installation on all devices. I know our connection is good (affecting LAN and WAN devices), but looking for any other ideas for correcting this? We have also tried increasing the number of poller processes. Only around 1000 ports on this install, 8 GB of RAM and 4 vCPU.
Thanks, Billy
[cid:image001.png@01D1EEFD.85030800]
[cid:DeanDortonbox.png]
Billy Zoellers Infrastructure Engineer
106 W. Vine St., Ste 600 Lexington, KY 40507 ph (859) 425-7640 fax (859) 425-3640
CONFIDENTIALITY NOTICE: The preceding email message, and any files or attachments transmitted with it, may contain confidential and privileged information, and may constitute protected work product, confidential client information, trade secrets, and/or other non-public information. Furthermore, this message and any attachments may be protected by accountant/client privilege, attorney/client privilege, and/or other applicable privileges. Any unauthorized review, use, dissemination, distribution, disclosure, or reproduction of this message is strictly prohibited and may be unlawful. This message is intended to be conveyed and transmitted only to the designated recipient(s). If you are not an intended recipient of this message, please notify the sender immediately by email or by telephone at 859-255-2341. Also, please delete this email message including all attachments and destroy all copies.
Hi,
Is there a firewall directly in front of your Observium server?
adam. On 05/08/2016 08:41:35, Zoellers, Billy bzoellers@ddaftech.com wrote: Hello all, We’re seeing holes on our graphs (example attached). This appears to be affecting all graphs in our installation on all devices. I know our connection is good (affecting LAN and WAN devices), but looking for any other ideas for correcting this? We have also tried increasing the number of poller processes. Only around 1000 ports on this install, 8 GB of RAM and 4 vCPU. Thanks, Billy Billy Zoellers Infrastructure Engineer
106 W. Vine St., Ste 600 Lexington, KY 40507 ph (859) 425-7640 fax (859) 425-3640
CONFIDENTIALITY NOTICE: The preceding email message, and any files or attachments transmitted with it, may contain confidential and privileged information, and may constitute protected work product, confidential client information, trade secrets, and/or other non-public information. Furthermore, this message and any attachments may be protected by accountant/client privilege, attorney/client privilege, and/or other applicable privileges. Any unauthorized review, use, dissemination, distribution, disclosure, or reproduction of this message is strictly prohibited and may be unlawful. This message is intended to be conveyed and transmitted only to the designated recipient(s). If you are not an intended recipient of this message, please notify the sender immediately by email or by telephone at 859-255-2341. Also, please delete this email message including all attachments and destroy all copies.
There is, but I’m seeing similar holes in my graphs of the host (localhost) that observium is sitting on.
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: Friday, August 5, 2016 1:22 PM To: observium@observium.org Subject: Re: [Observium] Holes in graphs
Hi,
Is there a firewall directly in front of your Observium server?
adam.
On 05/08/2016 08:41:35, Zoellers, Billy <bzoellers@ddaftech.commailto:bzoellers@ddaftech.com> wrote: Hello all,
We’re seeing holes on our graphs (example attached). This appears to be affecting all graphs in our installation on all devices. I know our connection is good (affecting LAN and WAN devices), but looking for any other ideas for correcting this? We have also tried increasing the number of poller processes. Only around 1000 ports on this install, 8 GB of RAM and 4 vCPU.
Thanks, Billy
[cid:image001.png@01D1EEFD.85030800]
[cid:image001.png@01D1EF22.CCE26290]
Billy Zoellers Infrastructure Engineer
106 W. Vine St., Ste 600 Lexington, KY 40507 ph (859) 425-7640 fax (859) 425-3640
CONFIDENTIALITY NOTICE: The preceding email message, and any files or attachments transmitted with it, may contain confidential and privileged information, and may constitute protected work product, confidential client information, trade secrets, and/or other non-public information. Furthermore, this message and any attachments may be protected by accountant/client privilege, attorney/client privilege, and/or other applicable privileges. Any unauthorized review, use, dissemination, distribution, disclosure, or reproduction of this message is strictly prohibited and may be unlawful. This message is intended to be conveyed and transmitted only to the designated recipient(s). If you are not an intended recipient of this message, please notify the sender immediately by email or by telephone at 859-255-2341. Also, please delete this email message including all attachments and destroy all copies.
Hi
Sounds like the problem I had in that the poller wasn't finishing in time so you need more poller pollers running
Rich
_____________________________ From: Zoellers, Billy <bzoellers@ddaftech.commailto:bzoellers@ddaftech.com> Sent: Friday, August 5, 2016 7:08 pm Subject: Re: [Observium] Holes in graphs To: Observium Network Observation System <observium@observium.orgmailto:observium@observium.org>
There is, but I’m seeing similar holes in my graphs of the host (localhost) that observium is sitting on.
From: observium [mailto:observium-bounces@observium.org]On Behalf Of Adam Armstrong Sent: Friday, August 5, 2016 1:22 PM To: observium@observium.orgmailto:observium@observium.org Subject: Re: [Observium] Holes in graphs
Hi,
Is there a firewall directly in front of your Observium server?
adam.
On 05/08/2016 08:41:35, Zoellers, Billy <bzoellers@ddaftech.commailto:bzoellers@ddaftech.com> wrote: Hello all,
We’re seeing holes on our graphs (example attached). This appears to be affecting all graphs in our installation on all devices. I know our connection is good (affecting LAN and WAN devices), but looking for any other ideas for correcting this? We have also tried increasing the number of poller processes. Only around 1000 ports on this install, 8 GB of RAM and 4 vCPU.
Thanks, Billy
[cid:image001.png@01D1EEFD.85030800]
[cid:image001.png@01D1EF22.CCE26290]
BillyZoellers Infrastructure Engineer
106 W. Vine St., Ste 600 Lexington, KY 40507 ph (859) 425-7640 fax (859) 425-3640
CONFIDENTIALITY NOTICE: The precedingemail message, and any files or attachments transmitted with it, may contain confidential and privileged information, and may constitute protected work product, confidential client information, trade secrets, and/or other non-public information. Furthermore, this message and any attachments may be protected by accountant/client privilege, attorney/client privilege, and/or other applicable privileges. Any unauthorized review, use, dissemination, distribution, disclosure, or reproduction of this message is strictly prohibited and may be unlawful. This message is intended to be conveyed andtransmitted only to the designated recipient(s). If you are not an intended recipient of this message, please notify the sender immediately by email or by telephone at 859-255-2341. Also, please delete this email message including all attachments and destroy all copies.
This e-mail is sent on behalf of Timico Partner Services Limited, a company registered in England and Wales, registered number 03128506, registered office Beacon Hill Park, Newark, Nottinghamshire, NG24 2TN and regulated by Ofcom. The information in this e-mail is confidential and is intended solely for the use of that individual or entity to which it is addressed. Unauthorised use, dissemination, distribution, publication or copying of this communication is strictly prohibited. If you receive this in error, please notify us by email to privacy@timico.co.ukmailto:privacy@timico.co.uk and delete any copies. For information about how we process data and monitor communications please see our privacy statementhttp://www.timico.co.uk/downloads/terms/Privacy_Statement.pdf.
HI,
Does not the rrd file step every 5 mins and if that is correct then, if the polling does not finish within that 5 min step then you will get a blank space. Solution is to get your polls under 5mins by doing some performance tuning.
Sam
On Fri, Aug 5, 2016 at 2:06 PM, Richard Savage richard.savage@timico.co.uk wrote:
Hi
Sounds like the problem I had in that the poller wasn't finishing in time so you need more poller pollers running
Rich
From: Zoellers, Billy bzoellers@ddaftech.com Sent: Friday, August 5, 2016 7:08 pm Subject: Re: [Observium] Holes in graphs To: Observium Network Observation System observium@observium.org
There is, but I’m seeing similar holes in my graphs of the host (localhost) that observium is sitting on.
*From:* observium [mailto:observium-bounces@observium.org observium-bounces@observium.org]*On Behalf Of *Adam Armstrong *Sent:* Friday, August 5, 2016 1:22 PM *To:* observium@observium.org *Subject:* Re: [Observium] Holes in graphs
Hi,
Is there a firewall directly in front of your Observium server?
adam.
On 05/08/2016 08:41:35, Zoellers, Billy bzoellers@ddaftech.com wrote:
Hello all,
We’re seeing holes on our graphs (example attached). This appears to be affecting all graphs in our installation on all devices. I know our connection is good (affecting LAN and WAN devices), but looking for any other ideas for correcting this? We have also tried increasing the number of poller processes. Only around 1000 ports on this install, 8 GB of RAM and 4 vCPU.
Thanks,
Billy
*Billy**Zoellers* *Infrastructure Engineer* 106 W. Vine St., Ste 600 Lexington, KY 40507 ph (859) 425-7640 fax (859) 425-3640CONFIDENTIALITY NOTICE: The precedingemail message, and any files or attachments transmitted with it, may contain confidential and privileged information, and may constitute protected work product, confidential client information, trade secrets, and/or other non-public information. Furthermore, this message and any attachments may be protected by accountant/client privilege, attorney/client privilege, and/or other applicable privileges. Any unauthorized review, use, dissemination, distribution, disclosure, or reproduction of this message is strictly prohibited and may be unlawful. This message is intended to be conveyed and transmitted only to the designated recipient(s). If you are not an intended recipient of this message, please notify the sender immediately by email or by telephone at 859-255-2341. Also, please delete this email message including all attachments and destroy all copies.
*This e-mail is sent on behalf of Timico Partner Services Limited, a company registered in England and Wales, registered number 03128506, registered office Beacon Hill Park, Newark, Nottinghamshire, NG24 2TN and regulated by Ofcom. The information in this e-mail is confidential and is intended solely for the use of that individual or entity to which it is addressed. Unauthorised use, dissemination, distribution, publication or copying of this communication is strictly prohibited. If you receive this in error, please notify us by email to privacy@timico.co.uk privacy@timico.co.uk and delete any copies. For information about how we process data and monitor communications please see our privacy statement http://www.timico.co.uk/downloads/terms/Privacy_Statement.pdf.*
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
If you're seeing holes in local host, it's almost certainly a problem with poller duration. You can see how long this takes in the poller information page (in the 'globe' menu, or /pollerlog/). If your poller wrapper is taking too long, that might cause the issue. You can see how long each device takes to poll on that page too. It's not that easy to cause the problem you're seeing without having a system which is too slow (you'd be able to tell pretty easily, i think), or running too few poller wrapper threads. For your setup, run maybe 8 threads. What is the content of /etc/cron.d/observium Thanks, adam. Sent from BlueMail [http://www.bluemail.me/r]
On 5 Aug 2016, at 13:08, "Zoellers, Billy" <bzoellers@ddaftech.com [mailto:bzoellers@ddaftech.com]> wrote: There is, but I’m seeing similar holes in my graphs of the host (localhost) that observium is sitting on. From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: Friday, August 5, 2016 1:22 PM To: observium@observium.org Subject: Re: [Observium] Holes in graphs Hi, Is there a firewall directly in front of your Observium server? adam. On 05/08/2016 08:41:35, Zoellers, Billy <bzoellers@ddaftech.com [mailto:bzoellers@ddaftech.com]> wrote: Hello all, We’re seeing holes on our graphs (example attached). This appears to be affecting all graphs in our installation on all devices. I know our connection is good (affecting LAN and WAN devices), but looking for any other ideas for correcting this? We have also tried increasing the number of poller processes. Only around 1000 ports on this install, 8 GB of RAM and 4 vCPU. Thanks, Billy Billy Zoellers Infrastructure Engineer
106 W. Vine St., Ste 600 Lexington, KY 40507 ph (859) 425-7640 fax (859) 425-3640
CONFIDENTIALITY NOTICE: The preceding email message, and any files or attachments transmitted with it, may contain confidential and privileged information, and may constitute protected work product, confidential client information, trade secrets, and/or other non-public information. Furthermore, this message and any attachments may be protected by accountant/client privilege, attorney/client privilege, and/or other applicable privileges. Any unauthorized review, use, dissemination, distribution, disclosure, or reproduction of this message is strictly prohibited and may be unlawful. This message is intended to be conveyed and transmitted only to the designated recipient(s). If you are not an intended recipient of this message, please notify the sender immediately by email or by telephone at 859-255-2341. Also, please delete this email message including all attachments and destroy all copies.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
Turned out to be a router that wanted to take 600s to poll ☺. Disabled and we’re graphing again. Is there any way to set a polling timeout to ensure that one device doesn’t cause a problem like this?
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: Friday, August 5, 2016 8:28 PM To: observium@observium.org Subject: Re: [Observium] Holes in graphs
If you're seeing holes in local host, it's almost certainly a problem with poller duration. You can see how long this takes in the poller information page (in the 'globe' menu, or /pollerlog/).
If your poller wrapper is taking too long, that might cause the issue. You can see how long each device takes to poll on that page too.
It's not that easy to cause the problem you're seeing without having a system which is too slow (you'd be able to tell pretty easily, i think), or running too few poller wrapper threads. For your setup, run maybe 8 threads.
What is the content of /etc/cron.d/observium
Thanks,
adam. Sent from BlueMailhttp://www.bluemail.me/r On 5 Aug 2016, at 13:08, "Zoellers, Billy" <bzoellers@ddaftech.commailto:bzoellers@ddaftech.com> wrote: There is, but I’m seeing similar holes in my graphs of the host (localhost) that observium is sitting on.
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: Friday, August 5, 2016 1:22 PM To: observium@observium.orgmailto:observium@observium.org Subject: Re: [Observium] Holes in graphs
Hi,
Is there a firewall directly in front of your Observium server?
adam.
On 05/08/2016 08:41:35, Zoellers, Billy <bzoellers@ddaftech.commailto:bzoellers@ddaftech.com> wrote: Hello all,
We’re seeing holes on our graphs (example attached). This appears to be affecting all graphs in our installation on all devices. I know our connection is good (affecting LAN and WAN devices), but looking for any other ideas for correcting this? We have also tried increasing the number of poller processes. Only around 1000 ports on this install, 8 GB of RAM and 4 vCPU.
Thanks, Billy
[cid:image001.png@01D1EEFD.85030800]
[cid:image001.png@01D1EF22.CCE26290]
Billy Zoellers Infrastructure Engineer
106 W. Vine St., Ste 600 Lexington, KY 40507 ph (859) 425-7640 fax (859) 425-3640
CONFIDENTIALITY NOTICE: The preceding email message, and any files or attachments transmitted with it, may contain confidential and privileged information, and may constitute protected work product, confidential client information, trade secrets, and/or other non-public information. Furthermore, this message and any attachments may be protected by accountant/client privilege, attorney/client privilege, and/or other applicable privileges. Any unauthorized review, use, dissemination, distribution, disclosure, or reproduction of this message is strictly prohibited and may be unlawful. This message is intended to be conveyed and transmitted only to the designated recipient(s). If you are not an intended recipient of this message, please notify the sender immediately by email or by telephone at 859-255-2341. Also, please delete this email message including all attachments and destroy all copies.
________________________________
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Add more poller_wrapper threads and a single device won't hold things up.
adam. On 08/08/2016 09:30:37, Zoellers, Billy bzoellers@ddaftech.com wrote: Turned out to be a router that wanted to take 600s to poll J. Disabled and we’re graphing again. Is there any way to set a polling timeout to ensure that one device doesn’t cause a problem like this? From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: Friday, August 5, 2016 8:28 PM To: observium@observium.org Subject: Re: [Observium] Holes in graphs If you're seeing holes in local host, it's almost certainly a problem with poller duration. You can see how long this takes in the poller information page (in the 'globe' menu, or /pollerlog/). If your poller wrapper is taking too long, that might cause the issue. You can see how long each device takes to poll on that page too. It's not that easy to cause the problem you're seeing without having a system which is too slow (you'd be able to tell pretty easily, i think), or running too few poller wrapper threads. For your setup, run maybe 8 threads. What is the content of /etc/cron.d/observium Thanks, adam. Sent from BlueMail [http://www.bluemail.me/r] On 5 Aug 2016, at 13:08, "Zoellers, Billy" <bzoellers@ddaftech.com [mailto:bzoellers@ddaftech.com]> wrote: There is, but I’m seeing similar holes in my graphs of the host (localhost) that observium is sitting on. From: observium [mailto:observium-bounces@observium.org [mailto:observium-bounces@observium.org]] On Behalf Of Adam Armstrong Sent: Friday, August 5, 2016 1:22 PM To: observium@observium.org [mailto:observium@observium.org] Subject: Re: [Observium] Holes in graphs Hi, Is there a firewall directly in front of your Observium server? adam. On 05/08/2016 08:41:35, Zoellers, Billy <bzoellers@ddaftech.com [mailto:bzoellers@ddaftech.com]> wrote: Hello all, We’re seeing holes on our graphs (example attached). This appears to be affecting all graphs in our installation on all devices. I know our connection is good (affecting LAN and WAN devices), but looking for any other ideas for correcting this? We have also tried increasing the number of poller processes. Only around 1000 ports on this install, 8 GB of RAM and 4 vCPU. Thanks, Billy Billy Zoellers Infrastructure Engineer
106 W. Vine St., Ste 600 Lexington, KY 40507 ph (859) 425-7640 fax (859) 425-3640 CONFIDENTIALITY NOTICE: The preceding email message, and any files or attachments transmitted with it, may contain confidential and privileged information, and may constitute protected work product, confidential client information, trade secrets, and/or other non-public information. Furthermore, this message and any attachments may be protected by accountant/client privilege, attorney/client privilege, and/or other applicable privileges. Any unauthorized review, use, dissemination, distribution, disclosure, or reproduction of this message is strictly prohibited and may be unlawful. This message is intended to be conveyed and transmitted only to the designated recipient(s). If you are not an intended recipient of this message, please notify the sender immediately by email or by telephone at 859-255-2341. Also, please delete this email message including all attachments and destroy all copies.
observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
participants (4)
-
Adam Armstrong -
Richard Savage -
Sam Jones -
Zoellers, Billy