Device taking > 5 minutes to poll
Hey all,
I’ve got a device (Zhone MXK-F) with over 10,000 ports on it, and Observium is taking well over 5 minutes to poll it (upwards of 15 minutes, actually). While there isn’t a definition specifically for MXK-F, it’s getting detected as MXK. os.inc.php is setting max-rep 50, so it’s at least trying to bulk walk, but I don’t know if it’s actually working. Overriding to max-rep 100 doesn’t have an affect on the polling time.
So, I guess my question is whether or not I’m on the right track in assuming that this issue is likely that of bulk walk on this device, or possibly some other issue, and how would I determine one way or the other? If it’s something that I can pinpoint, I can throw it back to the vendor to fix.
From the digging I’ve done so far, all other poller processes called from poller-wrapper complete in under 5 minutes, so I’m fairly certain that this issue is one of the device, not Observium. But, to be thorough, I moved my database and RRD storage over to local SSDs, and also enabled rrdcached - all that to ensure I didn’t have any I/O issues.
/pollerlog/view=devices indicates that it’s just the one device that has high timing (although other devices of the same type are spots 2-7 on the top 10 list high timers, but none is over 300 seconds except for this guy).
root@monitor:/home/jlixfeld# /opt/observium/poller.php -VV Observium 17.9.8837
##### Software versions #####
o OS Linux 3.16.0-4-amd64 [amd64] (Debian 8 (jessie)) o Apache 2.4.10 o PHP 5.6.30-0+deb8u1 (OPcache: DISABLED) o Python 2.7.9 o MySQL 5.5.55-0+deb8u1 (extension: mysqli 5.5.55) o SNMP NET-SNMP 5.7.2.1 o RRDtool 1.4.8 (rrdcached 1.4.8: unix:/opt/observium/rrdcached.sock)
##### Memory Limit #####
o PHP Unlimited
##### MySQL mode #####
o MySQL
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Monday, 02-Oct-17 12:55:15 EDT o PHP -04:00 o MySQL -04:00
root@monitor:/home/jlixfeld#
Thanks all!
10k ports is going to take a long time on a device without a blistering fast control plane.
What are the ports, mostly?
Adam.
Sent from BlueMail
On 2 Oct 2017, 19:05, at 19:05, Jason Lixfeld jason@lixfeld.ca wrote:
Hey all,
I’ve got a device (Zhone MXK-F) with over 10,000 ports on it, and Observium is taking well over 5 minutes to poll it (upwards of 15 minutes, actually). While there isn’t a definition specifically for MXK-F, it’s getting detected as MXK. os.inc.php is setting max-rep 50, so it’s at least trying to bulk walk, but I don’t know if it’s actually working. Overriding to max-rep 100 doesn’t have an affect on the polling time.
So, I guess my question is whether or not I’m on the right track in assuming that this issue is likely that of bulk walk on this device, or possibly some other issue, and how would I determine one way or the other? If it’s something that I can pinpoint, I can throw it back to the vendor to fix.
From the digging I’ve done so far, all other poller processes called from poller-wrapper complete in under 5 minutes, so I’m fairly certain that this issue is one of the device, not Observium. But, to be thorough, I moved my database and RRD storage over to local SSDs, and also enabled rrdcached - all that to ensure I didn’t have any I/O issues.
/pollerlog/view=devices indicates that it’s just the one device that has high timing (although other devices of the same type are spots 2-7 on the top 10 list high timers, but none is over 300 seconds except for this guy).
root@monitor:/home/jlixfeld# /opt/observium/poller.php -VV Observium 17.9.8837
##### Software versions #####
o OS Linux 3.16.0-4-amd64 [amd64] (Debian 8 (jessie)) o Apache 2.4.10 o PHP 5.6.30-0+deb8u1 (OPcache: DISABLED) o Python 2.7.9 o MySQL 5.5.55-0+deb8u1 (extension: mysqli 5.5.55) o SNMP NET-SNMP 5.7.2.1 o RRDtool 1.4.8 (rrdcached 1.4.8: unix:/opt/observium/rrdcached.sock)
##### Memory Limit #####
o PHP Unlimited
##### MySQL mode #####
o MySQL
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Monday, 02-Oct-17 12:55:15 EDT o PHP -04:00 o MySQL -04:00
root@monitor:/home/jlixfeld#
Thanks all! _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Mostly GPON ports (basically Ethernet, but I don’t know the ifType off the top of my head). Some transparent bridges too. Now to be fair, not all 10,000 ports are up, but I don’t recall if there’s a way to only poll ports that are up. I suppose though that you need to first poll all ports to determine status, then poll again only the ones you want to really poll. Cart, Horse. :)
On Oct 2, 2017, at 2:09 PM, Adam Armstrong adama@memetic.org wrote:
10k ports is going to take a long time on a device without a blistering fast control plane.
What are the ports, mostly?
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=10777 On 2 Oct 2017, at 19:05, Jason Lixfeld <jason@lixfeld.ca mailto:jason@lixfeld.ca> wrote: Hey all,
I’ve got a device (Zhone MXK-F) with over 10,000 ports on it, and Observium is taking well over 5 minutes to poll it (upwards of 15 minutes, actually). While there isn’t a definition specifically for MXK-F, it’s getting detected as MXK. os.inc.php is setting max-rep 50, so it’s at least trying to bulk walk, but I don’t know if it’s actually working. Overriding to max-rep 100 doesn’t have an affect on the polling time.
So, I guess my question is whether or not I’m on the right track in assuming that this issue is likely that of bulk walk on this device, or possibly some other issue, and how would I determine one way or the other? If it’s something that I can pinpoint, I can throw it back to the vendor to fix.
From the digging I’ve done so far, all other poller processes called from poller-wrapper complete in under 5 minutes, so I’m fairly certain that this issue is one of the device, not Observium. But, to be thorough, I moved my database and RRD storage over to local SSDs, and also enabled rrdcached - all that to ensure I didn’t have any I/O issues.
/pollerlog/view=devices indicates that it’s just the one device that has high timing (although other devices of the same type are spots 2-7 on the top 10 list high timers, but none is over 300 seconds except for this guy).
root@monitor:/home/jlixfeld# /opt/observium/poller.php -VV Observium 17.9.8837
##### Software versions #####
o OS Linux 3.16.0-4-amd64 [amd64] (Debian 8 (jessie)) o Apache 2.4.10 o PHP 5.6.30-0+deb8u1 (OPcache: DISABLED) o Python 2.7.9 o MySQL 5.5.55-0+deb8u1 (extension: mysqli 5.5.55) o SNMP NET-SNMP 5.7.2.1 http://5.7.2.1/ o RRDtool 1.4.8 (rrdcached 1.4.8: unix:/opt/observium/rrdcached.sock)
##### Memory Limit #####
o PHP Unlimited
##### MySQL mode #####
o MySQL
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Monday, 02-Oct-17 12:55:15 EDT o PHP -04:00 o MySQL -04:00
root@monitor:/home/jlixfeld#
Thanks all!
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Do you want to graph the traffic of all the 10 000 ports? You could filter out only the ports you are interested in and have Observium only poll those. Assuming the ports you are interested in are maybe uplinks and stuff, like <50 ports.
Den 2 okt. 2017 20:51 skrev "Jason Lixfeld" jason@lixfeld.ca:
Mostly GPON ports (basically Ethernet, but I don’t know the ifType off the top of my head). Some transparent bridges too. Now to be fair, not all 10,000 ports are up, but I don’t recall if there’s a way to only poll ports that are up. I suppose though that you need to first poll all ports to determine status, then poll again only the ones you want to really poll. Cart, Horse. :)
On Oct 2, 2017, at 2:09 PM, Adam Armstrong adama@memetic.org wrote:
10k ports is going to take a long time on a device without a blistering fast control plane.
What are the ports, mostly?
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=10777 On 2 Oct 2017, at 19:05, Jason Lixfeld jason@lixfeld.ca wrote:
Hey all,
I’ve got a device (Zhone MXK-F) with over 10,000 ports on it, and Observium is taking well over 5 minutes to poll it (upwards of 15 minutes, actually). While there isn’t a definition specifically for MXK-F, it’s getting detected as MXK. os.inc.php is setting max-rep 50, so it’s at least trying to bulk walk, but I don’t know if it’s actually working. Overriding to max-rep 100 doesn’t have an affect on the polling time.
So, I guess my question is whether or not I’m on the right track in assuming that this issue is likely that of bulk walk on this device, or possibly some other issue, and how would I determine one way or the other? If it’s something that I can pinpoint, I can throw it back to the vendor to fix.
From the digging I’ve done so far, all other poller processes called from poller-wrapper complete in under 5 minutes, so I’m fairly certain that this issue is one of the device, not Observium. But, to be thorough, I moved my database and RRD storage over to local SSDs, and also enabled rrdcached - all that to ensure I didn’t have any I/O issues.
/pollerlog/view=devices indicates that it’s just the one device that has high timing (although other devices of the same type are spots 2-7 on the top 10 list high timers, but none is over 300 seconds except for this guy).
root@monitor:/home/jlixfeld# /opt/observium/poller.php -VV Observium 17.9.8837
##### Software versions #####
o OS Linux 3.16.0-4-amd64 [amd64] (Debian 8 (jessie)) o Apache 2.4.10 o PHP 5.6.30-0+deb8u1 (OPcache: DISABLED) o Python 2.7.9 o MySQL 5.5.55-0+deb8u1 (extension: mysqli 5.5.55) o SNMP NET-SNMP 5.7.2.1 o RRDtool 1.4.8 (rrdcached 1.4.8: unix:/opt/observium/rrdcached.sock)
##### Memory Limit #####
o PHP Unlimited
##### MySQL mode #####
o MySQL
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Monday, 02-Oct-17 12:55:15 EDT o PHP -04:00 o MySQL -04:00
root@monitor:/home/jlixfeld#
Thanks all!
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
It would be each active customer’s port, and the device uplink(s). So pretty much a constantly moving target :/
On Oct 2, 2017, at 2:54 PM, Markus Klock markus@best-practice.se wrote:
Do you want to graph the traffic of all the 10 000 ports? You could filter out only the ports you are interested in and have Observium only poll those. Assuming the ports you are interested in are maybe uplinks and stuff, like <50 ports.
Den 2 okt. 2017 20:51 skrev "Jason Lixfeld" <jason@lixfeld.ca mailto:jason@lixfeld.ca>: Mostly GPON ports (basically Ethernet, but I don’t know the ifType off the top of my head). Some transparent bridges too. Now to be fair, not all 10,000 ports are up, but I don’t recall if there’s a way to only poll ports that are up. I suppose though that you need to first poll all ports to determine status, then poll again only the ones you want to really poll. Cart, Horse. :)
On Oct 2, 2017, at 2:09 PM, Adam Armstrong <adama@memetic.org mailto:adama@memetic.org> wrote:
10k ports is going to take a long time on a device without a blistering fast control plane.
What are the ports, mostly?
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=10777 On 2 Oct 2017, at 19:05, Jason Lixfeld <jason@lixfeld.ca mailto:jason@lixfeld.ca> wrote: Hey all,
I’ve got a device (Zhone MXK-F) with over 10,000 ports on it, and Observium is taking well over 5 minutes to poll it (upwards of 15 minutes, actually). While there isn’t a definition specifically for MXK-F, it’s getting detected as MXK. os.inc.php is setting max-rep 50, so it’s at least trying to bulk walk, but I don’t know if it’s actually working. Overriding to max-rep 100 doesn’t have an affect on the polling time.
So, I guess my question is whether or not I’m on the right track in assuming that this issue is likely that of bulk walk on this device, or possibly some other issue, and how would I determine one way or the other? If it’s something that I can pinpoint, I can throw it back to the vendor to fix.
From the digging I’ve done so far, all other poller processes called from poller-wrapper complete in under 5 minutes, so I’m fairly certain that this issue is one of the device, not Observium. But, to be thorough, I moved my database and RRD storage over to local SSDs, and also enabled rrdcached - all that to ensure I didn’t have any I/O issues.
/pollerlog/view=devices indicates that it’s just the one device that has high timing (although other devices of the same type are spots 2-7 on the top 10 list high timers, but none is over 300 seconds except for this guy).
root@monitor:/home/jlixfeld# /opt/observium/poller.php -VV Observium 17.9.8837
##### Software versions #####
o OS Linux 3.16.0-4-amd64 [amd64] (Debian 8 (jessie)) o Apache 2.4.10 o PHP 5.6.30-0+deb8u1 (OPcache: DISABLED) o Python 2.7.9 o MySQL 5.5.55-0+deb8u1 (extension: mysqli 5.5.55) o SNMP NET-SNMP 5.7.2.1 http://5.7.2.1/ o RRDtool 1.4.8 (rrdcached 1.4.8: unix:/opt/observium/rrdcached.sock)
##### Memory Limit #####
o PHP Unlimited
##### MySQL mode #####
o MySQL
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Monday, 02-Oct-17 12:55:15 EDT o PHP -04:00 o MySQL -04:00
root@monitor:/home/jlixfeld#
Thanks all!
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
If you can deal with not having the pon interfaces, I'd filter them out somehow and switch to the separate walk ports poller method.
Adam.
Sent from BlueMail
On 2 Oct 2017, 20:05, at 20:05, Jason Lixfeld jason@lixfeld.ca wrote:
It would be each active customer’s port, and the device uplink(s). So pretty much a constantly moving target :/
On Oct 2, 2017, at 2:54 PM, Markus Klock markus@best-practice.se
wrote:
Do you want to graph the traffic of all the 10 000 ports? You could
filter out only the ports you are interested in and have Observium only poll those. Assuming the ports you are interested in are maybe uplinks and stuff, like <50 ports.
Den 2 okt. 2017 20:51 skrev "Jason Lixfeld" <jason@lixfeld.ca
Mostly GPON ports (basically Ethernet, but I don’t know the ifType
off the top of my head). Some transparent bridges too. Now to be fair, not all 10,000 ports are up, but I don’t recall if there’s a way to only poll ports that are up. I suppose though that you need to first poll all ports to determine status, then poll again only the ones you want to really poll. Cart, Horse. :)
On Oct 2, 2017, at 2:09 PM, Adam Armstrong <adama@memetic.org
mailto:adama@memetic.org> wrote:
10k ports is going to take a long time on a device without a
blistering fast control plane.
What are the ports, mostly?
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=10777 On 2 Oct 2017, at 19:05, Jason Lixfeld <jason@lixfeld.ca
mailto:jason@lixfeld.ca> wrote:
Hey all,
I’ve got a device (Zhone MXK-F) with over 10,000 ports on it, and
Observium is taking well over 5 minutes to poll it (upwards of 15 minutes, actually). While there isn’t a definition specifically for MXK-F, it’s getting detected as MXK. os.inc.php is setting max-rep 50, so it’s at least trying to bulk walk, but I don’t know if it’s actually working. Overriding to max-rep 100 doesn’t have an affect on the polling time.
So, I guess my question is whether or not I’m on the right track in
assuming that this issue is likely that of bulk walk on this device, or possibly some other issue, and how would I determine one way or the other? If it’s something that I can pinpoint, I can throw it back to the vendor to fix.
From the digging I’ve done so far, all other poller processes called
from poller-wrapper complete in under 5 minutes, so I’m fairly certain that this issue is one of the device, not Observium. But, to be thorough, I moved my database and RRD storage over to local SSDs, and also enabled rrdcached - all that to ensure I didn’t have any I/O issues.
/pollerlog/view=devices indicates that it’s just the one device that
has high timing (although other devices of the same type are spots 2-7 on the top 10 list high timers, but none is over 300 seconds except for this guy).
root@monitor:/home/jlixfeld# /opt/observium/poller.php -VV Observium 17.9.8837
##### Software versions #####
o OS Linux 3.16.0-4-amd64 [amd64] (Debian 8
(jessie))
o Apache 2.4.10 o PHP 5.6.30-0+deb8u1 (OPcache: DISABLED) o Python 2.7.9 o MySQL 5.5.55-0+deb8u1 (extension: mysqli 5.5.55) o SNMP NET-SNMP 5.7.2.1 http://5.7.2.1/ o RRDtool 1.4.8 (rrdcached 1.4.8:
unix:/opt/observium/rrdcached.sock)
##### Memory Limit #####
o PHP Unlimited
##### MySQL mode #####
o MySQL
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Monday, 02-Oct-17 12:55:15 EDT o PHP -04:00 o MySQL -04:00
root@monitor:/home/jlixfeld#
Thanks all!
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
We’d be relying on the transparent bridge interfaces at that point. While that might work today, each customer occupies a > 1 number of transparent bridge interfaces, depending on the number of services they have subscribed to. At a certain point, the number of transparent bridge ports would eclipse the number of GPON ports.
On Oct 2, 2017, at 3:06 PM, Adam Armstrong adama@memetic.org wrote:
If you can deal with not having the pon interfaces, I'd filter them out somehow and switch to the separate walk ports poller method.
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=10777 On 2 Oct 2017, at 20:05, Jason Lixfeld <jason@lixfeld.ca mailto:jason@lixfeld.ca> wrote: It would be each active customer’s port, and the device uplink(s). So pretty much a constantly moving target :/
On Oct 2, 2017, at 2:54 PM, Markus Klock <markus@best-practice.se mailto:markus@best-practice.se> wrote:
Do you want to graph the traffic of all the 10 000 ports? You could filter out only the ports you are interested in and have Observium only poll those. Assuming the ports you are interested in are maybe uplinks and stuff, like <50 ports.
Den 2 okt. 2017 20:51 skrev "Jason Lixfeld" <jason@lixfeld.ca mailto:jason@lixfeld.ca>: Mostly GPON ports (basically Ethernet, but I don’t know the ifType off the top of my head). Some transparent bridges too. Now to be fair, not all 10,000 ports are up, but I don’t recall if there’s a way to only poll ports that are up. I suppose though that you need to first poll all ports to determine status, then poll again only the ones you want to really poll. Cart, Horse. :)
On Oct 2, 2017, at 2:09 PM, Adam Armstrong <adama@memetic.org mailto:adama@memetic.org> wrote:
10k ports is going to take a long time on a device without a blistering fast control plane.
What are the ports, mostly?
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=10777 On 2 Oct 2017, at 19:05, Jason Lixfeld <jason@lixfeld.ca mailto:jason@lixfeld.ca> wrote: Hey all,
I’ve got a device (Zhone MXK-F) with over 10,000 ports on it, and Observium is taking well over 5 minutes to poll it (upwards of 15 minutes, actually). While there isn’t a definition specifically for MXK-F, it’s getting detected as MXK. os.inc.php is setting max-rep 50, so it’s at least trying to bulk walk, but I don’t know if it’s actually working. Overriding to max-rep 100 doesn’t have an affect on the polling time.
So, I guess my question is whether or not I’m on the right track in assuming that this issue is likely that of bulk walk on this device, or possibly some other issue, and how would I determine one way or the other? If it’s something that I can pinpoint, I can throw it back to the vendor to fix.
From the digging I’ve done so far, all other poller processes called from poller-wrapper complete in under 5 minutes, so I’m fairly certain that this issue is one of the device, not Observium. But, to be thorough, I moved my database and RRD storage over to local SSDs, and also enabled rrdcached - all that to ensure I didn’t have any I/O issues.
/pollerlog/view=devices indicates that it’s just the one device that has high timing (although other devices of the same type are spots 2-7 on the top 10 list high timers, but none is over 300 seconds except for this guy).
root@monitor:/home/jlixfeld# /opt/observium/poller.php -VV Observium 17.9.8837
##### Software versions #####
o OS Linux 3.16.0-4-amd64 [amd64] (Debian 8 (jessie)) o Apache 2.4.10 o PHP 5.6.30-0+deb8u1 (OPcache: DISABLED) o Python 2.7.9 o MySQL 5.5.55-0+deb8u1 (extension: mysqli 5.5.55) o SNMP NET-SNMP 5.7.2.1 http://5.7.2.1/ o RRDtool 1.4.8 (rrdcached 1.4.8: unix:/opt/observium/rrdcached.sock)
##### Memory Limit #####
o PHP Unlimited
##### MySQL mode #####
o MySQL
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Monday, 02-Oct-17 12:55:15 EDT o PHP -04:00 o MySQL -04:00
root@monitor:/home/jlixfeld#
Thanks all!
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hey Adam,
To circle back around to this - I’m not sure what you mean by separate walk ports poller method. Is this concept on the site somewhere that I can read up on?
Thanks!
On Oct 2, 2017, at 3:12 PM, Jason Lixfeld jason@lixfeld.ca wrote:
We’d be relying on the transparent bridge interfaces at that point. While that might work today, each customer occupies a > 1 number of transparent bridge interfaces, depending on the number of services they have subscribed to. At a certain point, the number of transparent bridge ports would eclipse the number of GPON ports.
On Oct 2, 2017, at 3:06 PM, Adam Armstrong <adama@memetic.org mailto:adama@memetic.org> wrote:
If you can deal with not having the pon interfaces, I'd filter them out somehow and switch to the separate walk ports poller method.
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=10777 On 2 Oct 2017, at 20:05, Jason Lixfeld <jason@lixfeld.ca mailto:jason@lixfeld.ca> wrote: It would be each active customer’s port, and the device uplink(s). So pretty much a constantly moving target :/
On Oct 2, 2017, at 2:54 PM, Markus Klock <markus@best-practice.se mailto:markus@best-practice.se> wrote:
Do you want to graph the traffic of all the 10 000 ports? You could filter out only the ports you are interested in and have Observium only poll those. Assuming the ports you are interested in are maybe uplinks and stuff, like <50 ports.
Den 2 okt. 2017 20:51 skrev "Jason Lixfeld" <jason@lixfeld.ca mailto:jason@lixfeld.ca>: Mostly GPON ports (basically Ethernet, but I don’t know the ifType off the top of my head). Some transparent bridges too. Now to be fair, not all 10,000 ports are up, but I don’t recall if there’s a way to only poll ports that are up. I suppose though that you need to first poll all ports to determine status, then poll again only the ones you want to really poll. Cart, Horse. :)
On Oct 2, 2017, at 2:09 PM, Adam Armstrong <adama@memetic.org mailto:adama@memetic.org> wrote:
10k ports is going to take a long time on a device without a blistering fast control plane.
What are the ports, mostly?
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=10777 On 2 Oct 2017, at 19:05, Jason Lixfeld <jason@lixfeld.ca mailto:jason@lixfeld.ca> wrote: Hey all,
I’ve got a device (Zhone MXK-F) with over 10,000 ports on it, and Observium is taking well over 5 minutes to poll it (upwards of 15 minutes, actually). While there isn’t a definition specifically for MXK-F, it’s getting detected as MXK. os.inc.php is setting max-rep 50, so it’s at least trying to bulk walk, but I don’t know if it’s actually working. Overriding to max-rep 100 doesn’t have an affect on the polling time.
So, I guess my question is whether or not I’m on the right track in assuming that this issue is likely that of bulk walk on this device, or possibly some other issue, and how would I determine one way or the other? If it’s something that I can pinpoint, I can throw it back to the vendor to fix.
From the digging I’ve done so far, all other poller processes called from poller-wrapper complete in under 5 minutes, so I’m fairly certain that this issue is one of the device, not Observium. But, to be thorough, I moved my database and RRD storage over to local SSDs, and also enabled rrdcached - all that to ensure I didn’t have any I/O issues.
/pollerlog/view=devices indicates that it’s just the one device that has high timing (although other devices of the same type are spots 2-7 on the top 10 list high timers, but none is over 300 seconds except for this guy).
root@monitor:/home/jlixfeld# /opt/observium/poller.php -VV Observium 17.9.8837
##### Software versions #####
o OS Linux 3.16.0-4-amd64 [amd64] (Debian 8 (jessie)) o Apache 2.4.10 o PHP 5.6.30-0+deb8u1 (OPcache: DISABLED) o Python 2.7.9 o MySQL 5.5.55-0+deb8u1 (extension: mysqli 5.5.55) o SNMP NET-SNMP 5.7.2.1 http://5.7.2.1/ o RRDtool 1.4.8 (rrdcached 1.4.8: unix:/opt/observium/rrdcached.sock)
##### Memory Limit #####
o PHP Unlimited
##### MySQL mode #####
o MySQL
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Monday, 02-Oct-17 12:55:15 EDT o PHP -04:00 o MySQL -04:00
root@monitor:/home/jlixfeld#
Thanks all!
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hey, The default way of SNMP-polling ports is by doing a snmpbulkwalk of the whole table of interfaces on a device as this is usually faster. But in some cases the table of interfaces is huge, and you are really only interested in a smaller part of the interfaces, then the separate_walk-feature is usefull. Start by filtering out the interfaces you do not want by using interface filters: http://docs.observium.org/config_options/#filters_1 Then when the separate_walk-feature is enabled for the device, Observium will only do single snmpgets of the interfaces not filtered out instead of bulkwalking the whole interface table. You enable the feature by device->properties->modules->separate_walk=enabled
/Markus
2017-10-23 3:05 GMT+02:00 Jason Lixfeld jason@lixfeld.ca:
Hey Adam,
To circle back around to this - I’m not sure what you mean by separate walk ports poller method. Is this concept on the site somewhere that I can read up on?
Thanks!
On Oct 2, 2017, at 3:12 PM, Jason Lixfeld jason@lixfeld.ca wrote:
We’d be relying on the transparent bridge interfaces at that point. While that might work today, each customer occupies a > 1 number of transparent bridge interfaces, depending on the number of services they have subscribed to. At a certain point, the number of transparent bridge ports would eclipse the number of GPON ports.
On Oct 2, 2017, at 3:06 PM, Adam Armstrong adama@memetic.org wrote:
If you can deal with not having the pon interfaces, I'd filter them out somehow and switch to the separate walk ports poller method.
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=10777 On 2 Oct 2017, at 20:05, Jason Lixfeld jason@lixfeld.ca wrote:
It would be each active customer’s port, and the device uplink(s). So pretty much a constantly moving target :/
On Oct 2, 2017, at 2:54 PM, Markus Klock markus@best-practice.se wrote:
Do you want to graph the traffic of all the 10 000 ports? You could filter out only the ports you are interested in and have Observium only poll those. Assuming the ports you are interested in are maybe uplinks and stuff, like <50 ports.
Den 2 okt. 2017 20:51 skrev "Jason Lixfeld" jason@lixfeld.ca:
Mostly GPON ports (basically Ethernet, but I don’t know the ifType off the top of my head). Some transparent bridges too. Now to be fair, not all 10,000 ports are up, but I don’t recall if there’s a way to only poll ports that are up. I suppose though that you need to first poll all ports to determine status, then poll again only the ones you want to really poll. Cart, Horse. :)
On Oct 2, 2017, at 2:09 PM, Adam Armstrong adama@memetic.org wrote:
10k ports is going to take a long time on a device without a blistering fast control plane.
What are the ports, mostly?
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=10777 On 2 Oct 2017, at 19:05, Jason Lixfeld jason@lixfeld.ca wrote:
Hey all,
I’ve got a device (Zhone MXK-F) with over 10,000 ports on it, and Observium is taking well over 5 minutes to poll it (upwards of 15 minutes, actually). While there isn’t a definition specifically for MXK-F, it’s getting detected as MXK. os.inc.php is setting max-rep 50, so it’s at least trying to bulk walk, but I don’t know if it’s actually working. Overriding to max-rep 100 doesn’t have an affect on the polling time.
So, I guess my question is whether or not I’m on the right track in assuming that this issue is likely that of bulk walk on this device, or possibly some other issue, and how would I determine one way or the other? If it’s something that I can pinpoint, I can throw it back to the vendor to fix.
From the digging I’ve done so far, all other poller processes called from poller-wrapper complete in under 5 minutes, so I’m fairly certain that this issue is one of the device, not Observium. But, to be thorough, I moved my database and RRD storage over to local SSDs, and also enabled rrdcached - all that to ensure I didn’t have any I/O issues.
/pollerlog/view=devices indicates that it’s just the one device that has high timing (although other devices of the same type are spots 2-7 on the top 10 list high timers, but none is over 300 seconds except for this guy).
root@monitor:/home/jlixfeld# /opt/observium/poller.php -VV Observium 17.9.8837
##### Software versions #####
o OS Linux 3.16.0-4-amd64 [amd64] (Debian 8 (jessie)) o Apache 2.4.10 o PHP 5.6.30-0+deb8u1 (OPcache: DISABLED) o Python 2.7.9 o MySQL 5.5.55-0+deb8u1 (extension: mysqli 5.5.55) o SNMP NET-SNMP 5.7.2.1 o RRDtool 1.4.8 (rrdcached 1.4.8: unix:/opt/observium/rrdcached.sock)
##### Memory Limit #####
o PHP Unlimited
##### MySQL mode #####
o MySQL
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Monday, 02-Oct-17 12:55:15 EDT o PHP -04:00 o MySQL -04:00
root@monitor:/home/jlixfeld#
Thanks all!
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hey,
Got it. Thanks. I think this will work. This box uses ‘other’ ifType quite a bit, and it can be filtered out. Doing so dropped the port count on this device from over 10,000 to just over 1,500.
With regards to snmpbulkwalk vs snmpwalk - I’m a little unclear on where the filter is used and when it isn’t; trying to understand how this all works a bit better. So a couple questions, if you don’t mind -
- Are you saying that the filter is only used during walk and not during bulkwalk? - The contextual help in the GUI when populating these bad_if* fields suggests that these are considered during discovery, not polling. So 1. Every time discovery runs, it creates a list of ‘valid’ interfaces based on the results of snmp(bulk?)walk, then removing anything in bad_if* 2. Every polling interval it walks only that list for a device if the device has separate_walk enabled, and if the device does not have separate_walk enabled, the poller will just bulkwalk the entire table and stick everything into the database, create graphs for it all, etc?
Also, FWIW, manually doing a walk on IF only takes about 10s longer to complete than bulkwalk on just over 48,000 entries in the table for this particular device; this is the most dense device. That’s interesting to me in that I thought a bulkwalk would take much, much less time, or is the time dependant on a bunch of other stuff too?
Thanks in advance!
On Oct 23, 2017, at 2:23 AM, Markus Klock markus@best-practice.se wrote:
Hey, The default way of SNMP-polling ports is by doing a snmpbulkwalk of the whole table of interfaces on a device as this is usually faster. But in some cases the table of interfaces is huge, and you are really only interested in a smaller part of the interfaces, then the separate_walk-feature is usefull. Start by filtering out the interfaces you do not want by using interface filters: http://docs.observium.org/config_options/#filters_1 http://docs.observium.org/config_options/#filters_1 Then when the separate_walk-feature is enabled for the device, Observium will only do single snmpgets of the interfaces not filtered out instead of bulkwalking the whole interface table. You enable the feature by device->properties->modules->separate_walk=enabled
/Markus
2017-10-23 3:05 GMT+02:00 Jason Lixfeld <jason@lixfeld.ca mailto:jason@lixfeld.ca>: Hey Adam,
To circle back around to this - I’m not sure what you mean by separate walk ports poller method. Is this concept on the site somewhere that I can read up on?
Thanks!
On Oct 2, 2017, at 3:12 PM, Jason Lixfeld <jason@lixfeld.ca mailto:jason@lixfeld.ca> wrote:
We’d be relying on the transparent bridge interfaces at that point. While that might work today, each customer occupies a > 1 number of transparent bridge interfaces, depending on the number of services they have subscribed to. At a certain point, the number of transparent bridge ports would eclipse the number of GPON ports.
On Oct 2, 2017, at 3:06 PM, Adam Armstrong <adama@memetic.org mailto:adama@memetic.org> wrote:
If you can deal with not having the pon interfaces, I'd filter them out somehow and switch to the separate walk ports poller method.
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=10777 On 2 Oct 2017, at 20:05, Jason Lixfeld <jason@lixfeld.ca mailto:jason@lixfeld.ca> wrote: It would be each active customer’s port, and the device uplink(s). So pretty much a constantly moving target :/
On Oct 2, 2017, at 2:54 PM, Markus Klock <markus@best-practice.se mailto:markus@best-practice.se> wrote:
Do you want to graph the traffic of all the 10 000 ports? You could filter out only the ports you are interested in and have Observium only poll those. Assuming the ports you are interested in are maybe uplinks and stuff, like <50 ports.
Den 2 okt. 2017 20:51 skrev "Jason Lixfeld" <jason@lixfeld.ca mailto:jason@lixfeld.ca>: Mostly GPON ports (basically Ethernet, but I don’t know the ifType off the top of my head). Some transparent bridges too. Now to be fair, not all 10,000 ports are up, but I don’t recall if there’s a way to only poll ports that are up. I suppose though that you need to first poll all ports to determine status, then poll again only the ones you want to really poll. Cart, Horse. :)
On Oct 2, 2017, at 2:09 PM, Adam Armstrong <adama@memetic.org mailto:adama@memetic.org> wrote:
10k ports is going to take a long time on a device without a blistering fast control plane.
What are the ports, mostly?
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=10777 On 2 Oct 2017, at 19:05, Jason Lixfeld <jason@lixfeld.ca mailto:jason@lixfeld.ca> wrote: Hey all,
I’ve got a device (Zhone MXK-F) with over 10,000 ports on it, and Observium is taking well over 5 minutes to poll it (upwards of 15 minutes, actually). While there isn’t a definition specifically for MXK-F, it’s getting detected as MXK. os.inc.php is setting max-rep 50, so it’s at least trying to bulk walk, but I don’t know if it’s actually working. Overriding to max-rep 100 doesn’t have an affect on the polling time.
So, I guess my question is whether or not I’m on the right track in assuming that this issue is likely that of bulk walk on this device, or possibly some other issue, and how would I determine one way or the other? If it’s something that I can pinpoint, I can throw it back to the vendor to fix.
From the digging I’ve done so far, all other poller processes called from poller-wrapper complete in under 5 minutes, so I’m fairly certain that this issue is one of the device, not Observium. But, to be thorough, I moved my database and RRD storage over to local SSDs, and also enabled rrdcached - all that to ensure I didn’t have any I/O issues.
/pollerlog/view=devices indicates that it’s just the one device that has high timing (although other devices of the same type are spots 2-7 on the top 10 list high timers, but none is over 300 seconds except for this guy).
root@monitor:/home/jlixfeld# /opt/observium/poller.php -VV Observium 17.9.8837
##### Software versions #####
o OS Linux 3.16.0-4-amd64 [amd64] (Debian 8 (jessie)) o Apache 2.4.10 o PHP 5.6.30-0+deb8u1 (OPcache: DISABLED) o Python 2.7.9 o MySQL 5.5.55-0+deb8u1 (extension: mysqli 5.5.55) o SNMP NET-SNMP 5.7.2.1 http://5.7.2.1/ o RRDtool 1.4.8 (rrdcached 1.4.8: unix:/opt/observium/rrdcached.sock)
##### Memory Limit #####
o PHP Unlimited
##### MySQL mode #####
o MySQL
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Monday, 02-Oct-17 12:55:15 EDT o PHP -04:00 o MySQL -04:00
root@monitor:/home/jlixfeld#
Thanks all!
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
I have noticed this, too. For some reason Observium takes longer to poll a big switch than it does to walk it for discovery!
What has been killing me are the HPE Comware IRF Cluster switches. Evidently someone LOVES sensors because I have a cluster that has 600+ ports and over 1,000 sensors. Not only does this impact polling, but the user interface really struggles with all of those sensors.
Beware, my observations are based on the CE from last year. Have not upgraded yet.
Greg Hubbard
On Mon, Oct 23, 2017 at 8:51 AM Jason Lixfeld jason@lixfeld.ca wrote:
Hey,
Got it. Thanks. I think this will work. This box uses ‘other’ ifType quite a bit, and it can be filtered out. Doing so dropped the port count on this device from over 10,000 to just over 1,500.
With regards to snmpbulkwalk vs snmpwalk - I’m a little unclear on where the filter is used and when it isn’t; trying to understand how this all works a bit better. So a couple questions, if you don’t mind -
- Are you saying that the filter is only used during walk and not during
bulkwalk?
- The contextual help in the GUI when populating these bad_if* fields
suggests that these are considered during discovery, not polling. So 1. Every time discovery runs, it creates a list of ‘valid’ interfaces based on the results of snmp(bulk?)walk, then removing anything in bad_if* 2. Every polling interval it walks only that list for a device if the device has separate_walk enabled, and if the device does not have separate_walk enabled, the poller will just bulkwalk the entire table and stick everything into the database, create graphs for it all, etc?
Also, FWIW, manually doing a walk on IF only takes about 10s longer to complete than bulkwalk on just over 48,000 entries in the table for this particular device; this is the most dense device. That’s interesting to me in that I thought a bulkwalk would take much, much less time, or is the time dependant on a bunch of other stuff too?
Thanks in advance!
On Oct 23, 2017, at 2:23 AM, Markus Klock markus@best-practice.se wrote:
Hey, The default way of SNMP-polling ports is by doing a snmpbulkwalk of the whole table of interfaces on a device as this is usually faster. But in some cases the table of interfaces is huge, and you are really only interested in a smaller part of the interfaces, then the separate_walk-feature is usefull. Start by filtering out the interfaces you do not want by using interface filters: http://docs.observium.org/config_options/#filters_1 Then when the separate_walk-feature is enabled for the device, Observium will only do single snmpgets of the interfaces not filtered out instead of bulkwalking the whole interface table. You enable the feature by device->properties->modules->separate_walk=enabled
/Markus
2017-10-23 3:05 GMT+02:00 Jason Lixfeld jason@lixfeld.ca:
Hey Adam,
To circle back around to this - I’m not sure what you mean by separate walk ports poller method. Is this concept on the site somewhere that I can read up on?
Thanks!
On Oct 2, 2017, at 3:12 PM, Jason Lixfeld jason@lixfeld.ca wrote:
We’d be relying on the transparent bridge interfaces at that point. While that might work today, each customer occupies a > 1 number of transparent bridge interfaces, depending on the number of services they have subscribed to. At a certain point, the number of transparent bridge ports would eclipse the number of GPON ports.
On Oct 2, 2017, at 3:06 PM, Adam Armstrong adama@memetic.org wrote:
If you can deal with not having the pon interfaces, I'd filter them out somehow and switch to the separate walk ports poller method.
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=10777 On 2 Oct 2017, at 20:05, Jason Lixfeld jason@lixfeld.ca wrote:
It would be each active customer’s port, and the device uplink(s). So pretty much a constantly moving target :/
On Oct 2, 2017, at 2:54 PM, Markus Klock markus@best-practice.se wrote:
Do you want to graph the traffic of all the 10 000 ports? You could filter out only the ports you are interested in and have Observium only poll those. Assuming the ports you are interested in are maybe uplinks and stuff, like <50 ports.
Den 2 okt. 2017 20:51 skrev "Jason Lixfeld" jason@lixfeld.ca:
Mostly GPON ports (basically Ethernet, but I don’t know the ifType off the top of my head). Some transparent bridges too. Now to be fair, not all 10,000 ports are up, but I don’t recall if there’s a way to only poll ports that are up. I suppose though that you need to first poll all ports to determine status, then poll again only the ones you want to really poll. Cart, Horse. :)
On Oct 2, 2017, at 2:09 PM, Adam Armstrong adama@memetic.org wrote:
10k ports is going to take a long time on a device without a blistering fast control plane.
What are the ports, mostly?
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=10777 On 2 Oct 2017, at 19:05, Jason Lixfeld jason@lixfeld.ca wrote:
Hey all,
I’ve got a device (Zhone MXK-F) with over 10,000 ports on it, and Observium is taking well over 5 minutes to poll it (upwards of 15 minutes, actually). While there isn’t a definition specifically for MXK-F, it’s getting detected as MXK. os.inc.php is setting max-rep 50, so it’s at least trying to bulk walk, but I don’t know if it’s actually working. Overriding to max-rep 100 doesn’t have an affect on the polling time.
So, I guess my question is whether or not I’m on the right track in assuming that this issue is likely that of bulk walk on this device, or possibly some other issue, and how would I determine one way or the other? If it’s something that I can pinpoint, I can throw it back to the vendor to fix.
From the digging I’ve done so far, all other poller processes called from poller-wrapper complete in under 5 minutes, so I’m fairly certain that this issue is one of the device, not Observium. But, to be thorough, I moved my database and RRD storage over to local SSDs, and also enabled rrdcached - all that to ensure I didn’t have any I/O issues.
/pollerlog/view=devices indicates that it’s just the one device that has high timing (although other devices of the same type are spots 2-7 on the top 10 list high timers, but none is over 300 seconds except for this guy).
root@monitor:/home/jlixfeld# /opt/observium/poller.php -VV Observium 17.9.8837
##### Software versions #####
o OS Linux 3.16.0-4-amd64 [amd64] (Debian 8 (jessie)) o Apache 2.4.10 o PHP 5.6.30-0+deb8u1 (OPcache: DISABLED) o Python 2.7.9 o MySQL 5.5.55-0+deb8u1 (extension: mysqli 5.5.55) o SNMP NET-SNMP 5.7.2.1 o RRDtool 1.4.8 (rrdcached 1.4.8: unix:/opt/observium/rrdcached.sock)
##### Memory Limit #####
o PHP Unlimited
##### MySQL mode #####
o MySQL
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Monday, 02-Oct-17 12:55:15 EDT o PHP -04:00 o MySQL -04:00
root@monitor:/home/jlixfeld#
Thanks all!
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
The "separate walk" module actually uses snmpget. It gets all of the OIDs for each interface in a single query. It makes it a little more resistant to broken queries, but it can be a bit slower (more snmp binaries being called and more RTT latency)
It's confusing, I know.
BTW, there's no reason an snmpwalk would take less time than a bulkwalk unless your device has a really broken snmp stack.
adam. On 2017-10-23 14:51:31, Jason Lixfeld jason@lixfeld.ca wrote: Hey,
Got it. Thanks. I think this will work. This box uses ‘other’ ifType quite a bit, and it can be filtered out. Doing so dropped the port count on this device from over 10,000 to just over 1,500.
With regards to snmpbulkwalk vs snmpwalk - I’m a little unclear on where the filter is used and when it isn’t; trying to understand how this all works a bit better. So a couple questions, if you don’t mind -
- Are you saying that the filter is only used during walk and not during bulkwalk? - The contextual help in the GUI when populating these bad_if* fields suggests that these are considered during discovery, not polling. So 1. Every time discovery runs, it creates a list of ‘valid’ interfaces based on the results of snmp(bulk?)walk, then removing anything in bad_if* 2. Every polling interval it walks only that list for a device if the device has separate_walk enabled, and if the device does not have separate_walk enabled, the poller will just bulkwalk the entire table and stick everything into the database, create graphs for it all, etc?
Also, FWIW, manually doing a walk on IF only takes about 10s longer to complete than bulkwalk on just over 48,000 entries in the table for this particular device; this is the most dense device. That’s interesting to me in that I thought a bulkwalk would take much, much less time, or is the time dependant on a bunch of other stuff too?
Thanks in advance!
On Oct 23, 2017, at 2:23 AM, Markus Klock <markus@best-practice.se [mailto:markus@best-practice.se]> wrote:
Hey, The default way of SNMP-polling ports is by doing a snmpbulkwalk of the whole table of interfaces on a device as this is usually faster. But in some cases the table of interfaces is huge, and you are really only interested in a smaller part of the interfaces, then the separate_walk-feature is usefull. Start by filtering out the interfaces you do not want by using interface filters: http://docs.observium.org/config_options/#filters_1 [http://docs.observium.org/config_options/#filters_1] Then when the separate_walk-feature is enabled for the device, Observium will only do single snmpgets of the interfaces not filtered out instead of bulkwalking the whole interface table. You enable the feature by device->properties->modules->separate_walk=enabled
/Markus
2017-10-23 3:05 GMT+02:00 Jason Lixfeld <jason@lixfeld.ca [mailto:jason@lixfeld.ca]>:
Hey Adam,
To circle back around to this - I’m not sure what you mean by separate walk ports poller method. Is this concept on the site somewhere that I can read up on?
Thanks!
On Oct 2, 2017, at 3:12 PM, Jason Lixfeld <jason@lixfeld.ca [mailto:jason@lixfeld.ca]> wrote:
We’d be relying on the transparent bridge interfaces at that point. While that might work today, each customer occupies a > 1 number of transparent bridge interfaces, depending on the number of services they have subscribed to. At a certain point, the number of transparent bridge ports would eclipse the number of GPON ports.
On Oct 2, 2017, at 3:06 PM, Adam Armstrong <adama@memetic.org [mailto:adama@memetic.org]> wrote:
If you can deal with not having the pon interfaces, I'd filter them out somehow and switch to the separate walk ports poller method.
Adam.
Sent from BlueMail [http://www.bluemail.me/r?b=10777] On 2 Oct 2017, at 20:05, Jason Lixfeld <jason@lixfeld.ca [mailto:jason@lixfeld.ca]> wrote: It would be each active customer’s port, and the device uplink(s). So pretty much a constantly moving target :/
On Oct 2, 2017, at 2:54 PM, Markus Klock <markus@best-practice.se [mailto:markus@best-practice.se]> wrote:
Do you want to graph the traffic of all the 10 000 ports? You could filter out only the ports you are interested in and have Observium only poll those. Assuming the ports you are interested in are maybe uplinks and stuff, like <50 ports.
Den 2 okt. 2017 20:51 skrev "Jason Lixfeld" <jason@lixfeld.ca [mailto:jason@lixfeld.ca]>:
Mostly GPON ports (basically Ethernet, but I don’t know the ifType off the top of my head). Some transparent bridges too. Now to be fair, not all 10,000 ports are up, but I don’t recall if there’s a way to only poll ports that are up. I suppose though that you need to first poll all ports to determine status, then poll again only the ones you want to really poll. Cart, Horse. :)
On Oct 2, 2017, at 2:09 PM, Adam Armstrong <adama@memetic.org [mailto:adama@memetic.org]> wrote:
10k ports is going to take a long time on a device without a blistering fast control plane.
What are the ports, mostly?
Adam.
Sent from BlueMail [http://www.bluemail.me/r?b=10777] On 2 Oct 2017, at 19:05, Jason Lixfeld <jason@lixfeld.ca [mailto:jason@lixfeld.ca]> wrote: Hey all,
I’ve got a device (Zhone MXK-F) with over 10,000 ports on it, and Observium is taking well over 5 minutes to poll it (upwards of 15 minutes, actually). While there isn’t a definition specifically for MXK-F, it’s getting detected as MXK. os.inc.php is setting max-rep 50, so it’s at least trying to bulk walk, but I don’t know if it’s actually working. Overriding to max-rep 100 doesn’t have an affect on the polling time.
So, I guess my question is whether or not I’m on the right track in assuming that this issue is likely that of bulk walk on this device, or possibly some other issue, and how would I determine one way or the other? If it’s something that I can pinpoint, I can throw it back to the vendor to fix.
From the digging I’ve done so far, all other poller processes called from poller-wrapper complete in under 5 minutes, so I’m fairly certain that this issue is one of the device, not Observium. But, to be thorough, I moved my database and RRD storage over to local SSDs, and also enabled rrdcached - all that to ensure I didn’t have any I/O issues.
/pollerlog/view=devices indicates that it’s just the one device that has high timing (although other devices of the same type are spots 2-7 on the top 10 list high timers, but none is over 300 seconds except for this guy).
root@monitor:/home/jlixfeld# /opt/observium/poller.php -VV Observium 17.9.8837
##### Software versions #####
o OS Linux 3.16.0-4-amd64 [amd64] (Debian 8 (jessie)) o Apache 2.4.10 o PHP 5.6.30-0+deb8u1 (OPcache: DISABLED) o Python 2.7.9 o MySQL 5.5.55-0+deb8u1 (extension: mysqli 5.5.55) o SNMP NET-SNMP 5.7.2.1 [http://5.7.2.1/] o RRDtool 1.4.8 (rrdcached 1.4.8: unix:/opt/observium/rrdcached.sock)
##### Memory Limit #####
o PHP Unlimited
##### MySQL mode #####
o MySQL
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Monday, 02-Oct-17 12:55:15 EDT o PHP -04:00 o MySQL -04:00
root@monitor:/home/jlixfeld#
Thanks all!
observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (5)
-
Adam Armstrong -
Adam Armstrong -
Greg Hubbard -
Jason Lixfeld -
Markus Klock