Hello,
Just curious if anyone else is seeing long polling times for Nexus 5596UP switches. Everything else in our deployment is responding in a timely manner and there are no bottlenecks on the Observium server at this point.
./poller.php: access2-2a.sj2 - 1 devices polled in 243.6 secs
Polling times in the high 200 second range are not uncommon while most all of our other high port density devices are well under 30 seconds. If anyone has found a NXOS software version that works better than others please let me know. We have tried on various software version 5,6 and 7 releases. I can only imagine this is a Cisco issue but thought I'd ask here in case anyone solved it.
Regards,
Dan Bliss
Hi Dan,
Nexus are the worst devices in terms of SNMP polling, according to my experience, especially if you are using older firmwares. Discovering is even worse.
Though, I solved the slowness by disabling advanced vlan related MIBs, whose polling data is not represented anyway. Upgrading the devices firmware is also helpful.
Regards, Marco
________________________________________ From: observium observium-bounces@observium.org on behalf of Daniel Bliss dbliss@datapipe.com Sent: 02 November 2014 15:43 To: observium@observium.org Subject: [Observium] Nexus 5596 polling slowness
Hello,
Just curious if anyone else is seeing long polling times for Nexus 5596UP switches. Everything else in our deployment is responding in a timely manner and there are no bottlenecks on the Observium server at this point.
./poller.php: access2-2a.sj2 - 1 devices polled in 243.6 secs
Polling times in the high 200 second range are not uncommon while most all of our other high port density devices are well under 30 seconds. If anyone has found a NXOS software version that works better than others please let me know. We have tried on various software version 5,6 and 7 releases. I can only imagine this is a Cisco issue but thought I'd ask here in case anyone solved it.
Regards,
Dan Bliss
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Marco,
Thanks for the reply. Yep, I have the vlan module disabled. Specifically though, the slowness shows up in the ports module. Debugged using ./poller.php -h hostname -r -d -m ports. I have checked everything down to the control plane policy on the switch including disabling it while polling even though there were no policy violations. Bottom line is the switch is just slow to respond. Checking the release notes, there were numerous snmp related bug fixes in past code however nothing recent. Going to try stepping though some software updates next to see if anything changes in any more recent code revs.
Regards,
Dan ________________________________________ From: observium [observium-bounces@observium.org] on behalf of Marco Spicuglia [Marco.Spicuglia@reasonnet.com] Sent: Sunday, November 02, 2014 12:42 PM To: observium@observium.org Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Dan,
Nexus are the worst devices in terms of SNMP polling, according to my experience, especially if you are using older firmwares. Discovering is even worse.
Though, I solved the slowness by disabling advanced vlan related MIBs, whose polling data is not represented anyway. Upgrading the devices firmware is also helpful.
Regards, Marco
________________________________________ From: observium observium-bounces@observium.org on behalf of Daniel Bliss dbliss@datapipe.com Sent: 02 November 2014 15:43 To: observium@observium.org Subject: [Observium] Nexus 5596 polling slowness
Hello,
Just curious if anyone else is seeing long polling times for Nexus 5596UP switches. Everything else in our deployment is responding in a timely manner and there are no bottlenecks on the Observium server at this point.
./poller.php: access2-2a.sj2 - 1 devices polled in 243.6 secs
Polling times in the high 200 second range are not uncommon while most all of our other high port density devices are well under 30 seconds. If anyone has found a NXOS software version that works better than others please let me know. We have tried on various software version 5,6 and 7 releases. I can only imagine this is a Cisco issue but thought I'd ask here in case anyone solved it.
Regards,
Dan Bliss
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
IMHO, Nexus are garbage in terms of SNMP. This protocol is a wedged-in afterthought as Cisco was trying to upsell their NetConf-based monitoring products and didn't even have SNMP support originally until enough customers complained. This entire product line has caused our company nothing but grief in terms of anything SNMP (monitoring/inventory) as well as other factors such as prematurely dying power supplies. We've had to hack together so many workarounds and hacks to fold these into the fray that it's approached obscene at times.
That said, I wish you the best of luck with your experience with these. Hopefully you'll find some relief on your devices with newer code versions. We're still intermittently smacking our heads on random bugs and flaws.
Cheers, -Chris
On 11/2/14 9:30 PM, Daniel Bliss wrote:
Hi Marco,
Thanks for the reply. Yep, I have the vlan module disabled. Specifically though, the slowness shows up in the ports module. Debugged using ./poller.php -h hostname -r -d -m ports. I have checked everything down to the control plane policy on the switch including disabling it while polling even though there were no policy violations. Bottom line is the switch is just slow to respond. Checking the release notes, there were numerous snmp related bug fixes in past code however nothing recent. Going to try stepping though some software updates next to see if anything changes in any more recent code revs.
Regards,
Dan ________________________________________ From: observium [observium-bounces@observium.org] on behalf of Marco Spicuglia [Marco.Spicuglia@reasonnet.com] Sent: Sunday, November 02, 2014 12:42 PM To: observium@observium.org Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Dan,
Nexus are the worst devices in terms of SNMP polling, according to my experience, especially if you are using older firmwares. Discovering is even worse.
Though, I solved the slowness by disabling advanced vlan related MIBs, whose polling data is not represented anyway. Upgrading the devices firmware is also helpful.
Regards, Marco
From: observium observium-bounces@observium.org on behalf of Daniel Bliss dbliss@datapipe.com Sent: 02 November 2014 15:43 To: observium@observium.org Subject: [Observium] Nexus 5596 polling slowness
Hello,
Just curious if anyone else is seeing long polling times for Nexus 5596UP switches. Everything else in our deployment is responding in a timely manner and there are no bottlenecks on the Observium server at this point.
./poller.php: access2-2a.sj2 - 1 devices polled in 243.6 secs
Polling times in the high 200 second range are not uncommon while most all of our other high port density devices are well under 30 seconds. If anyone has found a NXOS software version that works better than others please let me know. We have tried on various software version 5,6 and 7 releases. I can only imagine this is a Cisco issue but thought I'd ask here in case anyone solved it.
Regards,
Dan Bliss
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
------ Original Message ------ From: "Chris Moody" chris@node-nine.com To: "Observium Network Observation System" observium@observium.org Sent: 11/2/2014 3:24:26 PM Subject: Re: [Observium] Nexus 5596 polling slowness
IMHO, Nexus are garbage in terms of SNMP. This protocol is a wedged-in afterthought as Cisco was trying to upsell their NetConf-based monitoring products and didn't even have SNMP support originally until enough customers complained.
Holy shit. This is information I didn't know. It explains *everything*!
adam.
Hi Dan,
after disabling a couple of vlan MIBs in the polling (moved the mib file away from the observium mibs directory), I was able to bring the polling of a couple of NX6K, each with 6 times NX2K FEX and thus hundreds of ports, down from about 700 seconds to 60 seconds (still the slowest devices in the network). The discovery is now down from 999+ seconds to 11-12 seconds. These results were reached without upgrading the firmware and without loosing any data from the Observium interface.
Considering the fact that those 6Ks were running in a fabric path with all possible vlans allowed on it, vlan data polling made those nexus extremely slow in SNMP response times. Specifically for one of the vlan MIBs I disabled (can't remember its name now), Cisco was recommending no other alternative than not polling with that MIB at all, if using certain NXOS versions.
Still now, while performing the 5min regular polling, the FEXs are often so busy that the parent 6Ks lose their peering keep alive. And observium thinks that all their ports suddenly disappeared from the network. Data traffic seems to be not affected though.
The latest firmware version for the NX6K is supposed to have solved many polling problems, hopefully. Including the advanced vlan and vtp slow polling issues such as above. But I haven't tested it yet. We'll see.
Regards, Marco
________________________________________ From: observium observium-bounces@observium.org on behalf of Daniel Bliss dbliss@datapipe.com Sent: 02 November 2014 21:30 To: Observium Network Observation System Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Marco,
Thanks for the reply. Yep, I have the vlan module disabled. Specifically though, the slowness shows up in the ports module. Debugged using ./poller.php -h hostname -r -d -m ports. I have checked everything down to the control plane policy on the switch including disabling it while polling even though there were no policy violations. Bottom line is the switch is just slow to respond. Checking the release notes, there were numerous snmp related bug fixes in past code however nothing recent. Going to try stepping though some software updates next to see if anything changes in any more recent code revs.
Regards,
Dan ________________________________________ From: observium [observium-bounces@observium.org] on behalf of Marco Spicuglia [Marco.Spicuglia@reasonnet.com] Sent: Sunday, November 02, 2014 12:42 PM To: observium@observium.org Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Dan,
Nexus are the worst devices in terms of SNMP polling, according to my experience, especially if you are using older firmwares. Discovering is even worse.
Though, I solved the slowness by disabling advanced vlan related MIBs, whose polling data is not represented anyway. Upgrading the devices firmware is also helpful.
Regards, Marco
________________________________________ From: observium observium-bounces@observium.org on behalf of Daniel Bliss dbliss@datapipe.com Sent: 02 November 2014 15:43 To: observium@observium.org Subject: [Observium] Nexus 5596 polling slowness
Hello,
Just curious if anyone else is seeing long polling times for Nexus 5596UP switches. Everything else in our deployment is responding in a timely manner and there are no bottlenecks on the Observium server at this point.
./poller.php: access2-2a.sj2 - 1 devices polled in 243.6 secs
Polling times in the high 200 second range are not uncommon while most all of our other high port density devices are well under 30 seconds. If anyone has found a NXOS software version that works better than others please let me know. We have tried on various software version 5,6 and 7 releases. I can only imagine this is a Cisco issue but thought I'd ask here in case anyone solved it.
Regards,
Dan Bliss
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Do not move anything in the mibs directory.
Disable the module. Duh.
adam. ------ Original Message ------ From: "Marco Spicuglia" Marco.Spicuglia@reasonnet.com To: "Observium Network Observation System" observium@observium.org Sent: 11/2/2014 6:02:46 PM Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Dan,
after disabling a couple of vlan MIBs in the polling (moved the mib file away from the observium mibs directory), I was able to bring the polling of a couple of NX6K, each with 6 times NX2K FEX and thus hundreds of ports, down from about 700 seconds to 60 seconds (still the slowest devices in the network). The discovery is now down from 999+ seconds to 11-12 seconds. These results were reached without upgrading the firmware and without loosing any data from the Observium interface.
Considering the fact that those 6Ks were running in a fabric path with all possible vlans allowed on it, vlan data polling made those nexus extremely slow in SNMP response times. Specifically for one of the vlan MIBs I disabled (can't remember its name now), Cisco was recommending no other alternative than not polling with that MIB at all, if using certain NXOS versions.
Still now, while performing the 5min regular polling, the FEXs are often so busy that the parent 6Ks lose their peering keep alive. And observium thinks that all their ports suddenly disappeared from the network. Data traffic seems to be not affected though.
The latest firmware version for the NX6K is supposed to have solved many polling problems, hopefully. Including the advanced vlan and vtp slow polling issues such as above. But I haven't tested it yet. We'll see.
Regards, Marco
From: observium observium-bounces@observium.org on behalf of Daniel Bliss dbliss@datapipe.com Sent: 02 November 2014 21:30 To: Observium Network Observation System Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Marco,
Thanks for the reply. Yep, I have the vlan module disabled. Specifically though, the slowness shows up in the ports module. Debugged using ./poller.php -h hostname -r -d -m ports. I have checked everything down to the control plane policy on the switch including disabling it while polling even though there were no policy violations. Bottom line is the switch is just slow to respond. Checking the release notes, there were numerous snmp related bug fixes in past code however nothing recent. Going to try stepping though some software updates next to see if anything changes in any more recent code revs.
Regards,
Dan ________________________________________ From: observium [observium-bounces@observium.org] on behalf of Marco Spicuglia [Marco.Spicuglia@reasonnet.com] Sent: Sunday, November 02, 2014 12:42 PM To: observium@observium.org Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Dan,
Nexus are the worst devices in terms of SNMP polling, according to my experience, especially if you are using older firmwares. Discovering is even worse.
Though, I solved the slowness by disabling advanced vlan related MIBs, whose polling data is not represented anyway. Upgrading the devices firmware is also helpful.
Regards, Marco
From: observium observium-bounces@observium.org on behalf of Daniel Bliss dbliss@datapipe.com Sent: 02 November 2014 15:43 To: observium@observium.org Subject: [Observium] Nexus 5596 polling slowness
Hello,
Just curious if anyone else is seeing long polling times for Nexus 5596UP switches. Everything else in our deployment is responding in a timely manner and there are no bottlenecks on the Observium server at this point.
./poller.php: access2-2a.sj2 - 1 devices polled in 243.6 secs
Polling times in the high 200 second range are not uncommon while most all of our other high port density devices are well under 30 seconds. If anyone has found a NXOS software version that works better than others please let me know. We have tried on various software version 5,6 and 7 releases. I can only imagine this is a Cisco issue but thought I'd ask here in case anyone solved it.
Regards,
Dan Bliss
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Disabling the module had not made any difference. If it's fixed now, I'd rather choose that method for sure.
Cheers, Marco
________________________________________ From: observium observium-bounces@observium.org on behalf of Adam Armstrong adama@memetic.org Sent: 03 November 2014 01:07 To: Observium Network Observation System Subject: Re: [Observium] Nexus 5596 polling slowness
Do not move anything in the mibs directory.
Disable the module. Duh.
adam. ------ Original Message ------ From: "Marco Spicuglia" Marco.Spicuglia@reasonnet.com To: "Observium Network Observation System" observium@observium.org Sent: 11/2/2014 6:02:46 PM Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Dan,
after disabling a couple of vlan MIBs in the polling (moved the mib file away from the observium mibs directory), I was able to bring the polling of a couple of NX6K, each with 6 times NX2K FEX and thus hundreds of ports, down from about 700 seconds to 60 seconds (still the slowest devices in the network). The discovery is now down from 999+ seconds to 11-12 seconds. These results were reached without upgrading the firmware and without loosing any data from the Observium interface.
Considering the fact that those 6Ks were running in a fabric path with all possible vlans allowed on it, vlan data polling made those nexus extremely slow in SNMP response times. Specifically for one of the vlan MIBs I disabled (can't remember its name now), Cisco was recommending no other alternative than not polling with that MIB at all, if using certain NXOS versions.
Still now, while performing the 5min regular polling, the FEXs are often so busy that the parent 6Ks lose their peering keep alive. And observium thinks that all their ports suddenly disappeared from the network. Data traffic seems to be not affected though.
The latest firmware version for the NX6K is supposed to have solved many polling problems, hopefully. Including the advanced vlan and vtp slow polling issues such as above. But I haven't tested it yet. We'll see.
Regards, Marco
From: observium observium-bounces@observium.org on behalf of Daniel Bliss dbliss@datapipe.com Sent: 02 November 2014 21:30 To: Observium Network Observation System Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Marco,
Thanks for the reply. Yep, I have the vlan module disabled. Specifically though, the slowness shows up in the ports module. Debugged using ./poller.php -h hostname -r -d -m ports. I have checked everything down to the control plane policy on the switch including disabling it while polling even though there were no policy violations. Bottom line is the switch is just slow to respond. Checking the release notes, there were numerous snmp related bug fixes in past code however nothing recent. Going to try stepping though some software updates next to see if anything changes in any more recent code revs.
Regards,
Dan ________________________________________ From: observium [observium-bounces@observium.org] on behalf of Marco Spicuglia [Marco.Spicuglia@reasonnet.com] Sent: Sunday, November 02, 2014 12:42 PM To: observium@observium.org Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Dan,
Nexus are the worst devices in terms of SNMP polling, according to my experience, especially if you are using older firmwares. Discovering is even worse.
Though, I solved the slowness by disabling advanced vlan related MIBs, whose polling data is not represented anyway. Upgrading the devices firmware is also helpful.
Regards, Marco
From: observium observium-bounces@observium.org on behalf of Daniel Bliss dbliss@datapipe.com Sent: 02 November 2014 15:43 To: observium@observium.org Subject: [Observium] Nexus 5596 polling slowness
Hello,
Just curious if anyone else is seeing long polling times for Nexus 5596UP switches. Everything else in our deployment is responding in a timely manner and there are no bottlenecks on the Observium server at this point.
./poller.php: access2-2a.sj2 - 1 devices polled in 243.6 secs
Polling times in the high 200 second range are not uncommon while most all of our other high port density devices are well under 30 seconds. If anyone has found a NXOS software version that works better than others please let me know. We have tried on various software version 5,6 and 7 releases. I can only imagine this is a Cisco issue but thought I'd ask here in case anyone solved it.
Regards,
Dan Bliss
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
You disabled the wrong module then, obviously.
I think the correct solution 'for sure' is actually identifying and fixing the problem, rather than deleting files until it goes away.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 2 November 2014 18:12:19 Marco Spicuglia Marco.Spicuglia@reasonnet.com wrote:
Disabling the module had not made any difference. If it's fixed now, I'd rather choose that method for sure.
Cheers, Marco
From: observium observium-bounces@observium.org on behalf of Adam Armstrong adama@memetic.org Sent: 03 November 2014 01:07 To: Observium Network Observation System Subject: Re: [Observium] Nexus 5596 polling slowness
Do not move anything in the mibs directory.
Disable the module. Duh.
adam. ------ Original Message ------ From: "Marco Spicuglia" Marco.Spicuglia@reasonnet.com To: "Observium Network Observation System" observium@observium.org Sent: 11/2/2014 6:02:46 PM Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Dan,
after disabling a couple of vlan MIBs in the polling (moved the mib file away from the observium mibs directory), I was able to bring the polling of a couple of NX6K, each with 6 times NX2K FEX and thus hundreds of ports, down from about 700 seconds to 60 seconds (still the slowest devices in the network). The discovery is now down from 999+ seconds to 11-12 seconds. These results were reached without upgrading the firmware and without loosing any data from the Observium interface.
Considering the fact that those 6Ks were running in a fabric path with all possible vlans allowed on it, vlan data polling made those nexus extremely slow in SNMP response times. Specifically for one of the vlan MIBs I disabled (can't remember its name now), Cisco was recommending no other alternative than not polling with that MIB at all, if using certain NXOS versions.
Still now, while performing the 5min regular polling, the FEXs are often so busy that the parent 6Ks lose their peering keep alive. And observium thinks that all their ports suddenly disappeared from the network. Data traffic seems to be not affected though.
The latest firmware version for the NX6K is supposed to have solved many polling problems, hopefully. Including the advanced vlan and vtp slow polling issues such as above. But I haven't tested it yet. We'll see.
Regards, Marco
From: observium observium-bounces@observium.org on behalf of Daniel Bliss dbliss@datapipe.com Sent: 02 November 2014 21:30 To: Observium Network Observation System Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Marco,
Thanks for the reply. Yep, I have the vlan module disabled. Specifically though, the slowness shows up in the ports module. Debugged using ./poller.php -h hostname -r -d -m ports. I have checked everything down to the control plane policy on the switch including disabling it while polling even though there were no policy violations. Bottom line is the switch is just slow to respond. Checking the release notes, there were numerous snmp related bug fixes in past code however nothing recent. Going to try stepping though some software updates next to see if anything changes in any more recent code revs.
Regards,
Dan ________________________________________ From: observium [observium-bounces@observium.org] on behalf of Marco Spicuglia [Marco.Spicuglia@reasonnet.com] Sent: Sunday, November 02, 2014 12:42 PM To: observium@observium.org Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Dan,
Nexus are the worst devices in terms of SNMP polling, according to my experience, especially if you are using older firmwares. Discovering is even worse.
Though, I solved the slowness by disabling advanced vlan related MIBs, whose polling data is not represented anyway. Upgrading the devices firmware is also helpful.
Regards, Marco
From: observium observium-bounces@observium.org on behalf of Daniel Bliss dbliss@datapipe.com Sent: 02 November 2014 15:43 To: observium@observium.org Subject: [Observium] Nexus 5596 polling slowness
Hello,
Just curious if anyone else is seeing long polling times for Nexus 5596UP switches. Everything else in our deployment is responding in a timely manner and there are no bottlenecks on the Observium server at this point.
./poller.php: access2-2a.sj2 - 1 devices polled in 243.6 secs
Polling times in the high 200 second range are not uncommon while most all of our other high port density devices are well under 30 seconds. If anyone has found a NXOS software version that works better than others please let me know. We have tried on various software version 5,6 and 7 releases. I can only imagine this is a Cisco issue but thought I'd ask here in case anyone solved it.
Regards,
Dan Bliss
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Adam,
I know you only write perfect software for sure, otherwise I would not use it :) However, if I got to the point of deleting a MIB, it probably means that it was needed.
No worth discussing about this anyway, too long ago, too different versions and conditions. I'll soon test it in a new network and keep you posted in case. Have a nice day!
Cheers, Marco
________________________________________ From: observium observium-bounces@observium.org on behalf of Adam Armstrong adama@memetic.org Sent: 03 November 2014 02:50 To: observium@observium.org Subject: Re: [Observium] Nexus 5596 polling slowness
You disabled the wrong module then, obviously.
I think the correct solution 'for sure' is actually identifying and fixing the problem, rather than deleting files until it goes away.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 2 November 2014 18:12:19 Marco Spicuglia Marco.Spicuglia@reasonnet.com wrote:
Disabling the module had not made any difference. If it's fixed now, I'd rather choose that method for sure.
Cheers, Marco
From: observium observium-bounces@observium.org on behalf of Adam Armstrong adama@memetic.org Sent: 03 November 2014 01:07 To: Observium Network Observation System Subject: Re: [Observium] Nexus 5596 polling slowness
Do not move anything in the mibs directory.
Disable the module. Duh.
adam. ------ Original Message ------ From: "Marco Spicuglia" Marco.Spicuglia@reasonnet.com To: "Observium Network Observation System" observium@observium.org Sent: 11/2/2014 6:02:46 PM Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Dan,
after disabling a couple of vlan MIBs in the polling (moved the mib file away from the observium mibs directory), I was able to bring the polling of a couple of NX6K, each with 6 times NX2K FEX and thus hundreds of ports, down from about 700 seconds to 60 seconds (still the slowest devices in the network). The discovery is now down from 999+ seconds to 11-12 seconds. These results were reached without upgrading the firmware and without loosing any data from the Observium interface.
Considering the fact that those 6Ks were running in a fabric path with all possible vlans allowed on it, vlan data polling made those nexus extremely slow in SNMP response times. Specifically for one of the vlan MIBs I disabled (can't remember its name now), Cisco was recommending no other alternative than not polling with that MIB at all, if using certain NXOS versions.
Still now, while performing the 5min regular polling, the FEXs are often so busy that the parent 6Ks lose their peering keep alive. And observium thinks that all their ports suddenly disappeared from the network. Data traffic seems to be not affected though.
The latest firmware version for the NX6K is supposed to have solved many polling problems, hopefully. Including the advanced vlan and vtp slow polling issues such as above. But I haven't tested it yet. We'll see.
Regards, Marco
From: observium observium-bounces@observium.org on behalf of Daniel Bliss dbliss@datapipe.com Sent: 02 November 2014 21:30 To: Observium Network Observation System Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Marco,
Thanks for the reply. Yep, I have the vlan module disabled. Specifically though, the slowness shows up in the ports module. Debugged using ./poller.php -h hostname -r -d -m ports. I have checked everything down to the control plane policy on the switch including disabling it while polling even though there were no policy violations. Bottom line is the switch is just slow to respond. Checking the release notes, there were numerous snmp related bug fixes in past code however nothing recent. Going to try stepping though some software updates next to see if anything changes in any more recent code revs.
Regards,
Dan ________________________________________ From: observium [observium-bounces@observium.org] on behalf of Marco Spicuglia [Marco.Spicuglia@reasonnet.com] Sent: Sunday, November 02, 2014 12:42 PM To: observium@observium.org Subject: Re: [Observium] Nexus 5596 polling slowness
Hi Dan,
Nexus are the worst devices in terms of SNMP polling, according to my experience, especially if you are using older firmwares. Discovering is even worse.
Though, I solved the slowness by disabling advanced vlan related MIBs, whose polling data is not represented anyway. Upgrading the devices firmware is also helpful.
Regards, Marco
From: observium observium-bounces@observium.org on behalf of Daniel Bliss dbliss@datapipe.com Sent: 02 November 2014 15:43 To: observium@observium.org Subject: [Observium] Nexus 5596 polling slowness
Hello,
Just curious if anyone else is seeing long polling times for Nexus 5596UP switches. Everything else in our deployment is responding in a timely manner and there are no bottlenecks on the Observium server at this point.
./poller.php: access2-2a.sj2 - 1 devices polled in 243.6 secs
Polling times in the high 200 second range are not uncommon while most all of our other high port density devices are well under 30 seconds. If anyone has found a NXOS software version that works better than others please let me know. We have tried on various software version 5,6 and 7 releases. I can only imagine this is a Cisco issue but thought I'd ask here in case anyone solved it.
Regards,
Dan Bliss
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
What CPU these boxes running? Does Cisco decide to reuse some old CPU from unused stock of 90's generation of switches?
Or this is snmp stack that poorly written in NX-OS?
On 02.11.2014 20:42, Marco Spicuglia wrote:
Nexus are the worst devices in terms of SNMP polling, according to my experience, especially if you are using older firmwares. Discovering is even worse.
Hello, I have two Nexus 5548-switches with 6 FEXes connected in my lab running NX-OS 6.0(2)N1(2) and today I upgraded them to NX-OS 7.0(4)N1(1) for testing. It seems the SNMP performance is greatly increased. The polling time for my observium instance decreased from 60-80s to 20-30s for this switches. Performance-graph about an hour after the upgrade: http://best-practice.se/dump/nexus5k.PNG
/Markus
2014-11-02 15:43 GMT+01:00 Daniel Bliss dbliss@datapipe.com:
Hello,
Just curious if anyone else is seeing long polling times for Nexus 5596UP switches. Everything else in our deployment is responding in a timely manner and there are no bottlenecks on the Observium server at this point.
./poller.php: access2-2a.sj2 - 1 devices polled in 243.6 secs
Polling times in the high 200 second range are not uncommon while most all of our other high port density devices are well under 30 seconds. If anyone has found a NXOS software version that works better than others please let me know. We have tried on various software version 5,6 and 7 releases. I can only imagine this is a Cisco issue but thought I'd ask here in case anyone solved it.
Regards,
Dan Bliss
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (6)
-
Adam Armstrong -
Chris Moody -
Daniel Bliss -
Marco Spicuglia -
Markus Klock -
Nikolay Shopik