Is anyone else having huge response times in their polling of a Cisco 3750 stack?
I've increased the number of pollers I have to something close to 17, but my 3750s seem to take forever to respond (anywhere between 200 - 999.99+ seconds). It may be an issue on my 3750s, but my polling intervals on my devices is several hours.
I have 2950s and 6509s that don't seem to have any issue responding in a short amount of time (<20 seconds).
Yes we have the same issue on some of our Cisco devices, we pulled them out of the list for now. I would like to know how to fix this issue.
[cid:image001.png@01CE3C2D.42B89130]
From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Michael Sweikata Sent: Thursday, April 18, 2013 11:59 AM To: observium@observium.org Subject: [Observium] Polling Rate on a 3750 Stack
Is anyone else having huge response times in their polling of a Cisco 3750 stack?
I've increased the number of pollers I have to something close to 17, but my 3750s seem to take forever to respond (anywhere between 200 - 999.99+ seconds). It may be an issue on my 3750s, but my polling intervals on my devices is several hours.
I have 2950s and 6509s that don't seem to have any issue responding in a short amount of time (<20 seconds).
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Have you run the poller in debug mode to see what commands it is hanging on? I just tested our largest 3750 stack (796 ports, 8 sensors) and it finished in 37 seconds one time and 96 seconds another time. It is possible that the longer time conflicted with a production poller or that something else was happening on the switch. If you can find one command that is causing the hang up you can open a TAC case with Cisco to find out why the switch is slow.
Mike
On 4/18/13 10:07 AM, Donald T. Currie wrote:
Yes we have the same issue on some of our Cisco devices, we pulled them out of the list for now. I would like to know how to fix this issue.
signature
*From:*observium-bounces@observium.org [mailto:observium-bounces@observium.org] *On Behalf Of *Michael Sweikata *Sent:* Thursday, April 18, 2013 11:59 AM *To:* observium@observium.org *Subject:* [Observium] Polling Rate on a 3750 Stack
Is anyone else having huge response times in their polling of a Cisco 3750 stack?
I?ve increased the number of pollers I have to something close to 17, but my 3750s seem to take forever to respond (anywhere between 200 ? 999.99+ seconds). It may be an issue on my 3750s, but my polling intervals on my devices is several hours.
I have 2950s and 6509s that don?t seem to have any issue responding in a short amount of time (<20 seconds).
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
That is a very helpful solution. Running the debug command on a 3750 stack, it didn't seem to hang on one specific command, it just seemed to run very slow on gathering information per port. I'm inclined to say that it's a problem on the 3750, we're using SNMPv3 with authPriv.
I'll throw TAC the question and see if they know why return information from the snmpbulkwalk is taking forever. I'm just trying to eliminate the server or the local MySQL DB from being the issue.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Michael Robbert Sent: Thursday, April 18, 2013 12:12 PM To: observium@observium.org Subject: Re: [Observium] Polling Rate on a 3750 Stack
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Have you run the poller in debug mode to see what commands it is hanging on? I just tested our largest 3750 stack (796 ports, 8 sensors) and it finished in 37 seconds one time and 96 seconds another time. It is possible that the longer time conflicted with a production poller or that something else was happening on the switch. If you can find one command that is causing the hang up you can open a TAC case with Cisco to find out why the switch is slow.
Mike
On 4/18/13 10:07 AM, Donald T. Currie wrote:
Yes we have the same issue on some of our Cisco devices, we pulled them out of the list for now. I would like to know how to fix this issue.
signature
*From:*observium-bounces@observium.org [mailto:observium-bounces@observium.org] *On Behalf Of *Michael Sweikata *Sent:* Thursday, April 18, 2013 11:59 AM *To:* observium@observium.org *Subject:* [Observium] Polling Rate on a 3750 Stack
Is anyone else having huge response times in their polling of a Cisco 3750 stack?
I?ve increased the number of pollers I have to something close to 17, but my 3750s seem to take forever to respond (anywhere between 200 ? 999.99+ seconds). It may be an issue on my 3750s, but my polling intervals on my devices is several hours.
I have 2950s and 6509s that don?t seem to have any issue responding in a short amount of time (<20 seconds).
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
I had this same issue with one of my ISR 891 routers. Polling the vlan interfaces took ages compared to all the other 891s. This one slow device had a different IOS revision. Updated to match the "faster" routers and the issue went away.
15.0(1)M4 was the slow IOS, 15.1(4)M6 is much faster.
On Thu, Apr 18, 2013 at 10:47 AM, Michael Sweikata sweikatam1@nku.edu wrote:
That is a very helpful solution. Running the debug command on a 3750 stack, it didn't seem to hang on one specific command, it just seemed to run very slow on gathering information per port. I'm inclined to say that it's a problem on the 3750, we're using SNMPv3 with authPriv.
I'll throw TAC the question and see if they know why return information from the snmpbulkwalk is taking forever. I'm just trying to eliminate the server or the local MySQL DB from being the issue.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Michael Robbert Sent: Thursday, April 18, 2013 12:12 PM To: observium@observium.org Subject: Re: [Observium] Polling Rate on a 3750 Stack
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Have you run the poller in debug mode to see what commands it is hanging on? I just tested our largest 3750 stack (796 ports, 8 sensors) and it finished in 37 seconds one time and 96 seconds another time. It is possible that the longer time conflicted with a production poller or that something else was happening on the switch. If you can find one command that is causing the hang up you can open a TAC case with Cisco to find out why the switch is slow.
Mike
On 4/18/13 10:07 AM, Donald T. Currie wrote:
Yes we have the same issue on some of our Cisco devices, we pulled them out of the list for now. I would like to know how to fix this issue.
signature
*From:*observium-bounces@observium.org [mailto:observium-bounces@observium.org] *On Behalf Of *Michael Sweikata *Sent:* Thursday, April 18, 2013 11:59 AM *To:* observium@observium.org *Subject:* [Observium] Polling Rate on a 3750 Stack
Is anyone else having huge response times in their polling of a Cisco 3750 stack?
I?ve increased the number of pollers I have to something close to 17, but my 3750s seem to take forever to respond (anywhere between 200 ? 999.99+ seconds). It may be an issue on my 3750s, but my polling intervals on my devices is several hours.
I have 2950s and 6509s that don?t seem to have any issue responding in a short amount of time (<20 seconds).
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org
iQEcBAEBAgAGBQJRcBtYAAoJEFmgPOBxQDtBMgQH/3Eck2xPLKdGkL7zHNVMYZEa gEFATDa98D9LGEjPNi3nv8h8rFAlwkaP6JYUyXh8YgTZhMn4bYkTWC1Xnv6SUyeD E9NL4Vf/tF9qZIqR6+/y15P4SkSbO5rM56QZltIbdSvp8yQqZUWfiNsUNb598zbX HGtuTdoQ3u+eb4Zfs+cZpp4DDbnP9mCBxKWZ2p1hvVkjelH+w09UX7Undx3Zc5Vn lh/4hlRWLrzh6N3FjbVT2xmQ2l9KcsafHD7lmsUxd2leNrxB6CeKW8iynZoErJoS TxIeFAkMqknxMd90b+H1G9TtCz1EqBg3/I1k8I4CRE6LMe/BN+oP4u9Wu2fGyCU= =+MPv -----END PGP SIGNATURE----- _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
I've used the same 3750 stack, and altered it from SNMPv3 authPriv, to authNoPriv, to v1 Community, and it doesn't appear to be different (minor differences in seconds for the entire polling time). I've checked through several of my 3750s, and of course I have versions all over the place, so I can't track it down to just one specific IOS version.
However, watching the debugging window, all of the 'buffering' moments occur when it calls the command to connect to the switch. This further emphasizes that I believe it's an issue with the switch, because it will execute the command, pause, receive the input, execute the next command, pause, receive the input, and then generate the rrd update statement, and repeat. Every time it pauses on waiting on the switch, so, I'm leaning toward calling Cisco and saying 'wat'.
There are a few other devices that are having this same issue that aren't 3750s, but I figure I'll contact Cisco with the issue and see what they have to say, and if it's anything helpful or relevant I'll forward it on back here.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Alex Pressé Sent: Thursday, April 18, 2013 1:28 PM To: Observium Network Observation System Subject: Re: [Observium] Polling Rate on a 3750 Stack
I had this same issue with one of my ISR 891 routers. Polling the vlan interfaces took ages compared to all the other 891s. This one slow device had a different IOS revision. Updated to match the "faster" routers and the issue went away.
15.0(1)M4 was the slow IOS, 15.1(4)M6 is much faster.
On Thu, Apr 18, 2013 at 10:47 AM, Michael Sweikata sweikatam1@nku.edu wrote:
That is a very helpful solution. Running the debug command on a 3750 stack, it didn't seem to hang on one specific command, it just seemed to run very slow on gathering information per port. I'm inclined to say that it's a problem on the 3750, we're using SNMPv3 with authPriv.
I'll throw TAC the question and see if they know why return information from the snmpbulkwalk is taking forever. I'm just trying to eliminate the server or the local MySQL DB from being the issue.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Michael Robbert Sent: Thursday, April 18, 2013 12:12 PM To: observium@observium.org Subject: Re: [Observium] Polling Rate on a 3750 Stack
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Have you run the poller in debug mode to see what commands it is hanging on? I just tested our largest 3750 stack (796 ports, 8 sensors) and it finished in 37 seconds one time and 96 seconds another time. It is possible that the longer time conflicted with a production poller or that something else was happening on the switch. If you can find one command that is causing the hang up you can open a TAC case with Cisco to find out why the switch is slow.
Mike
On 4/18/13 10:07 AM, Donald T. Currie wrote:
Yes we have the same issue on some of our Cisco devices, we pulled them out of the list for now. I would like to know how to fix this issue.
signature
*From:*observium-bounces@observium.org [mailto:observium-bounces@observium.org] *On Behalf Of *Michael Sweikata *Sent:* Thursday, April 18, 2013 11:59 AM *To:* observium@observium.org *Subject:* [Observium] Polling Rate on a 3750 Stack
Is anyone else having huge response times in their polling of a Cisco 3750 stack?
I?ve increased the number of pollers I have to something close to 17, but my 3750s seem to take forever to respond (anywhere between 200 ? 999.99+ seconds). It may be an issue on my 3750s, but my polling intervals on my devices is several hours.
I have 2950s and 6509s that don?t seem to have any issue responding in a short amount of time (<20 seconds).
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org
iQEcBAEBAgAGBQJRcBtYAAoJEFmgPOBxQDtBMgQH/3Eck2xPLKdGkL7zHNVMYZEa gEFATDa98D9LGEjPNi3nv8h8rFAlwkaP6JYUyXh8YgTZhMn4bYkTWC1Xnv6SUyeD E9NL4Vf/tF9qZIqR6+/y15P4SkSbO5rM56QZltIbdSvp8yQqZUWfiNsUNb598zbX HGtuTdoQ3u+eb4Zfs+cZpp4DDbnP9mCBxKWZ2p1hvVkjelH+w09UX7Undx3Zc5Vn lh/4hlRWLrzh6N3FjbVT2xmQ2l9KcsafHD7lmsUxd2leNrxB6CeKW8iynZoErJoS TxIeFAkMqknxMd90b+H1G9TtCz1EqBg3/I1k8I4CRE6LMe/BN+oP4u9Wu2fGyCU= =+MPv -----END PGP SIGNATURE----- _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Alex Presse "How much net work could a network work if a network could net work?" _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Sounds very much like it's going down the stack members collecting data.
Not much you can disable if the issue is in IF-MIB!
adam.
On 2013-04-18 18:47, Michael Sweikata wrote:
I've used the same 3750 stack, and altered it from SNMPv3 authPriv, to authNoPriv, to v1 Community, and it doesn't appear to be different (minor differences in seconds for the entire polling time). I've checked through several of my 3750s, and of course I have versions all over the place, so I can't track it down to just one specific IOS version.
However, watching the debugging window, all of the 'buffering' moments occur when it calls the command to connect to the switch. This further emphasizes that I believe it's an issue with the switch, because it will execute the command, pause, receive the input, execute the next command, pause, receive the input, and then generate the rrd update statement, and repeat. Every time it pauses on waiting on the switch, so, I'm leaning toward calling Cisco and saying 'wat'.
There are a few other devices that are having this same issue that aren't 3750s, but I figure I'll contact Cisco with the issue and see what they have to say, and if it's anything helpful or relevant I'll forward it on back here.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Alex Pressé Sent: Thursday, April 18, 2013 1:28 PM To: Observium Network Observation System Subject: Re: [Observium] Polling Rate on a 3750 Stack
I had this same issue with one of my ISR 891 routers. Polling the vlan interfaces took ages compared to all the other 891s. This one slow device had a different IOS revision. Updated to match the "faster" routers and the issue went away.
15.0(1)M4 was the slow IOS, 15.1(4)M6 is much faster.
On Thu, Apr 18, 2013 at 10:47 AM, Michael Sweikata sweikatam1@nku.edu wrote: That is a very helpful solution. Running the debug command on a 3750 stack, it didn't seem to hang on one specific command, it just seemed to run very slow on gathering information per port. I'm inclined to say that it's a problem on the 3750, we're using SNMPv3 with authPriv.
I'll throw TAC the question and see if they know why return information from the snmpbulkwalk is taking forever. I'm just trying to eliminate the server or the local MySQL DB from being the issue.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Michael Robbert Sent: Thursday, April 18, 2013 12:12 PM To: observium@observium.org Subject: Re: [Observium] Polling Rate on a 3750 Stack
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Have you run the poller in debug mode to see what commands it is hanging on? I just tested our largest 3750 stack (796 ports, 8 sensors) and it finished in 37 seconds one time and 96 seconds another time. It is possible that the longer time conflicted with a production poller or that something else was happening on the switch. If you can find one command that is causing the hang up you can open a TAC case with Cisco to find out why the switch is slow.
Mike
On 4/18/13 10:07 AM, Donald T. Currie wrote: Yes we have the same issue on some of our Cisco devices, we pulled them out of the list for now. I would like to know how to fix this issue.
signature
*From:*observium-bounces@observium.org [mailto:observium-bounces@observium.org] *On Behalf Of *Michael Sweikata *Sent:* Thursday, April 18, 2013 11:59 AM *To:* observium@observium.org *Subject:* [Observium] Polling Rate on a 3750 Stack
Is anyone else having huge response times in their polling of a Cisco 3750 stack?
I?ve increased the number of pollers I have to something close to 17, but my 3750s seem to take forever to respond (anywhere between 200 ? 999.99+ seconds). It may be an issue on my 3750s, but my polling intervals on my devices is several hours.
I have 2950s and 6509s that don?t seem to have any issue responding in a short amount of time (<20 seconds).
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org
iQEcBAEBAgAGBQJRcBtYAAoJEFmgPOBxQDtBMgQH/3Eck2xPLKdGkL7zHNVMYZEa gEFATDa98D9LGEjPNi3nv8h8rFAlwkaP6JYUyXh8YgTZhMn4bYkTWC1Xnv6SUyeD E9NL4Vf/tF9qZIqR6+/y15P4SkSbO5rM56QZltIbdSvp8yQqZUWfiNsUNb598zbX HGtuTdoQ3u+eb4Zfs+cZpp4DDbnP9mCBxKWZ2p1hvVkjelH+w09UX7Undx3Zc5Vn lh/4hlRWLrzh6N3FjbVT2xmQ2l9KcsafHD7lmsUxd2leNrxB6CeKW8iynZoErJoS TxIeFAkMqknxMd90b+H1G9TtCz1EqBg3/I1k8I4CRE6LMe/BN+oP4u9Wu2fGyCU= =+MPv -----END PGP SIGNATURE----- _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Alex Presse "How much net work could a network work if a network could net work?" _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
I think you're right. I worked with Michael Robbert a little more, and based on the information you just provided, I do believe it's the 3750 stack. I stuck with one stack, running the 12.2(44)SE5 OS, 7 switches with 357 ports. The first iteration took 36 minutes for it to poll. I then started customizing what modules the polling uses (taking out things I know it doesn't need: ospf, aruba, applications, etc.). It so far has appeared to speed up, but the wait time is most definitely coming from the responses from the switch.
I know that I can remove the pollers down to as minimal as possible, but any advice other than that?
I think this is the first time I've been annoyed by my 3750s, normally I'd praise them.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: Thursday, April 18, 2013 1:31 PM To: Observium Network Observation System Subject: Re: [Observium] Polling Rate on a 3750 Stack
Sounds very much like it's going down the stack members collecting data.
Not much you can disable if the issue is in IF-MIB!
adam.
On 2013-04-18 18:47, Michael Sweikata wrote:
I've used the same 3750 stack, and altered it from SNMPv3 authPriv, to authNoPriv, to v1 Community, and it doesn't appear to be different (minor differences in seconds for the entire polling time). I've checked through several of my 3750s, and of course I have versions all over the place, so I can't track it down to just one specific IOS version.
However, watching the debugging window, all of the 'buffering' moments occur when it calls the command to connect to the switch. This further emphasizes that I believe it's an issue with the switch, because it will execute the command, pause, receive the input, execute the next command, pause, receive the input, and then generate the rrd update statement, and repeat. Every time it pauses on waiting on the switch, so, I'm leaning toward calling Cisco and saying 'wat'.
There are a few other devices that are having this same issue that aren't 3750s, but I figure I'll contact Cisco with the issue and see what they have to say, and if it's anything helpful or relevant I'll forward it on back here.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Alex Pressé Sent: Thursday, April 18, 2013 1:28 PM To: Observium Network Observation System Subject: Re: [Observium] Polling Rate on a 3750 Stack
I had this same issue with one of my ISR 891 routers. Polling the vlan interfaces took ages compared to all the other 891s. This one slow device had a different IOS revision. Updated to match the "faster" routers and the issue went away.
15.0(1)M4 was the slow IOS, 15.1(4)M6 is much faster.
On Thu, Apr 18, 2013 at 10:47 AM, Michael Sweikata sweikatam1@nku.edu wrote: That is a very helpful solution. Running the debug command on a 3750 stack, it didn't seem to hang on one specific command, it just seemed to run very slow on gathering information per port. I'm inclined to say that it's a problem on the 3750, we're using SNMPv3 with authPriv.
I'll throw TAC the question and see if they know why return information from the snmpbulkwalk is taking forever. I'm just trying to eliminate the server or the local MySQL DB from being the issue.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Michael Robbert Sent: Thursday, April 18, 2013 12:12 PM To: observium@observium.org Subject: Re: [Observium] Polling Rate on a 3750 Stack
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Have you run the poller in debug mode to see what commands it is hanging on? I just tested our largest 3750 stack (796 ports, 8 sensors) and it finished in 37 seconds one time and 96 seconds another time. It is possible that the longer time conflicted with a production poller or that something else was happening on the switch. If you can find one command that is causing the hang up you can open a TAC case with Cisco to find out why the switch is slow.
Mike
On 4/18/13 10:07 AM, Donald T. Currie wrote: Yes we have the same issue on some of our Cisco devices, we pulled them out of the list for now. I would like to know how to fix this issue.
signature
*From:*observium-bounces@observium.org [mailto:observium-bounces@observium.org] *On Behalf Of *Michael Sweikata *Sent:* Thursday, April 18, 2013 11:59 AM *To:* observium@observium.org *Subject:* [Observium] Polling Rate on a 3750 Stack
Is anyone else having huge response times in their polling of a Cisco 3750 stack?
I?ve increased the number of pollers I have to something close to 17, but my 3750s seem to take forever to respond (anywhere between 200 ? 999.99+ seconds). It may be an issue on my 3750s, but my polling intervals on my devices is several hours.
I have 2950s and 6509s that don?t seem to have any issue responding in a short amount of time (<20 seconds).
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org
iQEcBAEBAgAGBQJRcBtYAAoJEFmgPOBxQDtBMgQH/3Eck2xPLKdGkL7zHNVMYZEa gEFATDa98D9LGEjPNi3nv8h8rFAlwkaP6JYUyXh8YgTZhMn4bYkTWC1Xnv6SUyeD E9NL4Vf/tF9qZIqR6+/y15P4SkSbO5rM56QZltIbdSvp8yQqZUWfiNsUNb598zbX HGtuTdoQ3u+eb4Zfs+cZpp4DDbnP9mCBxKWZ2p1hvVkjelH+w09UX7Undx3Zc5Vn lh/4hlRWLrzh6N3FjbVT2xmQ2l9KcsafHD7lmsUxd2leNrxB6CeKW8iynZoErJoS TxIeFAkMqknxMd90b+H1G9TtCz1EqBg3/I1k8I4CRE6LMe/BN+oP4u9Wu2fGyCU= =+MPv -----END PGP SIGNATURE----- _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Alex Presse "How much net work could a network work if a network could net work?" _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
It's definitely hanging up on the 'ports' module on the3750 stack. I removed writing to disk from the debugging option, removed every module possible and started testing one by one. Once it hit the 'ports' module, that’s where it takes forever to load.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Michael Sweikata Sent: Thursday, April 18, 2013 3:17 PM To: Observium Network Observation System Subject: Re: [Observium] Polling Rate on a 3750 Stack
I think you're right. I worked with Michael Robbert a little more, and based on the information you just provided, I do believe it's the 3750 stack. I stuck with one stack, running the 12.2(44)SE5 OS, 7 switches with 357 ports. The first iteration took 36 minutes for it to poll. I then started customizing what modules the polling uses (taking out things I know it doesn't need: ospf, aruba, applications, etc.). It so far has appeared to speed up, but the wait time is most definitely coming from the responses from the switch.
I know that I can remove the pollers down to as minimal as possible, but any advice other than that?
I think this is the first time I've been annoyed by my 3750s, normally I'd praise them.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: Thursday, April 18, 2013 1:31 PM To: Observium Network Observation System Subject: Re: [Observium] Polling Rate on a 3750 Stack
Sounds very much like it's going down the stack members collecting data.
Not much you can disable if the issue is in IF-MIB!
adam.
On 2013-04-18 18:47, Michael Sweikata wrote:
I've used the same 3750 stack, and altered it from SNMPv3 authPriv, to authNoPriv, to v1 Community, and it doesn't appear to be different (minor differences in seconds for the entire polling time). I've checked through several of my 3750s, and of course I have versions all over the place, so I can't track it down to just one specific IOS version.
However, watching the debugging window, all of the 'buffering' moments occur when it calls the command to connect to the switch. This further emphasizes that I believe it's an issue with the switch, because it will execute the command, pause, receive the input, execute the next command, pause, receive the input, and then generate the rrd update statement, and repeat. Every time it pauses on waiting on the switch, so, I'm leaning toward calling Cisco and saying 'wat'.
There are a few other devices that are having this same issue that aren't 3750s, but I figure I'll contact Cisco with the issue and see what they have to say, and if it's anything helpful or relevant I'll forward it on back here.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Alex Pressé Sent: Thursday, April 18, 2013 1:28 PM To: Observium Network Observation System Subject: Re: [Observium] Polling Rate on a 3750 Stack
I had this same issue with one of my ISR 891 routers. Polling the vlan interfaces took ages compared to all the other 891s. This one slow device had a different IOS revision. Updated to match the "faster" routers and the issue went away.
15.0(1)M4 was the slow IOS, 15.1(4)M6 is much faster.
On Thu, Apr 18, 2013 at 10:47 AM, Michael Sweikata sweikatam1@nku.edu wrote: That is a very helpful solution. Running the debug command on a 3750 stack, it didn't seem to hang on one specific command, it just seemed to run very slow on gathering information per port. I'm inclined to say that it's a problem on the 3750, we're using SNMPv3 with authPriv.
I'll throw TAC the question and see if they know why return information from the snmpbulkwalk is taking forever. I'm just trying to eliminate the server or the local MySQL DB from being the issue.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Michael Robbert Sent: Thursday, April 18, 2013 12:12 PM To: observium@observium.org Subject: Re: [Observium] Polling Rate on a 3750 Stack
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Have you run the poller in debug mode to see what commands it is hanging on? I just tested our largest 3750 stack (796 ports, 8 sensors) and it finished in 37 seconds one time and 96 seconds another time. It is possible that the longer time conflicted with a production poller or that something else was happening on the switch. If you can find one command that is causing the hang up you can open a TAC case with Cisco to find out why the switch is slow.
Mike
On 4/18/13 10:07 AM, Donald T. Currie wrote: Yes we have the same issue on some of our Cisco devices, we pulled them out of the list for now. I would like to know how to fix this issue.
signature
*From:*observium-bounces@observium.org [mailto:observium-bounces@observium.org] *On Behalf Of *Michael Sweikata *Sent:* Thursday, April 18, 2013 11:59 AM *To:* observium@observium.org *Subject:* [Observium] Polling Rate on a 3750 Stack
Is anyone else having huge response times in their polling of a Cisco 3750 stack?
I?ve increased the number of pollers I have to something close to 17, but my 3750s seem to take forever to respond (anywhere between 200 ? 999.99+ seconds). It may be an issue on my 3750s, but my polling intervals on my devices is several hours.
I have 2950s and 6509s that don?t seem to have any issue responding in a short amount of time (<20 seconds).
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org
iQEcBAEBAgAGBQJRcBtYAAoJEFmgPOBxQDtBMgQH/3Eck2xPLKdGkL7zHNVMYZEa gEFATDa98D9LGEjPNi3nv8h8rFAlwkaP6JYUyXh8YgTZhMn4bYkTWC1Xnv6SUyeD E9NL4Vf/tF9qZIqR6+/y15P4SkSbO5rM56QZltIbdSvp8yQqZUWfiNsUNb598zbX HGtuTdoQ3u+eb4Zfs+cZpp4DDbnP9mCBxKWZ2p1hvVkjelH+w09UX7Undx3Zc5Vn lh/4hlRWLrzh6N3FjbVT2xmQ2l9KcsafHD7lmsUxd2leNrxB6CeKW8iynZoErJoS TxIeFAkMqknxMd90b+H1G9TtCz1EqBg3/I1k8I4CRE6LMe/BN+oP4u9Wu2fGyCU= =+MPv -----END PGP SIGNATURE----- _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Alex Presse "How much net work could a network work if a network could net work?" _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Also 15.2 releases have issues with ipsec mib it will increase incrementally number of ipsec tunnels so every poll will be larger and larged. When mine hit about 1500 tunnels snmp polling about 12mb of data.
On 18.04.2013 21:28, Alex Pressé wrote:
I had this same issue with one of my ISR 891 routers. Polling the vlan interfaces took ages compared to all the other 891s. This one slow device had a different IOS revision. Updated to match the "faster" routers and the issue went away.
15.0(1)M4 was the slow IOS, 15.1(4)M6 is much faster.
On Thu, Apr 18, 2013 at 10:47 AM, Michael Sweikata sweikatam1@nku.edu wrote:
That is a very helpful solution. Running the debug command on a 3750 stack, it didn't seem to hang on one specific command, it just seemed to run very slow on gathering information per port. I'm inclined to say that it's a problem on the 3750, we're using SNMPv3 with authPriv.
I'll throw TAC the question and see if they know why return information from the snmpbulkwalk is taking forever. I'm just trying to eliminate the server or the local MySQL DB from being the issue.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Michael Robbert Sent: Thursday, April 18, 2013 12:12 PM To: observium@observium.org Subject: Re: [Observium] Polling Rate on a 3750 Stack
Have you run the poller in debug mode to see what commands it is hanging on? I just tested our largest 3750 stack (796 ports, 8 sensors) and it finished in 37 seconds one time and 96 seconds another time. It is possible that the longer time conflicted with a production poller or that something else was happening on the switch. If you can find one command that is causing the hang up you can open a TAC case with Cisco to find out why the switch is slow.
Mike
On 4/18/13 10:07 AM, Donald T. Currie wrote:
Yes we have the same issue on some of our Cisco devices, we pulled them out of the list for now. I would like to know how to fix this issue.
signature
*From:*observium-bounces@observium.org [mailto:observium-bounces@observium.org] *On Behalf Of *Michael Sweikata *Sent:* Thursday, April 18, 2013 11:59 AM *To:* observium@observium.org *Subject:* [Observium] Polling Rate on a 3750 Stack
Is anyone else having huge response times in their polling of a Cisco 3750 stack?
I?ve increased the number of pollers I have to something close to 17, but my 3750s seem to take forever to respond (anywhere between 200 ? 999.99+ seconds). It may be an issue on my 3750s, but my polling intervals on my devices is several hours.
I have 2950s and 6509s that don?t seem to have any issue responding in a short amount of time (<20 seconds).
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
That's good to know. So far, the majority of my 3750 equipment is running on the 12.* code ranges.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Nikolay Shopik Sent: Thursday, April 18, 2013 2:01 PM To: Observium Network Observation System Subject: Re: [Observium] Polling Rate on a 3750 Stack
Also 15.2 releases have issues with ipsec mib it will increase incrementally number of ipsec tunnels so every poll will be larger and larged. When mine hit about 1500 tunnels snmp polling about 12mb of data.
On 18.04.2013 21:28, Alex Pressé wrote:
I had this same issue with one of my ISR 891 routers. Polling the vlan interfaces took ages compared to all the other 891s. This one slow device had a different IOS revision. Updated to match the "faster" routers and the issue went away.
15.0(1)M4 was the slow IOS, 15.1(4)M6 is much faster.
On Thu, Apr 18, 2013 at 10:47 AM, Michael Sweikata sweikatam1@nku.edu wrote:
That is a very helpful solution. Running the debug command on a 3750 stack, it didn't seem to hang on one specific command, it just seemed to run very slow on gathering information per port. I'm inclined to say that it's a problem on the 3750, we're using SNMPv3 with authPriv.
I'll throw TAC the question and see if they know why return information from the snmpbulkwalk is taking forever. I'm just trying to eliminate the server or the local MySQL DB from being the issue.
-----Original Message----- From: observium-bounces@observium.org [mailto:observium-bounces@observium.org] On Behalf Of Michael Robbert Sent: Thursday, April 18, 2013 12:12 PM To: observium@observium.org Subject: Re: [Observium] Polling Rate on a 3750 Stack
Have you run the poller in debug mode to see what commands it is hanging on? I just tested our largest 3750 stack (796 ports, 8 sensors) and it finished in 37 seconds one time and 96 seconds another time. It is possible that the longer time conflicted with a production poller or that something else was happening on the switch. If you can find one command that is causing the hang up you can open a TAC case with Cisco to find out why the switch is slow.
Mike
On 4/18/13 10:07 AM, Donald T. Currie wrote:
Yes we have the same issue on some of our Cisco devices, we pulled them out of the list for now. I would like to know how to fix this issue.
signature
*From:*observium-bounces@observium.org [mailto:observium-bounces@observium.org] *On Behalf Of *Michael Sweikata *Sent:* Thursday, April 18, 2013 11:59 AM *To:* observium@observium.org *Subject:* [Observium] Polling Rate on a 3750 Stack
Is anyone else having huge response times in their polling of a Cisco 3750 stack?
I?ve increased the number of pollers I have to something close to 17, but my 3750s seem to take forever to respond (anywhere between 200 ? 999.99+ seconds). It may be an issue on my 3750s, but my polling intervals on my devices is several hours.
I have 2950s and 6509s that don?t seem to have any issue responding in a short amount of time (<20 seconds).
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
I recall something about this in the past couple of months.
3750 stacks are pretty painful to poll because of their architecture, the control plane has to sometimes interrogate the stack members to get statistics, which can be slow.
You can try running the poller manually and trying to work out which module it is that is slowing down the poll. Likely it is a single module hitting a MIB or OID which isn't being properly cached by IOS and it's being forced to go down the stack more often than it should.
You can then disable the affected poller/discovery modules.
adam.
On 2013-04-18 16:58, Michael Sweikata wrote:
Is anyone else having huge response times in their polling of a Cisco 3750 stack?
I've increased the number of pollers I have to something close to 17, but my 3750s seem to take forever to respond (anywhere between 200 - 999.99+ seconds). It may be an issue on my 3750s, but my polling intervals on my devices is several hours.
I have 2950s and 6509s that don't seem to have any issue responding in a short amount of time (<20 seconds). _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (6)
-
Adam Armstrong -
Alex Pressé -
Donald T. Currie -
Michael Robbert -
Michael Sweikata -
Nikolay Shopik