An interesting issue with Cisco 2960Cs...
Hi all,
This is just on the order of a 'watch out' notice.
We're currently on 0.15.1.6206, but had the same issue with earlier versions and with the CE.
We have 5 Cisco 2960Cs that we monitor, and all were taking horrible amounts of time to poll. Always at least 200 seconds, and often over 300.
After some experiment, we discovered that they apparently don't care to talk about CEF (Cisco Express Forwarding). If Observium asks them about it, the CPU usage climbs above 60% and stays there for quite a while.
If I go to the device --> edit --> Modules and disable cicso-cef, everything is fine. The CPU drops under 5%, and the poll time is 5 seconds or less.
I'm pretty sure we're not losing any useful info, since I don't believe the 2960Cs support CEF in the first place.
As I said, this is just a heads up.
Observium is a great tool.
Thanks!
I had the same issue with the 2960S switches, polling the cisco-cef and fdb-table modules would extend our polling time out to exorbitant amounts of time for those devices. Disabling those two modules, my polling time is now around 7-8 seconds for each device.
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Richard Kuhns Sent: Wednesday, January 28, 2015 2:52 PM To: observium@observium.org Subject: [Observium] An interesting issue with Cisco 2960Cs...
Hi all,
This is just on the order of a 'watch out' notice.
We're currently on 0.15.1.6206, but had the same issue with earlier versions and with the CE.
We have 5 Cisco 2960Cs that we monitor, and all were taking horrible amounts of time to poll. Always at least 200 seconds, and often over 300.
After some experiment, we discovered that they apparently don't care to talk about CEF (Cisco Express Forwarding). If Observium asks them about it, the CPU usage climbs above 60% and stays there for quite a while.
If I go to the device --> edit --> Modules and disable cicso-cef, everything is fine. The CPU drops under 5%, and the poll time is 5 seconds or less.
I'm pretty sure we're not losing any useful info, since I don't believe the 2960Cs support CEF in the first place.
As I said, this is just a heads up.
Observium is a great tool.
Thanks!
Hi Richard,
plese send me debug poller for this device with enabled module cef: ./poller.php -d -m cisco-cef -h <cisco_2960>
On Wed, Jan 28, 2015 at 11:52 PM, Richard Kuhns rjk@wintek.com wrote:
Hi all,
This is just on the order of a 'watch out' notice.
We're currently on 0.15.1.6206, but had the same issue with earlier versions and with the CE.
We have 5 Cisco 2960Cs that we monitor, and all were taking horrible amounts of time to poll. Always at least 200 seconds, and often over 300.
After some experiment, we discovered that they apparently don't care to talk about CEF (Cisco Express Forwarding). If Observium asks them about it, the CPU usage climbs above 60% and stays there for quite a while.
If I go to the device --> edit --> Modules and disable cicso-cef, everything is fine. The CPU drops under 5%, and the poll time is 5 seconds or less.
I'm pretty sure we're not losing any useful info, since I don't believe the 2960Cs support CEF in the first place.
As I said, this is just a heads up.
Observium is a great tool.
Thanks!
Richard Kuhns rjk@wintek.com Main Number: 765-742-8428 Wintek Corporation Direct: 765-269-8541 427 N 6th Street Internet Support: 765-269-8503 Lafayette, IN 47901-2211 Consulting: 765-269-8504 _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi,
Will do, but I'm afraid it won't be until tomorrow.
- Richard
On 01/28/15 16:17, Mike Stupalov wrote:
Hi Richard,
plese send me debug poller for this device with enabled module cef: ./poller.php -d -m cisco-cef -h <cisco_2960>
On Wed, Jan 28, 2015 at 11:52 PM, Richard Kuhns <rjk@wintek.com mailto:rjk@wintek.com> wrote:
Hi all, This is just on the order of a 'watch out' notice. We're currently on 0.15.1.6206, but had the same issue with earlier versions and with the CE. We have 5 Cisco 2960Cs that we monitor, and all were taking horrible amounts of time to poll. Always at least 200 seconds, and often over 300. After some experiment, we discovered that they apparently don't care to talk about CEF (Cisco Express Forwarding). If Observium asks them about it, the CPU usage climbs above 60% and stays there for quite a while. If I go to the device --> edit --> Modules and disable cicso-cef, everything is fine. The CPU drops under 5%, and the poll time is 5 seconds or less. I'm pretty sure we're not losing any useful info, since I don't believe the 2960Cs support CEF in the first place. As I said, this is just a heads up. Observium is a great tool. Thanks! -- Richard Kuhns <rjk@wintek.com <mailto:rjk@wintek.com>> Main Number: 765-742-8428 Wintek Corporation Direct: 765-269-8541 427 N 6th Street Internet Support: 765-269-8503 Lafayette, IN 47901-2211 Consulting: 765-269-8504 _______________________________________________ observium mailing list observium@observium.org <mailto:observium@observium.org> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium-- Mike Stupalov http://observium.org/
Attached! If the list server doesn't like it, I'll resend with it inline.
On 01/28/15 16:17, Mike Stupalov wrote:
Hi Richard,
plese send me debug poller for this device with enabled module cef: ./poller.php -d -m cisco-cef -h <cisco_2960>
On Wed, Jan 28, 2015 at 11:52 PM, Richard Kuhns <rjk@wintek.com mailto:rjk@wintek.com> wrote:
Hi all, This is just on the order of a 'watch out' notice. We're currently on 0.15.1.6206, but had the same issue with earlier versions and with the CE. We have 5 Cisco 2960Cs that we monitor, and all were taking horrible amounts of time to poll. Always at least 200 seconds, and often over 300. After some experiment, we discovered that they apparently don't care to talk about CEF (Cisco Express Forwarding). If Observium asks them about it, the CPU usage climbs above 60% and stays there for quite a while. If I go to the device --> edit --> Modules and disable cicso-cef, everything is fine. The CPU drops under 5%, and the poll time is 5 seconds or less. I'm pretty sure we're not losing any useful info, since I don't believe the 2960Cs support CEF in the first place. As I said, this is just a heads up. Observium is a great tool. Thanks! -- Richard Kuhns <rjk@wintek.com <mailto:rjk@wintek.com>> Main Number: 765-742-8428 Wintek Corporation Direct: 765-269-8541 427 N 6th Street Internet Support: 765-269-8503 Lafayette, IN 47901-2211 Consulting: 765-269-8504 _______________________________________________ observium mailing list observium@observium.org <mailto:observium@observium.org> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium-- Mike Stupalov http://observium.org/
Hi Richard
You're right, 2960C's are L2 only; and my understanding is the same in that they shouldn't be doing CEF at all.
That said they're a similar hardware flavour to 2960S/3560X/3750X hardware, which can have L3 features enabled, so the software release in use probably has CEF code riddled right the way thru it.
What happens when you attempt to run CEF related commands on the CLI, e.g. - show cef background - show platform ip unicast cef-idb
e.g. are they available, do they return anything, does the CPU spike with the CLI commands too?
The other obvious questions: 1) Which IOS version are you running? 2) Is it the latest stable/recommended release?
-Colin
On 29 January 2015 at 22:47, Richard Kuhns rjk@wintek.com wrote:
Attached! If the list server doesn't like it, I'll resend with it inline.
On 01/28/15 16:17, Mike Stupalov wrote:
Hi Richard,
plese send me debug poller for this device with enabled module cef: ./poller.php -d -m cisco-cef -h <cisco_2960>
On Wed, Jan 28, 2015 at 11:52 PM, Richard Kuhns <rjk@wintek.com mailto:rjk@wintek.com> wrote:
Hi all, This is just on the order of a 'watch out' notice. We're currently on 0.15.1.6206, but had the same issue with earlier versions and with the CE. We have 5 Cisco 2960Cs that we monitor, and all were taking horrible amounts of time to poll. Always at least 200 seconds, and often over 300. After some experiment, we discovered that they apparently don't careto
talk about CEF (Cisco Express Forwarding). If Observium asks themabout
it, the CPU usage climbs above 60% and stays there for quite a while. If I go to the device --> edit --> Modules and disable cicso-cef, everything is fine. The CPU drops under 5%, and the poll time is 5 seconds or less. I'm pretty sure we're not losing any useful info, since I don'tbelieve
the 2960Cs support CEF in the first place. As I said, this is just a heads up. Observium is a great tool. Thanks! -- Richard Kuhns <rjk@wintek.com <mailto:rjk@wintek.com>> Main Number: 765-742-8428 Wintek Corporation Direct: 765-269-8541 427 N 6th Street Internet Support: 765-269-8503 Lafayette, IN 47901-2211 Consulting: 765-269-8504 _______________________________________________ observium mailing list observium@observium.org <mailto:observium@observium.org> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium-- Mike Stupalov http://observium.org/
-- Richard Kuhns rjk@wintek.com Main Number: 765-742-8428 Wintek Corporation Direct: 765-269-8541 427 N 6th Street Internet Support: 765-269-8503 Lafayette, IN 47901-2211 Consulting: 765-269-8504
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Colin,
On 01/29/15 08:19, Colin Stubbs wrote:
Hi Richard
You're right, 2960C's are L2 only; and my understanding is the same in that they shouldn't be doing CEF at all.
That said they're a similar hardware flavour to 2960S/3560X/3750X hardware, which can have L3 features enabled, so the software release in use probably has CEF code riddled right the way thru it.
What happens when you attempt to run CEF related commands on the CLI, e.g.
- show cef background
This command isn't available.
- show platform ip unicast cef-idb
This command returns immediately with no output.
e.g. are they available, do they return anything, does the CPU spike with the CLI commands too?
Nope, no CPU spike.
The other obvious questions:
- Which IOS version are you running?
We've now tested on 12.2(55)EX, 15.0(2)SE7, and 15.2(1)E3. All have the same problem. I can use snmpbulkwalk to query CISCO-CEF-MIB::cefFIBSummaryEntry and the SNMP ENGINE process CPU use jumps to over 80%, and stays there for 20 to 50 seconds.
- Is it the latest stable/recommended release?
15.0(2)SE7 is.
-Colin
Hopefully this info will be useful to other Observium users. It's definitely a Cisco problem, not an Observium problem. As long as we remember to disable cisco-cef when installing a 2960C everything is fine.
- Richard
Try latest revision (r6218). For my test device (WS-C2960C-8PC-L) polling time reduced from 124.78s -> to 17.52s (without disabling any module) :)
On Thu, Jan 29, 2015 at 7:14 PM, Richard Kuhns rjk@wintek.com wrote:
Hi Colin,
On 01/29/15 08:19, Colin Stubbs wrote:
Hi Richard
You're right, 2960C's are L2 only; and my understanding is the same in that they shouldn't be doing CEF at all.
That said they're a similar hardware flavour to 2960S/3560X/3750X hardware, which can have L3 features enabled, so the software release in use probably has CEF code riddled right the way thru it.
What happens when you attempt to run CEF related commands on the CLI,
e.g.
- show cef background
This command isn't available.
- show platform ip unicast cef-idb
This command returns immediately with no output.
e.g. are they available, do they return anything, does the CPU spike with the CLI commands too?
Nope, no CPU spike.
The other obvious questions:
- Which IOS version are you running?
We've now tested on 12.2(55)EX, 15.0(2)SE7, and 15.2(1)E3. All have the same problem. I can use snmpbulkwalk to query CISCO-CEF-MIB::cefFIBSummaryEntry and the SNMP ENGINE process CPU use jumps to over 80%, and stays there for 20 to 50 seconds.
- Is it the latest stable/recommended release?
15.0(2)SE7 is.
-Colin
Hopefully this info will be useful to other Observium users. It's definitely a Cisco problem, not an Observium problem. As long as we remember to disable cisco-cef when installing a 2960C everything is fine.
- Richard-- Richard Kuhns rjk@wintek.com Main Number: 765-742-8428 Wintek Corporation Direct: 765-269-8541 427 N 6th Street Internet Support: 765-269-8503 Lafayette, IN 47901-2211 Consulting: 765-269-8504 _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Mike,
disabling CEF is the worst thing to do on a cisco switch. The Problem is just type error in the polling script.
The polled OID name is _not_ CISCO-CEF-MIB::cefFIBSummaryEntry! It is CISCO-CEF-MIB::CefFIBSummaryEntry
The Mib lists the the description as :cefFIBSummaryEntry, but the value name is CISCO-CEF-MIB::CefFIBSummaryEntry
regards
Karsten Schwarz Plateau des Systèmes d'Information Infrastructure
Centre Hospitalier du Nord 120, avenue Salentiny L-9080 Ettelbruck
+352 8166-63418 karsten.schwarz@chdn.lu http://www.chdn.lu
From: Mike Stupalov mike@observium.org To: Observium Network Observation System observium@observium.org Date: 29.01.2015 20:39 Subject: Re: [Observium] An interesting issue with Cisco 2960Cs... Sent by: "observium" observium-bounces@observium.org
Try latest revision (r6218). For my test device (WS-C2960C-8PC-L) polling time reduced from 124.78s -> to 17.52s (without disabling any module) :)
On Thu, Jan 29, 2015 at 7:14 PM, Richard Kuhns rjk@wintek.com wrote: Hi Colin,
On 01/29/15 08:19, Colin Stubbs wrote:
Hi Richard
You're right, 2960C's are L2 only; and my understanding is the same in that they shouldn't be doing CEF at all.
That said they're a similar hardware flavour to 2960S/3560X/3750X hardware, which can have L3 features enabled, so the software release in use probably has CEF code riddled right the way thru it.
What happens when you attempt to run CEF related commands on the CLI,
e.g.
- show cef background
This command isn't available.
- show platform ip unicast cef-idb
This command returns immediately with no output.
e.g. are they available, do they return anything, does the CPU spike with the CLI commands too?
Nope, no CPU spike.
The other obvious questions:
- Which IOS version are you running?
We've now tested on 12.2(55)EX, 15.0(2)SE7, and 15.2(1)E3. All have the same problem. I can use snmpbulkwalk to query CISCO-CEF-MIB::cefFIBSummaryEntry and the SNMP ENGINE process CPU use jumps to over 80%, and stays there for 20 to 50 seconds.
- Is it the latest stable/recommended release?
15.0(2)SE7 is.
-Colin
Hopefully this info will be useful to other Observium users. It's definitely a Cisco problem, not an Observium problem. As long as we remember to disable cisco-cef when installing a 2960C everything is fine.
- Richard
-- Richard Kuhns rjk@wintek.com Main Number: 765-742-8428 Wintek Corporation Direct: 765-269-8541 427 N 6th Street Internet Support: 765-269-8503 Lafayette, IN 47901-2211 Consulting: 765-269-8504 _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi,
On 03.02.2015 11:41, Karsten Schwarz wrote:
Hi Mike,
disabling CEF is the worst thing to do on a cisco switch. The Problem is just type error in the polling script.
The polled OID name is _not_ CISCO-CEF-MIB::cefFIBSummaryEntry! It is CISCO-CEF-MIB::CefFIBSummaryEntry
The Mib lists the the description as :cefFIBSummaryEntry, but the value name is CISCO-CEF-MIB::CefFIBSummaryEntry
You are sure that know this better than me? ;)
CefFIBSummaryEntry is syntax description for cefFIBSummaryEntry.
regards
*Karsten Schwarz*/ Plateau des Systèmes d'Information Infrastructure/
Centre Hospitalier du Nord 120, avenue Salentiny L-9080 Ettelbruck
+352 8166-63418 karsten.schwarz@chdn.lu http://www.chdn.lu http://www.chdn.lu/
From: Mike Stupalov mike@observium.org To: Observium Network Observation System observium@observium.org Date: 29.01.2015 20:39 Subject: Re: [Observium] An interesting issue with Cisco 2960Cs... Sent by: "observium" observium-bounces@observium.org
Try latest revision (r6218). For my test device (WS-C2960C-8PC-L) polling time reduced from 124.78s -> to 17.52s (without disabling any module) :)
On Thu, Jan 29, 2015 at 7:14 PM, Richard Kuhns <_rjk@wintek.com_ mailto:rjk@wintek.com> wrote: Hi Colin,
On 01/29/15 08:19, Colin Stubbs wrote:
Hi Richard
You're right, 2960C's are L2 only; and my understanding is the same in that they shouldn't be doing CEF at all.
That said they're a similar hardware flavour to 2960S/3560X/3750X hardware, which can have L3 features enabled, so the software release in use probably has CEF code riddled right the way thru it.
What happens when you attempt to run CEF related commands on the
CLI, e.g.
- show cef background
This command isn't available.
- show platform ip unicast cef-idb
This command returns immediately with no output.
e.g. are they available, do they return anything, does the CPU spike with the CLI commands too?
Nope, no CPU spike.
The other obvious questions:
- Which IOS version are you running?
We've now tested on 12.2(55)EX, 15.0(2)SE7, and 15.2(1)E3. All have the same problem. I can use snmpbulkwalk to query CISCO-CEF-MIB::cefFIBSummaryEntry and the SNMP ENGINE process CPU use jumps to over 80%, and stays there for 20 to 50 seconds.
- Is it the latest stable/recommended release?
15.0(2)SE7 is.
-Colin
Hopefully this info will be useful to other Observium users. It's definitely a Cisco problem, not an Observium problem. As long as we remember to disable cisco-cef when installing a 2960C everything is fine.
- Richard-- Richard Kuhns <_rjk@wintek.com_ mailto:rjk@wintek.com> Main Number: 765-742-8428 Wintek Corporation Direct: 765-269-8541 427 N 6th Street Internet Support: 765-269-8503 Lafayette, IN 47901-2211 Consulting: 765-269-8504 _______________________________________________ observium mailing list_ __observium@observium.org_ mailto:observium@observium.org_ __http://postman.memetic.org/cgi-bin/mailman/listinfo/observium_
-- Mike Stupalov_ __http://observium.org/________________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
*Pensez à l'environnement avant d'imprimer ce message / Think of the environment before printing out this message*
File this one in the "idiot who knows enough to break something category"
CCNA people - take it!
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Karsten Schwarz Sent: Tuesday, February 03, 2015 3:42 AM To: mike@observium.org; Observium Network Observation System Subject: Re: [Observium] An interesting issue with Cisco 2960Cs...
Hi Mike,
disabling CEF is the worst thing to do on a cisco switch. The Problem is just type error in the polling script.
The polled OID name is _not_ CISCO-CEF-MIB::cefFIBSummaryEntry! It is CISCO-CEF-MIB::CefFIBSummaryEntry
The Mib lists the the description as :cefFIBSummaryEntry, but the value name is CISCO-CEF-MIB::CefFIBSummaryEntry
regards
Karsten Schwarz Plateau des Systèmes d'Information Infrastructure
Centre Hospitalier du Nord 120, avenue Salentiny L-9080 Ettelbruck
[cid:image002.png@01D03F68.560289E0]+352 8166-63418 [cid:image003.png@01D03F68.560289E0]karsten.schwarz@chdn.lumailto:karsten.schwarz@chdn.lu [cid:image004.png@01D03F68.560289E0]http://www.chdn.luhttp://www.chdn.lu/
From: Mike Stupalov <mike@observium.orgmailto:mike@observium.org> To: Observium Network Observation System <observium@observium.orgmailto:observium@observium.org> Date: 29.01.2015 20:39 Subject: Re: [Observium] An interesting issue with Cisco 2960Cs... Sent by: "observium" <observium-bounces@observium.orgmailto:observium-bounces@observium.org> ________________________________
Try latest revision (r6218). For my test device (WS-C2960C-8PC-L) polling time reduced from 124.78s -> to 17.52s (without disabling any module) :)
On Thu, Jan 29, 2015 at 7:14 PM, Richard Kuhns <rjk@wintek.commailto:rjk@wintek.com> wrote: Hi Colin,
On 01/29/15 08:19, Colin Stubbs wrote:
Hi Richard
You're right, 2960C's are L2 only; and my understanding is the same in that they shouldn't be doing CEF at all.
That said they're a similar hardware flavour to 2960S/3560X/3750X hardware, which can have L3 features enabled, so the software release in use probably has CEF code riddled right the way thru it.
What happens when you attempt to run CEF related commands on the CLI, e.g.
- show cef background
This command isn't available.
- show platform ip unicast cef-idb
This command returns immediately with no output.
e.g. are they available, do they return anything, does the CPU spike with the CLI commands too?
Nope, no CPU spike.
The other obvious questions:
- Which IOS version are you running?
We've now tested on 12.2(55)EX, 15.0(2)SE7, and 15.2(1)E3. All have the same problem. I can use snmpbulkwalk to query CISCO-CEF-MIB::cefFIBSummaryEntry and the SNMP ENGINE process CPU use jumps to over 80%, and stays there for 20 to 50 seconds.
- Is it the latest stable/recommended release?
15.0(2)SE7 is.
-Colin
Hopefully this info will be useful to other Observium users. It's definitely a Cisco problem, not an Observium problem. As long as we remember to disable cisco-cef when installing a 2960C everything is fine.
- Richard
-- Richard Kuhns <rjk@wintek.commailto:rjk@wintek.com> Main Number: 765-742-8428 Wintek Corporation Direct: 765-269-8541 427 N 6th Street Internet Support: 765-269-8503 Lafayette, IN 47901-2211 Consulting: 765-269-8504 _______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Mike Stupalov http://observium.org/_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
________________________________ Pensez à l'environnement avant d'imprimer ce message / Think of the environment before printing out this message
participants (6)
-
Colin Stubbs -
Joseph L. Brunner -
Karsten Schwarz -
Mike Stupalov -
Richard Kuhns -
Ron Marosko