HIgh CPU usage caused by snmp process
Hi All,
Couple of our Force 10 (S4810) Switches are reporting high CPU usage which seems to be caused by snmp processes. This is impacting the network performance.
Do we know the MIBS (modules) that cause high load? And how can we disable these MIBs.
Thanks!
Regards Renuka
________________________________ This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
High CPU load from SNMP shouldn't affect network performance, unless the OS in question is completely missing any form of scheduling, which I doubt FTOS is. It's far more likely that you have an unrelated network issue that is being blamed on high cpu usage.
You can see which modules take which amount of time by clicking on the clock icon to the right of the navbar on a device page.
adam. On 23/02/2015 22:49:07, Arya, Renuka [BSD] - MED rarya@medicine.bsd.uchicago.edu wrote: Hi All,
Couple of our Force 10 (S4810) Switches are reporting high CPU usage which seems to be caused by snmp processes. This is impacting the network performance.
Do we know the MIBS (modules) that cause high load? And how can we disable these MIBs.
Thanks!
Regards Renuka
This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
Thanks Adam!
The modules page shows the duration for individual modules. How can I disable these modules to see if that helps?
Regards Renuka ________________________________ From: observium [observium-bounces@observium.org] on behalf of Adam Armstrong [adama@memetic.org] Sent: Monday, February 23, 2015 10:51 AM To: observium@observium.org Subject: Re: [Observium] HIgh CPU usage caused by snmp process
High CPU load from SNMP shouldn't affect network performance, unless the OS in question is completely missing any form of scheduling, which I doubt FTOS is. It's far more likely that you have an unrelated network issue that is being blamed on high cpu usage.
You can see which modules take which amount of time by clicking on the clock icon to the right of the navbar on a device page.
adam.
On 23/02/2015 22:49:07, Arya, Renuka [BSD] - MED rarya@medicine.bsd.uchicago.edu wrote:
Hi All,
Couple of our Force 10 (S4810) Switches are reporting high CPU usage which seems to be caused by snmp processes. This is impacting the network performance.
Do we know the MIBS (modules) that cause high load? And how can we disable these MIBs.
Thanks!
Regards Renuka
________________________________ This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
________________________________ This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
In the settings for the device (or globally in the config), the settings for the device is the gear icon, near the clock icon.
MIBs and Modules can be disabled on a device in the MIBs and Modules sections of the device settings.
adam. On 24/02/2015 14:40:51, Arya, Renuka [BSD] - MED rarya@medicine.bsd.uchicago.edu wrote: Thanks Adam!
The modules page shows the duration for individual modules. How can I disable these modules to see if that helps?
Regards Renuka
From: observium [observium-bounces@observium.org] on behalf of Adam Armstrong [adama@memetic.org] Sent: Monday, February 23, 2015 10:51 AM To: observium@observium.org Subject: Re: [Observium] HIgh CPU usage caused by snmp process
High CPU load from SNMP shouldn't affect network performance, unless the OS in question is completely missing any form of scheduling, which I doubt FTOS is. It's far more likely that you have an unrelated network issue that is being blamed on high cpu usage.
You can see which modules take which amount of time by clicking on the clock icon to the right of the navbar on a device page.
adam. On 23/02/2015 22:49:07, Arya, Renuka [BSD] - MED rarya@medicine.bsd.uchicago.edu wrote: Hi All,
Couple of our Force 10 (S4810) Switches are reporting high CPU usage which seems to be caused by snmp processes. This is impacting the network performance.
Do we know the MIBS (modules) that cause high load? And how can we disable these MIBs.
Thanks!
Regards Renuka
This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
Thanks Adam!
I was digging into the logs of the two Force 10 switches and it seems that both of the switches started reporting high CPU at the same time.
The other thing we noticed was the system reported down for the first time at about the same time (2015-02-20 at 09:57 AM) and after it came back all the ports on the switch reported deleted.
After all ports were deleted we see this weird issue of system reporting up and down intermittently and high CPU.
Please notice we don't see any issue on the switches itself. I don't see high CPU and never loose connectivity to the switches.
Thanks!
Regards Renuka
________________________________ From: observium [observium-bounces@observium.org] on behalf of Adam Armstrong [adama@memetic.org] Sent: Tuesday, February 24, 2015 1:24 PM To: observium@observium.org Subject: Re: [Observium] HIgh CPU usage caused by snmp process
In the settings for the device (or globally in the config), the settings for the device is the gear icon, near the clock icon.
MIBs and Modules can be disabled on a device in the MIBs and Modules sections of the device settings.
adam.
On 24/02/2015 14:40:51, Arya, Renuka [BSD] - MED rarya@medicine.bsd.uchicago.edu wrote:
Thanks Adam!
The modules page shows the duration for individual modules. How can I disable these modules to see if that helps?
Regards Renuka ________________________________ From: observium [observium-bounces@observium.org] on behalf of Adam Armstrong [adama@memetic.org] Sent: Monday, February 23, 2015 10:51 AM To: observium@observium.org Subject: Re: [Observium] HIgh CPU usage caused by snmp process
High CPU load from SNMP shouldn't affect network performance, unless the OS in question is completely missing any form of scheduling, which I doubt FTOS is. It's far more likely that you have an unrelated network issue that is being blamed on high cpu usage.
You can see which modules take which amount of time by clicking on the clock icon to the right of the navbar on a device page.
adam.
On 23/02/2015 22:49:07, Arya, Renuka [BSD] - MED rarya@medicine.bsd.uchicago.edu wrote:
Hi All,
Couple of our Force 10 (S4810) Switches are reporting high CPU usage which seems to be caused by snmp processes. This is impacting the network performance.
Do we know the MIBS (modules) that cause high load? And how can we disable these MIBs.
Thanks!
Regards Renuka
________________________________ This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
________________________________ This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
________________________________ This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
Make sure you're running the latest software for the switches. SNMP bugs are very common, and due to the comprehensive polling Observium does, we hit every last one of them!
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 25 February 2015 08:58:57 "Arya, Renuka [BSD] - MED" rarya@medicine.bsd.uchicago.edu wrote:
Thanks Adam!
I was digging into the logs of the two Force 10 switches and it seems that both of the switches started reporting high CPU at the same time.
The other thing we noticed was the system reported down for the first time at about the same time (2015-02-20 at 09:57 AM) and after it came back all the ports on the switch reported deleted.
After all ports were deleted we see this weird issue of system reporting up and down intermittently and high CPU.
Please notice we don't see any issue on the switches itself. I don't see high CPU and never loose connectivity to the switches.
Thanks!
Regards Renuka
From: observium [observium-bounces@observium.org] on behalf of Adam Armstrong [adama@memetic.org] Sent: Tuesday, February 24, 2015 1:24 PM To: observium@observium.org Subject: Re: [Observium] HIgh CPU usage caused by snmp process
In the settings for the device (or globally in the config), the settings for the device is the gear icon, near the clock icon.
MIBs and Modules can be disabled on a device in the MIBs and Modules sections of the device settings.
adam.
On 24/02/2015 14:40:51, Arya, Renuka [BSD] - MED rarya@medicine.bsd.uchicago.edu wrote:
Thanks Adam!
The modules page shows the duration for individual modules. How can I disable these modules to see if that helps?
Regards Renuka ________________________________ From: observium [observium-bounces@observium.org] on behalf of Adam Armstrong [adama@memetic.org] Sent: Monday, February 23, 2015 10:51 AM To: observium@observium.org Subject: Re: [Observium] HIgh CPU usage caused by snmp process
High CPU load from SNMP shouldn't affect network performance, unless the OS in question is completely missing any form of scheduling, which I doubt FTOS is. It's far more likely that you have an unrelated network issue that is being blamed on high cpu usage.
You can see which modules take which amount of time by clicking on the clock icon to the right of the navbar on a device page.
adam.
On 23/02/2015 22:49:07, Arya, Renuka [BSD] - MED rarya@medicine.bsd.uchicago.edu wrote:
Hi All,
Couple of our Force 10 (S4810) Switches are reporting high CPU usage which seems to be caused by snmp processes. This is impacting the network performance.
Do we know the MIBS (modules) that cause high load? And how can we disable these MIBs.
Thanks!
Regards Renuka
This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
We ran into SNMP memory leaks on our Force10 S4810 switches, which eventually hung the devices and required a hard power reset. (Yuck!)
Dell has published new software that might fix this (Note: I have not tested this yet, and there are still some open SNMP bugs):
(Sadly, this link requires login) https://www.force10networks.com/CSPortal20/Software/documentation/S4810_RN_9...
Resolved Caveats:
PR# 139394 Severity: Sev 2 Synopsis: System in stacking configuration may reboot when SNMP operations are executed to retrieve CPU/Memory utilization statistics
PR# 140566 Severity: Sev 2 Synopsis: The system does not return valid data when SNMP Get operations are performed to retrieve CPU and memory utilization statistics
*Tristan Rhodes* Network Engineer Weber State University 801.626.8549
On Mon, Feb 23, 2015 at 3:48 PM, Arya, Renuka [BSD] - MED < rarya@medicine.bsd.uchicago.edu> wrote:
Hi All,
Couple of our Force 10 (S4810) Switches are reporting high CPU usage which seems to be caused by snmp processes. This is impacting the network performance.
Do we know the MIBS (modules) that cause high load? And how can we disable these MIBs.
Thanks!
Regards Renuka
This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Which software versions were you running when you saw these bugs?
Bw
Rob
Sent from my iPhone
On 23 Feb 2015, at 23:45, Tristan Rhodes <tristanrhodes@weber.edumailto:tristanrhodes@weber.edu> wrote:
We ran into SNMP memory leaks on our Force10 S4810 switches, which eventually hung the devices and required a hard power reset. (Yuck!)
Dell has published new software that might fix this (Note: I have not tested this yet, and there are still some open SNMP bugs):
(Sadly, this link requires login) https://www.force10networks.com/CSPortal20/Software/documentation/S4810_RN_9...
Resolved Caveats:
PR# 139394 Severity: Sev 2 Synopsis: System in stacking configuration may reboot when SNMP operations are executed to retrieve CPU/Memory utilization statistics
PR# 140566 Severity: Sev 2 Synopsis: The system does not return valid data when SNMP Get operations are performed to retrieve CPU and memory utilization statistics
Tristan Rhodes Network Engineer Weber State University 801.626.8549 [http://www.weber.edu/wsuimages/brand/templates/emailsig_sig1.jpg]
On Mon, Feb 23, 2015 at 3:48 PM, Arya, Renuka [BSD] - MED <rarya@medicine.bsd.uchicago.edumailto:rarya@medicine.bsd.uchicago.edu> wrote: Hi All,
Couple of our Force 10 (S4810) Switches are reporting high CPU usage which seems to be caused by snmp processes. This is impacting the network performance.
Do we know the MIBS (modules) that cause high load? And how can we disable these MIBs.
Thanks!
Regards Renuka
________________________________ This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi All,
Thanks for you reply!
We have 9 Force 10 S4810 switches. Out of which three are running 8.3.12.0 and others are running 9.4(0.0P1).
We seeing this issue with two of the switches running 8.3.12.0. However, one of the Force 10 running 8.3.12.0 doesn't have the issue.
I believe, I have started seeing this issue couple of days back.
The issue seems intermittent. Please see the attachments.
I ran a continuous ping to the switches but never is any disconnection there. The CPU processes on the switches doesn't show high CPU. So, not sure what causing the issue.
Regards Renuka
________________________________ From: observium [observium-bounces@observium.org] on behalf of Robert Hatch [Robert.Hatch@vorboss.com] Sent: Tuesday, February 24, 2015 4:03 AM To: Observium Network Observation System Subject: Re: [Observium] HIgh CPU usage caused by snmp process
Which software versions were you running when you saw these bugs?
Bw
Rob
Sent from my iPhone
On 23 Feb 2015, at 23:45, Tristan Rhodes <tristanrhodes@weber.edumailto:tristanrhodes@weber.edu> wrote:
We ran into SNMP memory leaks on our Force10 S4810 switches, which eventually hung the devices and required a hard power reset. (Yuck!)
Dell has published new software that might fix this (Note: I have not tested this yet, and there are still some open SNMP bugs):
(Sadly, this link requires login) https://www.force10networks.com/CSPortal20/Software/documentation/S4810_RN_9...
Resolved Caveats:
PR# 139394 Severity: Sev 2 Synopsis: System in stacking configuration may reboot when SNMP operations are executed to retrieve CPU/Memory utilization statistics
PR# 140566 Severity: Sev 2 Synopsis: The system does not return valid data when SNMP Get operations are performed to retrieve CPU and memory utilization statistics
Tristan Rhodes Network Engineer Weber State University 801.626.8549 [http://www.weber.edu/wsuimages/brand/templates/emailsig_sig1.jpg]
On Mon, Feb 23, 2015 at 3:48 PM, Arya, Renuka [BSD] - MED <rarya@medicine.bsd.uchicago.edumailto:rarya@medicine.bsd.uchicago.edu> wrote: Hi All,
Couple of our Force 10 (S4810) Switches are reporting high CPU usage which seems to be caused by snmp processes. This is impacting the network performance.
Do we know the MIBS (modules) that cause high load? And how can we disable these MIBs.
Thanks!
Regards Renuka
________________________________ This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
________________________________ This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
FTOS (DNOS) 9.5 is the version that has the memory leak that will hang a S4810 after a few weeks of Observium polling.
Tristan
*Tristan Rhodes* Network Engineer Weber State University 801.626.8549
On Tue, Feb 24, 2015 at 3:03 AM, Robert Hatch Robert.Hatch@vorboss.com wrote:
Which software versions were you running when you saw these bugs?
Bw
Rob
Sent from my iPhone
On 23 Feb 2015, at 23:45, Tristan Rhodes tristanrhodes@weber.edu wrote:
We ran into SNMP memory leaks on our Force10 S4810 switches, which eventually hung the devices and required a hard power reset. (Yuck!)
Dell has published new software that might fix this (Note: I have not tested this yet, and there are still some open SNMP bugs):
(Sadly, this link requires login)
https://www.force10networks.com/CSPortal20/Software/documentation/S4810_RN_9...
Resolved Caveats:
PR# 139394 Severity: Sev 2 Synopsis: System in stacking configuration may reboot when SNMP operations are executed to retrieve CPU/Memory utilization statistics
PR# 140566 Severity: Sev 2 Synopsis: The system does not return valid data when SNMP Get operations are performed to retrieve CPU and memory utilization statistics
*Tristan Rhodes* Network Engineer Weber State University 801.626.8549
On Mon, Feb 23, 2015 at 3:48 PM, Arya, Renuka [BSD] - MED < rarya@medicine.bsd.uchicago.edu> wrote:
Hi All,
Couple of our Force 10 (S4810) Switches are reporting high CPU usage which seems to be caused by snmp processes. This is impacting the network performance.
Do we know the MIBS (modules) that cause high load? And how can we disable these MIBs.
Thanks!
Regards Renuka
This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this email message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is prohibited. If you have received this email in error, please notify the sender and destroy/delete all copies of the transmittal.
Thank you.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (4)
-
Adam Armstrong -
Arya, Renuka [BSD] - MED -
Robert Hatch -
Tristan Rhodes