High CPU load on Cisco S500G Small business switches
Hi,
We've noticed an issue with Cisco SG500 switches and Observium, where if you enable stacking mode (the default on the switch), regardless of if it is in a stack or not, the CPU reports a 50% average. Our investigations show it runs at 95% + for a couple of minutes whilst Observium is polling it, then drops back down to <10% when the query finishes. Turning stacking mode off drops the average CPU to <10% and CPU does not ramp up and down.
We raised a case with Cisco TAC, who said this is "expected behaviour":
At every 5 min cycle, the CPU rises to 85- 95% temporarily and then drops down. The reason for this is, Observium is checking for interface status. When switch is in standalone mode the interface count database is the max switch capacity. Example 48 ports = 48 ports plus channel ports ~ 60 ports In stack mode the predefined port count in the data base is around 880 ports. While the switch is responding to SNMP get , it is using up resources and that is why the CPU utilization will be high. While I was testing it, I did not see or encountered any network outage or Web access issues. We have already made an enhancement request for this behaviour, to limit the port count to any stack configuration.
Has anyone else encountered anything similar?
Adam - I realise that Observium is only polling what the switch is reporting, but if Observium can ascertain how many switches are in the stack, would it be possible to limit polling to only query active/connected stack members?
Thanks,
Jamie
Short answer:
Stacking is retarded bullshit.
Long answer:
Aarrrghhrgrhrgrhrgrhhrhgghrhgrgrhtgrh.
Adam.
Sent from BlueMail
On 29 Sep 2016, 09:27, at 09:27, Jamie Tynan jamie.tynan@g.timico.net wrote:
Hi,
We've noticed an issue with Cisco SG500 switches and Observium, where if you enable stacking mode (the default on the switch), regardless of if it is in a stack or not, the CPU reports a 50% average. Our investigations show it runs at 95% + for a couple of minutes whilst Observium is polling it, then drops back down to <10% when the query finishes. Turning stacking mode off drops the average CPU to <10% and CPU does not ramp up and down.
We raised a case with Cisco TAC, who said this is "expected behaviour":
At every 5 min cycle, the CPU rises to 85- 95% temporarily and then drops down. The reason for this is, Observium is checking for interface status. When switch is in standalone mode the interface count database is the max switch capacity. Example 48 ports = 48 ports plus channel ports ~ 60 ports In stack mode the predefined port count in the data base is around 880 ports. While the switch is responding to SNMP get , it is using up resources and that is why the CPU utilization will be high. While I was testing it, I did not see or encountered any network outage or Web access issues. We have already made an enhancement request for this behaviour, to limit the port count to any stack configuration.
Has anyone else encountered anything similar?
Adam - I realise that Observium is only polling what the switch is reporting, but if Observium can ascertain how many switches are in the stack, would it be possible to limit polling to only query active/connected stack members?
Thanks,
Jamie
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Lol :-)
Is this something you could/will have a look into (I wasn't sure from your reply)?
Thanks,
Jamie
On 29 September 2016 at 15:33, Adam Armstrong adama@memetic.org wrote:
Short answer:
Stacking is retarded bullshit.
Long answer:
Aarrrghhrgrhrgrhrgrhhrhgghrhgrgrhtgrh.
Adam.
Sent from BlueMail http://www.bluemail.me/r
On 29 Sep 2016, at 09:27, Jamie Tynan jamie.tynan@g.timico.net wrote:
Hi,
We've noticed an issue with Cisco SG500 switches and Observium, where if you enable stacking mode (the default on the switch), regardless of if it is in a stack or not, the CPU reports a 50% average. Our investigations show it runs at 95% + for a couple of minutes whilst Observium is polling it, then drops back down to <10% when the query finishes. Turning stacking mode off drops the average CPU to <10% and CPU does not ramp up and down.
We raised a case with Cisco TAC, who said this is "expected behaviour":
At every 5 min cycle, the CPU rises to 85- 95% temporarily and then drops down. The reason for this is, Observium is checking for interface status. When switch is in standalone mode the interface count database is the max switch capacity. Example 48 ports = 48 ports plus channel ports ~ 60 ports In stack mode the predefined port count in the data base is around 880 ports. While the switch is responding to SNMP get , it is using up resources and that is why the CPU utilization will be high. While I was testing it, I did not see or encountered any network outage or Web access issues. We have already made an enhancement request for this behaviour, to limit the port count to any stack configuration.
Has anyone else encountered anything similar?
Adam - I realise that Observium is only polling what the switch is reporting, but if Observium can ascertain how many switches are in the stack, would it be possible to limit polling to only query active/connected stack members?
Thanks,
Jamie
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
There's nothing we can do! If your device is taking forever to return data, how do we fix that?
Polling across stacks is slow enough to by unusable on a lot of platforms, but this is the least of reasons why one should never use stacking.
adam. On 29/09/2016 11:22:33, Jamie Tynan jamie.tynan@g.timico.net wrote: Lol :-)
Is this something you could/will have a look into (I wasn't sure from your reply)?
Thanks,
Jamie
On 29 September 2016 at 15:33, Adam Armstrong <adama@memetic.org [mailto:adama@memetic.org]> wrote:
Short answer: Stacking is retarded bullshit. Long answer: Aarrrghhrgrhrgrhrgrhhrhgghrhgrgrhtgrh. Adam. Sent from BlueMail [http://www.bluemail.me/r]
On 29 Sep 2016, at 09:27, Jamie Tynan <jamie.tynan@g.timico.net [mailto:jamie.tynan@g.timico.net]> wrote: Hi,
We've noticed an issue with Cisco SG500 switches and Observium, where if you enable stacking mode (the default on the switch), regardless of if it is in a stack or not, the CPU reports a 50% average. Our investigations show it runs at 95% + for a couple of minutes whilst Observium is polling it, then drops back down to <10% when the query finishes. Turning stacking mode off drops the average CPU to <10% and CPU does not ramp up and down.
We raised a case with Cisco TAC, who said this is "expected behaviour":
At every 5 min cycle, the CPU rises to 85- 95% temporarily and then drops down. The reason for this is, Observium is checking for interface status. When switch is in standalone mode the interface count database is the max switch capacity. Example 48 ports = 48 ports plus channel ports ~ 60 ports In stack mode the predefined port count in the data base is around 880 ports. While the switch is responding to SNMP get , it is using up resources and that is why the CPU utilization will be high. While I was testing it, I did not see or encountered any network outage or Web access issues. We have already made an enhancement request for this behaviour, to limit the port count to any stack configuration.
Has anyone else encountered anything similar?
Adam - I realise that Observium is only polling what the switch is reporting, but if Observium can ascertain how many switches are in the stack, would it be possible to limit polling to only query active/connected stack members?
Thanks,
Jamie
observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
OK understood.
Thanks for the speedy reply.
Jamie
On 29 September 2016 at 17:24, Adam Armstrong adama@memetic.org wrote:
There's nothing we can do! If your device is taking forever to return data, how do we fix that?
Polling across stacks is slow enough to by unusable on a lot of platforms, but this is the least of reasons why one should never use stacking.
adam.
On 29/09/2016 11:22:33, Jamie Tynan jamie.tynan@g.timico.net wrote: Lol :-)
Is this something you could/will have a look into (I wasn't sure from your reply)?
Thanks,
Jamie
On 29 September 2016 at 15:33, Adam Armstrong adama@memetic.org wrote:
Short answer:
Stacking is retarded bullshit.
Long answer:
Aarrrghhrgrhrgrhrgrhhrhgghrhgrgrhtgrh.
Adam.
Sent from BlueMail http://www.bluemail.me/r
On 29 Sep 2016, at 09:27, Jamie Tynan jamie.tynan@g.timico.net wrote:
Hi,
We've noticed an issue with Cisco SG500 switches and Observium, where if you enable stacking mode (the default on the switch), regardless of if it is in a stack or not, the CPU reports a 50% average. Our investigations show it runs at 95% + for a couple of minutes whilst Observium is polling it, then drops back down to <10% when the query finishes. Turning stacking mode off drops the average CPU to <10% and CPU does not ramp up and down.
We raised a case with Cisco TAC, who said this is "expected behaviour":
At every 5 min cycle, the CPU rises to 85- 95% temporarily and then drops down. The reason for this is, Observium is checking for interface status. When switch is in standalone mode the interface count database is the max switch capacity. Example 48 ports = 48 ports plus channel ports ~ 60 ports In stack mode the predefined port count in the data base is around 880 ports. While the switch is responding to SNMP get , it is using up resources and that is why the CPU utilization will be high. While I was testing it, I did not see or encountered any network outage or Web access issues. We have already made an enhancement request for this behaviour, to limit the port count to any stack configuration.
Has anyone else encountered anything similar?
Adam - I realise that Observium is only polling what the switch is reporting, but if Observium can ascertain how many switches are in the stack, would it be possible to limit polling to only query active/connected stack members?
Thanks,
Jamie
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/ cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (2)
-
Adam Armstrong -
Jamie Tynan