Arista "sensors" module taking a very long time
I'm hoping the answer isn't "make arista fix their SNMP engine"
Polling a 7280 (64 ports of 10G) takes about 180 second, with 99% of that time in the sensors module.
Running it in debug I see a lot of stalls, and this effectively repeating
CMD[/usr/bin/snmpget -v2c -c *** -Pu -OUqnv -m SNMPv2-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':' core-1.aa.it.arbor.net':'161' aristaEntSensorThresholdHighWarning]
EXITCODE[1] CMD RUNTIME[0.005s] STDOUT[
] STDERR[ aristaEntSensorThresholdHighWarning: Unknown Object Identifier (Sub-id not found: (top) -> aristaEntSensorThresholdHighWarning) ] SNMP_STATUS[FALSE]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '96' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OUqnv' AND `mib` = 'SNMPv2-MIB' AND `oid` = 'aristaEntSensorThresholdHighWarning';] SQL RUNTIME[0.00026798s]
SQL[UPDATE `snmp_errors` set `error_count` ='471101',`updated` ='1448318766' WHERE `error_id` = '802'] SQL RUNTIME[0.00107193s] Array ( [sensor_value] => ) [RRD Disabled - update /opt/observium/rrd/ core-1.aa.it.arbor.net/sensor-power-entity-sensor-100315212.rrd N:0]
Any idea if that's something you can fix?
*Spencer Ryan* | Senior Systems Administrator | sryan@arbor.net *Arbor Networks* +1.734.794.5033 (d) | +1.734.846.2053 (m) www.arbornetworks.com
Arista is here, they wrote this code!
The repeats are probably related to missing MIB? I think that was fixed recently. Is this the latest version?
adam. On 23/11/2015 22:49:26, Spencer Ryan sryan@arbor.net wrote: I'm hoping the answer isn't "make arista fix their SNMP engine"
Polling a 7280 (64 ports of 10G) takes about 180 second, with 99% of that time in the sensors module.
Running it in debug I see a lot of stalls, and this effectively repeating
CMD[/usr/bin/snmpget -v2c -c *** -Pu -OUqnv -m SNMPv2-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'core-1.aa.it.arbor.net [http://core-1.aa.it.arbor.net%5D%27:%27161' aristaEntSensorThresholdHighWarning]
EXITCODE[1] CMD RUNTIME[0.005s] STDOUT[
] STDERR[ aristaEntSensorThresholdHighWarning: Unknown Object Identifier (Sub-id not found: (top) -> aristaEntSensorThresholdHighWarning) ] SNMP_STATUS[FALSE]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '96' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OUqnv' AND `mib` = 'SNMPv2-MIB' AND `oid` = 'aristaEntSensorThresholdHighWarning';] SQL RUNTIME[0.00026798s]
SQL[UPDATE `snmp_errors` set `error_count` ='471101',`updated` ='1448318766' WHERE `error_id` = '802'] SQL RUNTIME[0.00107193s] Array ( [sensor_value] => ) [RRD Disabled - update /opt/observium/rrd/core-1.aa.it.arbor.net/sensor-power-entity-sensor-100315212.rrd [http://core-1.aa.it.arbor.net/sensor-power-entity-sensor-100315212.rrd] N:0]
Any idea if that's something you can fix?
Spencer Ryan | Senior Systems Administrator | sryan@arbor.net [mailto:sryan@arbor.net] Arbor Networks +1.734.794.5033 (d) | +1.734.846.2053 (m) www.arbornetworks.com [http://www.arbornetworks.com/] _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Was on code from a few days ago, tried updating to head and still the same issues. Interestingly enough this started happening last Wednesday, which is when (I think) I moved some optics around on it......For fun I killed snmpd on the switch but that didn't help. Unfortunately I can't reboot the switch to see if that would fix it.
*Spencer Ryan* | Senior Systems Administrator | sryan@arbor.net *Arbor Networks* +1.734.794.5033 (d) | +1.734.846.2053 (m) www.arbornetworks.com
On Mon, Nov 23, 2015 at 6:02 PM, Adam Armstrong adama@memetic.org wrote:
Arista is here, they wrote this code!
The repeats are probably related to missing MIB? I think that was fixed recently. Is this the latest version?
adam.
On 23/11/2015 22:49:26, Spencer Ryan sryan@arbor.net wrote: I'm hoping the answer isn't "make arista fix their SNMP engine"
Polling a 7280 (64 ports of 10G) takes about 180 second, with 99% of that time in the sensors module.
Running it in debug I see a lot of stalls, and this effectively repeating
CMD[/usr/bin/snmpget -v2c -c *** -Pu -OUqnv -m SNMPv2-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':' core-1.aa.it.arbor.net':'161' aristaEntSensorThresholdHighWarning]
EXITCODE[1] CMD RUNTIME[0.005s] STDOUT[
] STDERR[ aristaEntSensorThresholdHighWarning: Unknown Object Identifier (Sub-id not found: (top) -> aristaEntSensorThresholdHighWarning) ] SNMP_STATUS[FALSE]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '96' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OUqnv' AND `mib` = 'SNMPv2-MIB' AND `oid` = 'aristaEntSensorThresholdHighWarning';] SQL RUNTIME[0.00026798s]
SQL[UPDATE `snmp_errors` set `error_count` ='471101',`updated` ='1448318766' WHERE `error_id` = '802'] SQL RUNTIME[0.00107193s] Array ( [sensor_value] => ) [RRD Disabled - update /opt/observium/rrd/ core-1.aa.it.arbor.net/sensor-power-entity-sensor-100315212.rrd N:0]
Any idea if that's something you can fix?
*Spencer Ryan* | Senior Systems Administrator | sryan@arbor.net *Arbor Networks* +1.734.794.5033 (d) | +1.734.846.2053 (m) www.arbornetworks.com _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
This was reported on IRC indeed, apparently some variable reuse? I think it's in Jira, but I thought it was committed, too...
On 24/11/2015 00:02, Adam Armstrong wrote:
Arista is here, they wrote this code!
The repeats are probably related to missing MIB? I think that was fixed recently. Is this the latest version?
adam.
On 23/11/2015 22:49:26, Spencer Ryan sryan@arbor.net wrote:
I'm hoping the answer isn't "make arista fix their SNMP engine"
Polling a 7280 (64 ports of 10G) takes about 180 second, with 99% of that time in the sensors module.
Running it in debug I see a lot of stalls, and this effectively repeating
CMD[/usr/bin/snmpget -v2c -c *** -Pu -OUqnv -m SNMPv2-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'core-1.aa.it.arbor.net http://core-1.aa.it.arbor.net':'161' aristaEntSensorThresholdHighWarning]
EXITCODE[1] CMD RUNTIME[0.005s] STDOUT[
] STDERR[ aristaEntSensorThresholdHighWarning: Unknown Object Identifier (Sub-id not found: (top) -> aristaEntSensorThresholdHighWarning) ] SNMP_STATUS[FALSE]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '96' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OUqnv' AND `mib` = 'SNMPv2-MIB' AND `oid` = 'aristaEntSensorThresholdHighWarning';] SQL RUNTIME[0.00026798s]
SQL[UPDATE `snmp_errors` set `error_count` ='471101',`updated` ='1448318766' WHERE `error_id` = '802'] SQL RUNTIME[0.00107193s] Array ( [sensor_value] => ) [RRD Disabled - update /opt/observium/rrd/core-1.aa.it.arbor.net/sensor-power-entity-sensor-100315212.rrd http://core-1.aa.it.arbor.net/sensor-power-entity-sensor-100315212.rrd N:0]
Any idea if that's something you can fix?
Spencer Ryan* | Senior Systems Administrator | sryan@arbor.net mailto:sryan@arbor.net *Arbor Networks* +1.734.794.5033 (d) | +1.734.846.2053 (m) www.arbornetworks.com http://www.arbornetworks.com/ _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
You're right Tom, it was fixed in OBSERVIUM-1570 http://jira.observium.org/browse/OBSERVIUM-1570 but I had to rerun discovery on the device for it to fix the broken sensor tables :)
*Spencer Ryan* | Senior Systems Administrator | sryan@arbor.net *Arbor Networks* +1.734.794.5033 (d) | +1.734.846.2053 (m) www.arbornetworks.com
On Mon, Nov 23, 2015 at 6:34 PM, Tom Laermans tom.laermans@powersource.cx wrote:
This was reported on IRC indeed, apparently some variable reuse? I think it's in Jira, but I thought it was committed, too...
On 24/11/2015 00:02, Adam Armstrong wrote:
Arista is here, they wrote this code!
The repeats are probably related to missing MIB? I think that was fixed recently. Is this the latest version?
adam.
On 23/11/2015 22:49:26, Spencer Ryan sryan@arbor.net sryan@arbor.net wrote: I'm hoping the answer isn't "make arista fix their SNMP engine"
Polling a 7280 (64 ports of 10G) takes about 180 second, with 99% of that time in the sensors module.
Running it in debug I see a lot of stalls, and this effectively repeating
CMD[/usr/bin/snmpget -v2c -c *** -Pu -OUqnv -m SNMPv2-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':' core-1.aa.it.arbor.net':'161' aristaEntSensorThresholdHighWarning]
EXITCODE[1] CMD RUNTIME[0.005s] STDOUT[
] STDERR[ aristaEntSensorThresholdHighWarning: Unknown Object Identifier (Sub-id not found: (top) -> aristaEntSensorThresholdHighWarning) ] SNMP_STATUS[FALSE]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '96' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OUqnv' AND `mib` = 'SNMPv2-MIB' AND `oid` = 'aristaEntSensorThresholdHighWarning';] SQL RUNTIME[0.00026798s]
SQL[UPDATE `snmp_errors` set `error_count` ='471101',`updated` ='1448318766' WHERE `error_id` = '802'] SQL RUNTIME[0.00107193s] Array ( [sensor_value] => ) [RRD Disabled - update /opt/observium/rrd/ core-1.aa.it.arbor.net/sensor-power-entity-sensor-100315212.rrd N:0]
Any idea if that's something you can fix?
- Spencer Ryan* | Senior Systems Administrator | sryan@arbor.net
sryan@arbor.net *Arbor Networks* +1.734.794.5033 (d) | +1.734.846.2053 (m) http://www.arbornetworks.com/www.arbornetworks.com _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing listobservium@observium.orghttp://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (3)
-
Adam Armstrong -
Spencer Ryan -
Tom Laermans