Discovery crashing UPS management card
Good morning all,
I'm a recent adopter of Observium so still finding my feet and slowly learning how best to set it up. However, I have noticed that since installing it, it was reporting that my main UPS management card was rebooting every 6 hours. The UPS is an Eaton 9SX 5000i + battery pack, with Network-MS management card, firmware HF which is the latest available for card tech level 16.
Dabbling more, it seems that it's the discovery process that is causing it, specifically when it walks the xupsOutput table (.1.3.6.1.4.1.534.1.4) in the Powerware MIB, the UPS management card immediately stops responding and reboots. It actually appears to be crashing on doing a GetNext on .1.3.6.1.4.1.534.1.4.7.0, which is beyond the defined values in that table. The UPS continues to operate, but the management card stops. This also means that Observium has only managed to discover the 'input' voltage - output, bypass and current/power readings all just read as zero as the card has stopped responding when it tries to read those. The poll runs fine and gets data from the input voltage sensor.
Clearly this is not an Observium issue: I can reproduce the reboot with a simple snmpwalk on the commandline, which should be a valid operation on a published MIB. I do have a ticket open with the manufacturer, although the first response was 'does it work with our management software?'. It may even be a fault on my unit, but I don't have more than one of these to test.
I'm assuming there's a reason for the discovery module to need to walk the tables rather than just read the specific values - presumably bigger UPS units might have multiple metered AC inputs and outputs rather than just the one?
I could disable sensor discovery, which will stop the reboots but that won't let it learn about the additional sensors that it still hasn't found. Unless I can manually insert data into the database for the poller to pick up?
Alternatively, is there a way for me to override the detection of a Powerware UPS and use the generic UPS MIB rather than the XUPS MIB?
Any other thoughts?
Steve.
Hi,
We walk tables because that's the only way to discover what's actually in them.
Different devices have different numbers and locations of sensors, so it's the only realistic way of discovering 90% of things.
You can probably disable XUPS-MIB in the device module settings, but I'm not sure.
adam.
On 2014-04-01 04:43, Steve Kersley wrote:
Good morning all,
I'm a recent adopter of Observium so still finding my feet and slowly learning how best to set it up. However, I have noticed that since installing it, it was reporting that my main UPS management card was rebooting every 6 hours. The UPS is an Eaton 9SX 5000i + battery pack, with Network-MS management card, firmware HF which is the latest available for card tech level 16.
Dabbling more, it seems that it's the discovery process that is causing it, specifically when it walks the xupsOutput table (.1.3.6.1.4.1.534.1.4) in the Powerware MIB, the UPS management card immediately stops responding and reboots. It actually appears to be crashing on doing a GetNext on .1.3.6.1.4.1.534.1.4.7.0, which is beyond the defined values in that table. The UPS continues to operate, but the management card stops. This also means that Observium has only managed to discover the 'input' voltage - output, bypass and current/power readings all just read as zero as the card has stopped responding when it tries to read those. The poll runs fine and gets data from the input voltage sensor.
Clearly this is not an Observium issue: I can reproduce the reboot with a simple snmpwalk on the commandline, which should be a valid operation on a published MIB. I do have a ticket open with the manufacturer, although the first response was 'does it work with our management software?'. It may even be a fault on my unit, but I don't have more than one of these to test.
I'm assuming there's a reason for the discovery module to need to walk the tables rather than just read the specific values - presumably bigger UPS units might have multiple metered AC inputs and outputs rather than just the one?
I could disable sensor discovery, which will stop the reboots but that won't let it learn about the additional sensors that it still hasn't found. Unless I can manually insert data into the database for the poller to pick up?
Alternatively, is there a way for me to override the detection of a Powerware UPS and use the generic UPS MIB rather than the XUPS MIB?
Any other thoughts?
Steve. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
It's not a separate module, we don't have toggles for MIBs at this point. We could add this of course.
I don't think powerware has UPS-MIB in its mibs list right now, do they (also) support it? Enabling both will give you duplicate readings, I have uncommitted unfinished code to do away with the dupes.
Tom
On 01/04/2014 23:14, Adam Armstrong wrote:
Hi,
We walk tables because that's the only way to discover what's actually in them.
Different devices have different numbers and locations of sensors, so it's the only realistic way of discovering 90% of things.
You can probably disable XUPS-MIB in the device module settings, but I'm not sure.
adam.
On 2014-04-01 04:43, Steve Kersley wrote:
Good morning all,
I'm a recent adopter of Observium so still finding my feet and slowly learning how best to set it up. However, I have noticed that since installing it, it was reporting that my main UPS management card was rebooting every 6 hours. The UPS is an Eaton 9SX 5000i + battery pack, with Network-MS management card, firmware HF which is the latest available for card tech level 16.
Dabbling more, it seems that it's the discovery process that is causing it, specifically when it walks the xupsOutput table (.1.3.6.1.4.1.534.1.4) in the Powerware MIB, the UPS management card immediately stops responding and reboots. It actually appears to be crashing on doing a GetNext on .1.3.6.1.4.1.534.1.4.7.0, which is beyond the defined values in that table. The UPS continues to operate, but the management card stops. This also means that Observium has only managed to discover the 'input' voltage - output, bypass and current/power readings all just read as zero as the card has stopped responding when it tries to read those. The poll runs fine and gets data from the input voltage sensor.
Clearly this is not an Observium issue: I can reproduce the reboot with a simple snmpwalk on the commandline, which should be a valid operation on a published MIB. I do have a ticket open with the manufacturer, although the first response was 'does it work with our management software?'. It may even be a fault on my unit, but I don't have more than one of these to test.
I'm assuming there's a reason for the discovery module to need to walk the tables rather than just read the specific values - presumably bigger UPS units might have multiple metered AC inputs and outputs rather than just the one?
I could disable sensor discovery, which will stop the reboots but that won't let it learn about the additional sensors that it still hasn't found. Unless I can manually insert data into the database for the poller to pick up?
Alternatively, is there a way for me to override the detection of a Powerware UPS and use the generic UPS MIB rather than the XUPS MIB?
Any other thoughts?
Steve. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi,
I've got the same issue with an EATON 5PX-2200 with NetworkMS card (with latest firmware JB for card tech level 17).
Voltage input value in observium is at 0 when a reboot happens, but in between, correct values show up. Otherwise, voltage output/bypass is always at 0.
The thing here is that it didn't happen at first and nothing has been changed is the setup / environnement before and after the beginning of this issue. Also, it doesn't happen on another UPS with same model same card same firmware.
Any workaround for this issue ? Or is it hardware related and network card needs RMA ?
Stephen
participants (4)
-
Adam Armstrong -
Stephen Morri -
Steve Kersley -
Tom Laermans