Observium SNMP polling crashes various Adtran devices
Hi,
We recently installed Observium in an effort to replace our aging Cacti and Nagios systems. We are running the latest code checked out of SVN (We paid the money to Adam for the Professional Version)
Today we added 180+ Adtran based devices to the system. All 180+ devices began rebooting. They would become stable for a few hours and then start rebooting again.
Important to note that most of these items have been in our network for 3+ years, are running recent code, and have been rock solid stable for their entire service life. Cacti and Nagios poll them every minute and have been doing so for the past 3 years.
Insert Observium and the devices start failing.
My initial feeling is that there is something in the way Observium does its SNMP.
Curious if anyone else has experienced this or something like this. Its caused us to halt putting more devices on Observium.
Devices are TA 5000 / TA 352 GPON ONT's, TA 9xx iad's
When I add a single device via
add_device <name>
and then do a
discover.php -h new
as soon as the code hits ENTITY-SENSOR-MIB the adtran device reboots
noc@observium:/home/observium# ./discovery.php -h new Observium 0.14.12.6107 Discovery
ont-03.sip.pon-1-6-1.fttx.XXX.YYYYY.net 38 adtran-aos (adtran-aos) Module [ os ] time: 0.0005s Ports : .... Module [ ports ] time: 0.1889s Port Stacks: Module [ ports-stack ] time: 0.0774s Processors : ADTRAN-AOSCPU + hrDevice: Module [ processors ] time: 0.2734s Memory : ADTRAN-AOSCPU + Module [ mempools ] time: 0.2019s IPv4 Addresses : .. Module [ ipv4-addresses ] time: 0.1366s IPv6 Addresses : Module [ ipv6-addresses ] time: 0.5068s Sensors: ENTITY-SENSOR-MIB
So? Disable discovery sensors and you should be fine. Don't forget to complain vendor about faulty snmp stack. It's common to see low-end devices crashing because of snmp polling.
On 17 дек. 2014 г., at 3:03, John Brown john@citylinkfiber.com wrote:
Hi,
We recently installed Observium in an effort to replace our aging Cacti and Nagios systems. We are running the latest code checked out of SVN (We paid the money to Adam for the Professional Version)
Today we added 180+ Adtran based devices to the system. All 180+ devices began rebooting. They would become stable for a few hours and then start rebooting again.
Important to note that most of these items have been in our network for 3+ years, are running recent code, and have been rock solid stable for their entire service life. Cacti and Nagios poll them every minute and have been doing so for the past 3 years.
Insert Observium and the devices start failing.
My initial feeling is that there is something in the way Observium does its SNMP.
Curious if anyone else has experienced this or something like this. Its caused us to halt putting more devices on Observium.
Devices are TA 5000 / TA 352 GPON ONT's, TA 9xx iad's
When I add a single device via
add_device <name>
and then do a
discover.php -h new
as soon as the code hits ENTITY-SENSOR-MIB the adtran device reboots
noc@observium:/home/observium# ./discovery.php -h new Observium 0.14.12.6107 Discovery
ont-03.sip.pon-1-6-1.fttx.XXX.YYYYY.net 38 adtran-aos (adtran-aos) Module [ os ] time: 0.0005s Ports : .... Module [ ports ] time: 0.1889s Port Stacks: Module [ ports-stack ] time: 0.0774s Processors : ADTRAN-AOSCPU + hrDevice: Module [ processors ] time: 0.2734s Memory : ADTRAN-AOSCPU + Module [ mempools ] time: 0.2019s IPv4 Addresses : .. Module [ ipv4-addresses ] time: 0.1366s IPv6 Addresses : Module [ ipv6-addresses ] time: 0.5068s Sensors: ENTITY-SENSOR-MIB
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Oops, time to complain to Adtran their SNMP stack is shit.
On 17/12/2014 01:03, John Brown wrote:
Hi,
We recently installed Observium in an effort to replace our aging Cacti and Nagios systems. We are running the latest code checked out of SVN (We paid the money to Adam for the Professional Version)
Today we added 180+ Adtran based devices to the system. All 180+ devices began rebooting. They would become stable for a few hours and then start rebooting again.
Important to note that most of these items have been in our network for 3+ years, are running recent code, and have been rock solid stable for their entire service life. Cacti and Nagios poll them every minute and have been doing so for the past 3 years.
Insert Observium and the devices start failing.
My initial feeling is that there is something in the way Observium does its SNMP.
Curious if anyone else has experienced this or something like this. Its caused us to halt putting more devices on Observium.
Devices are TA 5000 / TA 352 GPON ONT's, TA 9xx iad's
When I add a single device via
add_device <name>
and then do a
discover.php -h new
as soon as the code hits ENTITY-SENSOR-MIB the adtran device reboots
noc@observium:/home/observium# ./discovery.php -h new Observium 0.14.12.6107 Discovery
ont-03.sip.pon-1-6-1.fttx.XXX.YYYYY.net http://ont-03.sip.pon-1-6-1.fttx.XXX.YYYYY.net 38 adtran-aos (adtran-aos) Module [ os ] time: 0.0005s Ports : .... Module [ ports ] time: 0.1889s Port Stacks: Module [ ports-stack ] time: 0.0774s Processors : ADTRAN-AOSCPU + hrDevice: Module [ processors ] time: 0.2734s Memory : ADTRAN-AOSCPU + Module [ mempools ] time: 0.2019s IPv4 Addresses : .. Module [ ipv4-addresses ] time: 0.1366s IPv6 Addresses : Module [ ipv6-addresses ] time: 0.5068s Sensors: ENTITY-SENSOR-MIB
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
This is pretty much NotOurProblem™ we have no control over the quality or stability of 3rd party vendor SNMP implementations.
It seems your devices reboot when some part of ENTITY-SENSOR-MIB is walked. This is definitely a vendor bug, and not really our responsibility.
We don't generate SNMP requests ourselves, we rely on the net-snmp binaries to do that, so there is nothing special we're doing with regards to the SNMP protocol, other than making use of bulkwalk, which many other applications aren't smart enough to so.
The most we can do is give you the ability to selectively enable/disable modules or MIBs so you can manually work around vendor bugs, and we already do that.
Really, if /all/ of your devices hit this bug, it should not be difficult to fix. If you run discovery with -d, you'll see the snmp command being executed and exactly which OID causes the device to crash. You can then send that to them when you ask them to fix it, because I'm pretty sure you send them even more money for their magical rebooting ONTs.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 16 December 2014 18:04:03 John Brown john@citylinkfiber.com wrote:
Hi,
We recently installed Observium in an effort to replace our aging Cacti and Nagios systems. We are running the latest code checked out of SVN (We paid the money to Adam for the Professional Version)
Today we added 180+ Adtran based devices to the system. All 180+ devices began rebooting. They would become stable for a few hours and then start rebooting again.
Important to note that most of these items have been in our network for 3+ years, are running recent code, and have been rock solid stable for their entire service life. Cacti and Nagios poll them every minute and have been doing so for the past 3 years.
Insert Observium and the devices start failing.
My initial feeling is that there is something in the way Observium does its SNMP.
Curious if anyone else has experienced this or something like this. Its caused us to halt putting more devices on Observium.
Devices are TA 5000 / TA 352 GPON ONT's, TA 9xx iad's
When I add a single device via
add_device <name>
and then do a
discover.php -h new
as soon as the code hits ENTITY-SENSOR-MIB the adtran device reboots
noc@observium:/home/observium# ./discovery.php -h new Observium 0.14.12.6107 Discovery
ont-03.sip.pon-1-6-1.fttx.XXX.YYYYY.net 38 adtran-aos (adtran-aos) Module [ os ] time: 0.0005s Ports : .... Module [ ports ] time: 0.1889s Port Stacks: Module [ ports-stack ] time: 0.0774s Processors : ADTRAN-AOSCPU + hrDevice: Module [ processors ] time: 0.2734s Memory : ADTRAN-AOSCPU + Module [ mempools ] time: 0.2019s IPv4 Addresses : .. Module [ ipv4-addresses ] time: 0.1366s IPv6 Addresses : Module [ ipv6-addresses ] time: 0.5068s Sensors: ENTITY-SENSOR-MIB
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (4)
-
Adam Armstrong -
John Brown -
Nikolay Shopik -
Tom Laermans