Spent some time trying to get Observium to deal with MIBs as produced by LLDP (Link Layer Discovery Protocol) on a couple of diverse network equipment types. Running the latest CE edition 0.15.6.6430.
After initial meager success (only two boxes recognized and several L2 neighbours reported with wrong interfaces), a couple of changes in includes/discovery/discovery-protocols/lldp-mib.inc.php brought some sense into results.
There are essentially two problems there:
- some lldpRemoteSystemsData OIDs could not be correlated with an interface on the other side (for various reasons, see below);
- a fallback mechanism which tries to find a match on an interface's MAC address produces a wrong match, because typically all ports on larger switches have the same MAC address and the heuristics just picks the first match, which is almost always wrong. In case of multiple matches the matching on MAC should just fail, which at least would not produce a senseless match causing operators to scratch their heads and wonder who crossed the cables. The comment in lldp-mib.inc.php : // Find the device by MAC address, still matches multiple ports sometimes, we use the first one and hope we're lucky if very much misguided, as all port MAC addresses on a switch are typically the same, at least on all device types that I checked.
The first issue above needs more explanation. I found a couple of different reasons for a failed match, as follows.
Case 1:
$ snmpbulkwalk -v2c -c ... -M mibs -m LLDP-MIB ... lldpRemoteSystemsData LLDP-MIB::lldpRemChassisIdSubtype.5360466.52.2 = INTEGER: macAddress(4) LLDP-MIB::lldpRemChassisId.5360466.52.2 = Hex-STRING: C4 54 44 1E 71 4E LLDP-MIB::lldpRemPortIdSubtype.5360466.52.2 = INTEGER: local(7) LLDP-MIB::lldpRemPortId.5360466.52.2 = Hex-STRING: 30 2F 34 37 00 LLDP-MIB::lldpRemPortDesc.5360466.52.2 = STRING: "Slot: 0 Port: 47 10G - Level" LLDP-MIB::lldpRemSysName.5360466.52.2 = STRING: "<hidden>" LLDP-MIB::lldpRemSysDesc.5360466.52.2 = STRING: "T3048-LY2R L2/L3 Switch" LLDP-MIB::lldpRemSysCapSupported.5360466.52.2 = BITS: 28 00 bridge(2) router(4) LLDP-MIB::lldpRemSysCapEnabled.5360466.52.2 = BITS: 20 00 bridge(2) LLDP-MIB::lldpRemManAddrIfSubtype.5360466.52.2.6.6.196.84.68.30.113.78 = INTEGER: ifIndex(2) LLDP-MIB::lldpRemManAddrIfId.5360466.52.2.6.6.196.84.68.30.113.78 = INTEGER: 69 LLDP-MIB::lldpRemManAddrOID.5360466.52.2.6.6.196.84.68.30.113.78 = OID: iso.3.6.1.4.1.7244 LLDP-MIB::lldpRemOrgDefInfo.5360466.52.2.0.18.15.1.1 = Hex-STRING: 01 00 00 00 24 LLDP-MIB::lldpRemOrgDefInfo.5360466.52.2.0.18.15.2.1 = Hex-STRING: 00 00 00 LLDP-MIB::lldpRemOrgDefInfo.5360466.52.2.0.18.15.3.1 = Hex-STRING: 01 00 00 00 00 LLDP-MIB::lldpRemOrgDefInfo.5360466.52.2.0.18.15.4.1 = Hex-STRING: 23 F2 LLDP-MIB::lldpRemOrgDefInfo.5360466.52.2.0.128.194.1.1 = Hex-STRING: 00 01 LLDP-MIB::lldpRemOrgDefInfo.5360466.52.2.0.128.194.2.1 = Hex-STRING: 02 00 00 LLDP-MIB::lldpRemOrgDefInfo.5360466.52.2.0.128.194.3.1 = Hex-STRING: 00 01 00
Case 2: LLDP-MIB::lldpRemChassisIdSubtype.9511080.4.5 = INTEGER: macAddress(4) LLDP-MIB::lldpRemChassisIdSubtype.9511149.2.6 = INTEGER: macAddress(4) LLDP-MIB::lldpRemChassisIdSubtype.9511174.3.7 = INTEGER: macAddress(4) LLDP-MIB::lldpRemChassisIdSubtype.9511209.26.8 = INTEGER: macAddress(4) LLDP-MIB::lldpRemChassisIdSubtype.9587238.48.9 = INTEGER: macAddress(4) LLDP-MIB::lldpRemChassisIdSubtype.9588774.27.10 = INTEGER: macAddress(4) LLDP-MIB::lldpRemChassisIdSubtype.9588907.28.11 = INTEGER: macAddress(4) LLDP-MIB::lldpRemChassisIdSubtype.9592381.42.12 = INTEGER: macAddress(4) LLDP-MIB::lldpRemChassisIdSubtype.9827133.8.14 = INTEGER: macAddress(4) LLDP-MIB::lldpRemChassisIdSubtype.9843364.9.16 = INTEGER: macAddress(4) LLDP-MIB::lldpRemChassisId.9511080.4.5 = Hex-STRING: 00 04 96 34 CB 7D LLDP-MIB::lldpRemChassisId.9511149.2.6 = Hex-STRING: 00 04 96 34 CB 78 LLDP-MIB::lldpRemChassisId.9511174.3.7 = Hex-STRING: 00 04 96 34 CB 8D LLDP-MIB::lldpRemChassisId.9511209.26.8 = Hex-STRING: 00 04 96 34 CB 93 LLDP-MIB::lldpRemChassisId.9587238.48.9 = Hex-STRING: A4 8C DB 0C 4B 00 LLDP-MIB::lldpRemChassisId.9588774.27.10 = Hex-STRING: 00 04 96 34 CB 7F LLDP-MIB::lldpRemChassisId.9588907.28.11 = Hex-STRING: 00 04 96 34 CB 8B LLDP-MIB::lldpRemChassisId.9592381.42.12 = Hex-STRING: 44 AD D9 6C 2D 80 LLDP-MIB::lldpRemChassisId.9827133.8.14 = Hex-STRING: C4 54 44 1E 71 4E LLDP-MIB::lldpRemChassisId.9843364.9.16 = Hex-STRING: 2C 60 0C 33 8F FF LLDP-MIB::lldpRemPortIdSubtype.9511080.4.5 = INTEGER: interfaceName(5) LLDP-MIB::lldpRemPortIdSubtype.9511149.2.6 = INTEGER: interfaceName(5) LLDP-MIB::lldpRemPortIdSubtype.9511174.3.7 = INTEGER: interfaceName(5) LLDP-MIB::lldpRemPortIdSubtype.9511209.26.8 = INTEGER: interfaceName(5) LLDP-MIB::lldpRemPortIdSubtype.9587238.48.9 = INTEGER: local(7) LLDP-MIB::lldpRemPortIdSubtype.9588774.27.10 = INTEGER: interfaceName(5) LLDP-MIB::lldpRemPortIdSubtype.9588907.28.11 = INTEGER: interfaceName(5) LLDP-MIB::lldpRemPortIdSubtype.9592381.42.12 = INTEGER: interfaceName(5) LLDP-MIB::lldpRemPortIdSubtype.9827133.8.14 = INTEGER: interfaceAlias(1) LLDP-MIB::lldpRemPortIdSubtype.9843364.9.16 = INTEGER: interfaceAlias(1) LLDP-MIB::lldpRemPortId.9511080.4.5 = STRING: "25" LLDP-MIB::lldpRemPortId.9511149.2.6 = STRING: "25" LLDP-MIB::lldpRemPortId.9511174.3.7 = STRING: "25" LLDP-MIB::lldpRemPortId.9511209.26.8 = STRING: "25" LLDP-MIB::lldpRemPortId.9587238.48.9 = STRING: "50" LLDP-MIB::lldpRemPortId.9588774.27.10 = STRING: "26" LLDP-MIB::lldpRemPortId.9588907.28.11 = STRING: "26" LLDP-MIB::lldpRemPortId.9592381.42.12 = STRING: "Te1/1/2" LLDP-MIB::lldpRemPortId.9827133.8.14 = Hex-STRING: 4C 69 6E 6B 20 74 6F 20 43 4D 49 2C 20 6E 65 75 68 75 62 00 LLDP-MIB::lldpRemPortId.9843364.9.16 = Hex-STRING: 55 70 6C 69 6E 6B 20 74 6F 20 49 4A 53 2D 4E 45 54 00 LLDP-MIB::lldpRemPortDesc.9511080.4.5 = STRING: LLDP-MIB::lldpRemPortDesc.9511149.2.6 = STRING: LLDP-MIB::lldpRemPortDesc.9511174.3.7 = STRING: LLDP-MIB::lldpRemPortDesc.9511209.26.8 = STRING: LLDP-MIB::lldpRemPortDesc.9587238.48.9 = STRING: XGE2 LLDP-MIB::lldpRemPortDesc.9588774.27.10 = STRING: LLDP-MIB::lldpRemPortDesc.9588907.28.11 = STRING: LLDP-MIB::lldpRemPortDesc.9592381.42.12 = STRING: TenGigabitEthernet1/1/2 LLDP-MIB::lldpRemPortDesc.9827133.8.14 = STRING: Slot: 0 Port: 48 10G - Level LLDP-MIB::lldpRemPortDesc.9843364.9.16 = STRING: Slot: 0 Port: 48 10G - Level LLDP-MIB::lldpRemSysName.9511080.4.5 = STRING: <hidden1>.ijs.si LLDP-MIB::lldpRemSysName.9511149.2.6 = STRING: <hidden2>.ijs.si LLDP-MIB::lldpRemSysName.9511174.3.7 = STRING: <hidden3>.ijs.si LLDP-MIB::lldpRemSysName.9511209.26.8 = STRING: <hidden4>.ijs.si LLDP-MIB::lldpRemSysName.9587238.48.9 = STRING: <hidden5> LLDP-MIB::lldpRemSysName.9588774.27.10 = STRING: <hidden6> LLDP-MIB::lldpRemSysName.9588907.28.11 = STRING: <hidden7> LLDP-MIB::lldpRemSysName.9592381.42.12 = STRING: <hidden8> LLDP-MIB::lldpRemSysName.9827133.8.14 = STRING: <hidden9> LLDP-MIB::lldpRemSysName.9843364.9.16 = STRING: <hidden10> LLDP-MIB::lldpRemSysDesc.9511080.4.5 = STRING: ExtremeXOS (X450a-24t) ... LLDP-MIB::lldpRemSysDesc.9511149.2.6 = STRING: ExtremeXOS (X450a-24t) ... LLDP-MIB::lldpRemSysDesc.9511174.3.7 = STRING: ExtremeXOS (X450a-24t) ... LLDP-MIB::lldpRemSysDesc.9511209.26.8 = STRING: ExtremeXOS (X450a-24t) ... LLDP-MIB::lldpRemSysDesc.9587238.48.9 = STRING: Networking Operating System RackSwitch G7052, IBM Networking OS: version 7.6.5, Boot image: version 7.6.5 LLDP-MIB::lldpRemSysDesc.9588774.27.10 = STRING: ExtremeXOS (X450a-24t) ... LLDP-MIB::lldpRemSysDesc.9588907.28.11 = STRING: ExtremeXOS (X450a-24t) ... LLDP-MIB::lldpRemSysDesc.9592381.42.12 = STRING: Cisco IOS Software, Catalyst L3 Switch Software... LLDP-MIB::lldpRemSysDesc.9827133.8.14 = STRING: T3048-LY2R L2/L3 Switch LLDP-MIB::lldpRemSysDesc.9843364.9.16 = STRING: T3048-LY2R L2/L3 Switch [...]
In both cases 1 & 2 the lldpRemPortIdSubtype of 'interfaceAlias' and 'local' fell under the "default:" branch of a "switch ($lldp['lldpRemPortIdSubtype'])", instead of being treated just like a "case 'interfaceName':" branch. So my solution was to add "case 'interfaceAlias':" and "case 'local':" there;
switch ($lldp['lldpRemPortIdSubtype']) { case 'interfaceName': + case 'interfaceAlias': + case 'local':
A Case 3 is a bit nastier and is specific to our Summit Extreme X450a switches. The problem there is that the interface name (i.e. a port number) as reported in the LLDP MIB does not use the same naming/numbering convention as the IF-MIB and SQL: the LLDP MIB just uses a port number (1 .. 26), while the SQL field ports.ifName prefixes a port number with a chassis number, which is always a 1 in a non-stackable switch. Moreover the ifIndex uses a formula 1000*chassis + port number. On top of it the PortDesc in LLDP MIB is an empty string, so cannot be used to correlate with IF-MIB::ifDescr. To illustrate:
IF-MIB::ifIndex.1001 = INTEGER: 1001 IF-MIB::ifIndex.1002 = INTEGER: 1002 IF-MIB::ifIndex.1003 = INTEGER: 1003 [...] IF-MIB::ifIndex.1026 = INTEGER: 1026
IF-MIB::ifDescr.1001 = STRING: X450a-24t Port 1 IF-MIB::ifDescr.1002 = STRING: X450a-24t Port 2 IF-MIB::ifDescr.1003 = STRING: X450a-24t Port 3 [...] IF-MIB::ifDescr.1026 = STRING: X450a-24t Port 26
LLDP-MIB::lldpLocPortIdSubtype.1 = INTEGER: interfaceName(5) LLDP-MIB::lldpLocPortIdSubtype.2 = INTEGER: interfaceName(5) LLDP-MIB::lldpLocPortIdSubtype.3 = INTEGER: interfaceName(5) [...]
LLDP-MIB::lldpLocPortId.1 = STRING: "1" LLDP-MIB::lldpLocPortId.2 = STRING: "2" LLDP-MIB::lldpLocPortId.3 = STRING: "3" [...] LLDP-MIB::lldpLocPortId.26 = STRING: "26"
LLDP-MIB::lldpLocPortDesc.1 = STRING: LLDP-MIB::lldpLocPortDesc.2 = STRING: LLDP-MIB::lldpLocPortDesc.3 = STRING: [...] LLDP-MIB::lldpLocPortDesc.26 = STRING:
The corresponding SQL field ports.ifName looks like "1:26", i.e. the port number is prefixed by a chassis number and a colon.
So neither the SQL query on ifDescr ("") OR ifName ("26") can succeed, nor the query on ifIndex ("1026"). I ended up with a hackish workaround kludge in lldp-mib.inc.php :
case 'interfaceName': $remote_port_id = dbFetchCell("SELECT `port_id` FROM `ports` WHERE (`ifDescr`= ? OR `ifName`= ?) AND `device_id` = ?",array($if, $id, $remote_device_id)); + if (!$remote_port_id && is_numeric($id) && $id < 1000) { + $remote_port_id = dbFetchCell("SELECT `port_id` FROM `ports` WHERE (`ifIndex`= ? OR `ifName`= ?) AND `device_id` = ?", array(1000+$id, "1:$id", $remote_device_id)); + }
Actually there is also a Case 4, where lldpRemSysName shows a string "(none).(none)" instead of an empty string, when the (optional) system name is not configured. To fix:
- if (empty($lldp['lldpRemSysName']) && $lldp['lldpRemChassisIdSubtype'] == 'macAddress') + if ((empty($lldp['lldpRemSysName']) || $lldp['lldpRemSysName'] == '(none).(none)') && $lldp['lldpRemChassisIdSubtype'] == 'macAddress')
The affected devices were: Summit Extreme X450 switches (Case #3), a Quanta LY2R switch (the 'interfaceAlias' Case #2), and a Networking Operating System RackSwitch G7052 (the 'local', Case #1). Not sure what switch was connected there in Case 4, possibly a smaller Quanta, identifies as "armv5tejl/Linux 2.6.17.WB_WPCM450.1.3".
The full patch is attached (I hope it survives the mailing list). If more information is needed I'd be gladly providing.
Regards Mark