On Tue, Oct 27, 2015 at 4:25 AM, Mark Martinec <Mark.Martinec+observium@ijs.si> wrote:

Spent some time trying to get Observium to deal with MIBs as produced
by LLDP (Link Layer Discovery Protocol) on a couple of diverse
network equipment types. Running the latest CE edition 0.15.6.6430.

After initial meager success (only two boxes recognized and
several L2 neighbours reported with wrong interfaces), a couple of
changes in includes/discovery/discovery-protocols/lldp-mib.inc.php
brought some sense into results.

There are essentially two problems there:

- some lldpRemoteSystemsData OIDs could not be correlated with
an interface on the other side (for various reasons, see below);

- a fallback mechanism which tries to find a match on an
interface's MAC address produces a wrong match, because
typically all ports on larger switches have the same MAC address
and the heuristics just picks the first match, which is almost
always wrong. In case of multiple matches the matching on MAC
should just fail, which at least would not produce a senseless
match causing operators to scratch their heads and wonder who
crossed the cables.
The comment in lldp-mib.inc.php :
// Find the device by MAC address, still matches multiple ports
sometimes, we use the first one and hope we're lucky
if very much misguided, as all port MAC addresses on a switch are
typically the same, at least on all device types that I checked.

The first issue above needs more explanation. I found a couple of
different reasons for a failed match, as follows.

Case 1:

$ snmpbulkwalk -v2c -c ... -M mibs -m LLDP-MIB ... lldpRemoteSystemsData
LLDP-MIB::lldpRemChassisIdSubtype.5360466.52.2 = INTEGER: macAddress(4)
LLDP-MIB::lldpRemChassisId.5360466.52.2 = Hex-STRING: C4 54 44 1E 71 4E
LLDP-MIB::lldpRemPortIdSubtype.5360466.52.2 = INTEGER: local(7)
LLDP-MIB::lldpRemPortId.5360466.52.2 = Hex-STRING: 30 2F 34 37 00
LLDP-MIB::lldpRemPortDesc.5360466.52.2 = STRING: "Slot: 0 Port: 47 10G - Level"
LLDP-MIB::lldpRemSysName.5360466.52.2 = STRING: "<hidden>"
LLDP-MIB::lldpRemSysDesc.5360466.52.2 = STRING: "T3048-LY2R L2/L3 Switch"
LLDP-MIB::lldpRemSysCapSupported.5360466.52.2 = BITS: 28 00 bridge(2) router(4)
LLDP-MIB::lldpRemSysCapEnabled.5360466.52.2 = BITS: 20 00 bridge(2)
LLDP-MIB::lldpRemManAddrIfSubtype.5360466.52.2.6.6.196.84.68.30.113.78 = INTEGER: ifIndex(2)
LLDP-MIB::lldpRemManAddrIfId.5360466.52.2.6.6.196.84.68.30.113.78 = INTEGER: 69
LLDP-MIB::lldpRemManAddrOID.5360466.52.2.6.6.196.84.68.30.113.78 = OID: iso.3.6.1.4.1.7244
LLDP-MIB::lldpRemOrgDefInfo.5360466.52.2.0.18.15.1.1 = Hex-STRING: 01 00 00 00 24
LLDP-MIB::lldpRemOrgDefInfo.5360466.52.2.0.18.15.2.1 = Hex-STRING: 00 00 00
LLDP-MIB::lldpRemOrgDefInfo.5360466.52.2.0.18.15.3.1 = Hex-STRING: 01 00 00 00 00
LLDP-MIB::lldpRemOrgDefInfo.5360466.52.2.0.18.15.4.1 = Hex-STRING: 23 F2
LLDP-MIB::lldpRemOrgDefInfo.5360466.52.2.0.128.194.1.1 = Hex-STRING: 00 01
LLDP-MIB::lldpRemOrgDefInfo.5360466.52.2.0.128.194.2.1 = Hex-STRING: 02 00 00
LLDP-MIB::lldpRemOrgDefInfo.5360466.52.2.0.128.194.3.1 = Hex-STRING: 00 01 00

Case 2:
LLDP-MIB::lldpRemChassisIdSubtype.9511080.4.5 = INTEGER: macAddress(4)
LLDP-MIB::lldpRemChassisIdSubtype.9511149.2.6 = INTEGER: macAddress(4)
LLDP-MIB::lldpRemChassisIdSubtype.9511174.3.7 = INTEGER: macAddress(4)
LLDP-MIB::lldpRemChassisIdSubtype.9511209.26.8 = INTEGER: macAddress(4)
LLDP-MIB::lldpRemChassisIdSubtype.9587238.48.9 = INTEGER: macAddress(4)
LLDP-MIB::lldpRemChassisIdSubtype.9588774.27.10 = INTEGER: macAddress(4)
LLDP-MIB::lldpRemChassisIdSubtype.9588907.28.11 = INTEGER: macAddress(4)
LLDP-MIB::lldpRemChassisIdSubtype.9592381.42.12 = INTEGER: macAddress(4)
LLDP-MIB::lldpRemChassisIdSubtype.9827133.8.14 = INTEGER: macAddress(4)
LLDP-MIB::lldpRemChassisIdSubtype.9843364.9.16 = INTEGER: macAddress(4)
LLDP-MIB::lldpRemChassisId.9511080.4.5 = Hex-STRING: 00 04 96 34 CB 7D
LLDP-MIB::lldpRemChassisId.9511149.2.6 = Hex-STRING: 00 04 96 34 CB 78
LLDP-MIB::lldpRemChassisId.9511174.3.7 = Hex-STRING: 00 04 96 34 CB 8D
LLDP-MIB::lldpRemChassisId.9511209.26.8 = Hex-STRING: 00 04 96 34 CB 93
LLDP-MIB::lldpRemChassisId.9587238.48.9 = Hex-STRING: A4 8C DB 0C 4B 00
LLDP-MIB::lldpRemChassisId.9588774.27.10 = Hex-STRING: 00 04 96 34 CB 7F
LLDP-MIB::lldpRemChassisId.9588907.28.11 = Hex-STRING: 00 04 96 34 CB 8B
LLDP-MIB::lldpRemChassisId.9592381.42.12 = Hex-STRING: 44 AD D9 6C 2D 80
LLDP-MIB::lldpRemChassisId.9827133.8.14 = Hex-STRING: C4 54 44 1E 71 4E
LLDP-MIB::lldpRemChassisId.9843364.9.16 = Hex-STRING: 2C 60 0C 33 8F FF
LLDP-MIB::lldpRemPortIdSubtype.9511080.4.5 = INTEGER: interfaceName(5)
LLDP-MIB::lldpRemPortIdSubtype.9511149.2.6 = INTEGER: interfaceName(5)
LLDP-MIB::lldpRemPortIdSubtype.9511174.3.7 = INTEGER: interfaceName(5)
LLDP-MIB::lldpRemPortIdSubtype.9511209.26.8 = INTEGER: interfaceName(5)
LLDP-MIB::lldpRemPortIdSubtype.9587238.48.9 = INTEGER: local(7)
LLDP-MIB::lldpRemPortIdSubtype.9588774.27.10 = INTEGER: interfaceName(5)
LLDP-MIB::lldpRemPortIdSubtype.9588907.28.11 = INTEGER: interfaceName(5)
LLDP-MIB::lldpRemPortIdSubtype.9592381.42.12 = INTEGER: interfaceName(5)
LLDP-MIB::lldpRemPortIdSubtype.9827133.8.14 = INTEGER: interfaceAlias(1)
LLDP-MIB::lldpRemPortIdSubtype.9843364.9.16 = INTEGER: interfaceAlias(1)
LLDP-MIB::lldpRemPortId.9511080.4.5 = STRING: "25"
LLDP-MIB::lldpRemPortId.9511149.2.6 = STRING: "25"
LLDP-MIB::lldpRemPortId.9511174.3.7 = STRING: "25"
LLDP-MIB::lldpRemPortId.9511209.26.8 = STRING: "25"
LLDP-MIB::lldpRemPortId.9587238.48.9 = STRING: "50"
LLDP-MIB::lldpRemPortId.9588774.27.10 = STRING: "26"
LLDP-MIB::lldpRemPortId.9588907.28.11 = STRING: "26"
LLDP-MIB::lldpRemPortId.9592381.42.12 = STRING: "Te1/1/2"
LLDP-MIB::lldpRemPortId.9827133.8.14 = Hex-STRING: 4C 69 6E 6B 20 74 6F 20 43 4D 49 2C 20 6E 65 75
68 75 62 00
LLDP-MIB::lldpRemPortId.9843364.9.16 = Hex-STRING: 55 70 6C 69 6E 6B 20 74 6F 20 49 4A 53 2D 4E 45
54 00
LLDP-MIB::lldpRemPortDesc.9511080.4.5 = STRING:
LLDP-MIB::lldpRemPortDesc.9511149.2.6 = STRING:
LLDP-MIB::lldpRemPortDesc.9511174.3.7 = STRING:
LLDP-MIB::lldpRemPortDesc.9511209.26.8 = STRING:
LLDP-MIB::lldpRemPortDesc.9587238.48.9 = STRING: XGE2
LLDP-MIB::lldpRemPortDesc.9588774.27.10 = STRING:
LLDP-MIB::lldpRemPortDesc.9588907.28.11 = STRING:
LLDP-MIB::lldpRemPortDesc.9592381.42.12 = STRING: TenGigabitEthernet1/1/2
LLDP-MIB::lldpRemPortDesc.9827133.8.14 = STRING: Slot: 0 Port: 48 10G - Level
LLDP-MIB::lldpRemPortDesc.9843364.9.16 = STRING: Slot: 0 Port: 48 10G - Level
LLDP-MIB::lldpRemSysName.9511080.4.5 = STRING: <hidden1>.ijs.si
LLDP-MIB::lldpRemSysName.9511149.2.6 = STRING: <hidden2>.ijs.si
LLDP-MIB::lldpRemSysName.9511174.3.7 = STRING: <hidden3>.ijs.si
LLDP-MIB::lldpRemSysName.9511209.26.8 = STRING: <hidden4>.ijs.si
LLDP-MIB::lldpRemSysName.9587238.48.9 = STRING: <hidden5>
LLDP-MIB::lldpRemSysName.9588774.27.10 = STRING: <hidden6>
LLDP-MIB::lldpRemSysName.9588907.28.11 = STRING: <hidden7>
LLDP-MIB::lldpRemSysName.9592381.42.12 = STRING: <hidden8>
LLDP-MIB::lldpRemSysName.9827133.8.14 = STRING: <hidden9>
LLDP-MIB::lldpRemSysName.9843364.9.16 = STRING: <hidden10>
LLDP-MIB::lldpRemSysDesc.9511080.4.5 = STRING: ExtremeXOS (X450a-24t) ...
LLDP-MIB::lldpRemSysDesc.9511149.2.6 = STRING: ExtremeXOS (X450a-24t) ...
LLDP-MIB::lldpRemSysDesc.9511174.3.7 = STRING: ExtremeXOS (X450a-24t) ...
LLDP-MIB::lldpRemSysDesc.9511209.26.8 = STRING: ExtremeXOS (X450a-24t) ...
LLDP-MIB::lldpRemSysDesc.9587238.48.9 = STRING: Networking Operating System RackSwitch G7052,
IBM Networking OS: version 7.6.5, Boot image: version 7.6.5
LLDP-MIB::lldpRemSysDesc.9588774.27.10 = STRING: ExtremeXOS (X450a-24t) ...
LLDP-MIB::lldpRemSysDesc.9588907.28.11 = STRING: ExtremeXOS (X450a-24t) ...
LLDP-MIB::lldpRemSysDesc.9592381.42.12 = STRING: Cisco IOS Software, Catalyst L3 Switch Software...
LLDP-MIB::lldpRemSysDesc.9827133.8.14 = STRING: T3048-LY2R L2/L3 Switch
LLDP-MIB::lldpRemSysDesc.9843364.9.16 = STRING: T3048-LY2R L2/L3 Switch
[...]

In both cases 1 & 2 the lldpRemPortIdSubtype of 'interfaceAlias' and 'local'
fell under the "default:" branch of a "switch ($lldp['lldpRemPortIdSubtype'])",
instead of being treated just like a "case 'interfaceName':" branch. So my
solution was to add "case 'interfaceAlias':" and "case 'local':" there;

switch ($lldp['lldpRemPortIdSubtype'])
{
case 'interfaceName':
+ case 'interfaceAlias':
+ case 'local':

A Case 3 is a bit nastier and is specific to our Summit Extreme X450a
switches. The problem there is that the interface name (i.e. a port number)
as reported in the LLDP MIB does not use the same naming/numbering
convention as the IF-MIB and SQL: the LLDP MIB just uses a port number
(1 .. 26), while the SQL field ports.ifName prefixes a port number
with a chassis number, which is always a 1 in a non-stackable switch.
Moreover the ifIndex uses a formula 1000*chassis + port number.
On top of it the PortDesc in LLDP MIB is an empty string, so cannot
be used to correlate with IF-MIB::ifDescr. To illustrate:

IF-MIB::ifIndex.1001 = INTEGER: 1001
IF-MIB::ifIndex.1002 = INTEGER: 1002
IF-MIB::ifIndex.1003 = INTEGER: 1003
[...]
IF-MIB::ifIndex.1026 = INTEGER: 1026

IF-MIB::ifDescr.1001 = STRING: X450a-24t Port 1
IF-MIB::ifDescr.1002 = STRING: X450a-24t Port 2
IF-MIB::ifDescr.1003 = STRING: X450a-24t Port 3
[...]
IF-MIB::ifDescr.1026 = STRING: X450a-24t Port 26

LLDP-MIB::lldpLocPortIdSubtype.1 = INTEGER: interfaceName(5)
LLDP-MIB::lldpLocPortIdSubtype.2 = INTEGER: interfaceName(5)
LLDP-MIB::lldpLocPortIdSubtype.3 = INTEGER: interfaceName(5)
[...]

LLDP-MIB::lldpLocPortId.1 = STRING: "1"
LLDP-MIB::lldpLocPortId.2 = STRING: "2"
LLDP-MIB::lldpLocPortId.3 = STRING: "3"
[...]
LLDP-MIB::lldpLocPortId.26 = STRING: "26"

LLDP-MIB::lldpLocPortDesc.1 = STRING:
LLDP-MIB::lldpLocPortDesc.2 = STRING:
LLDP-MIB::lldpLocPortDesc.3 = STRING:
[...]
LLDP-MIB::lldpLocPortDesc.26 = STRING:

The corresponding SQL field ports.ifName looks like "1:26", i.e. the
port number is prefixed by a chassis number and a colon.

So neither the SQL query on ifDescr ("") OR ifName ("26") can
succeed, nor the query on ifIndex ("1026"). I ended up with a
hackish workaround kludge in lldp-mib.inc.php :

case 'interfaceName':
$remote_port_id = dbFetchCell("SELECT `port_id` FROM `ports` WHERE (`ifDescr`= ? OR `ifName`= ?) AND `device_id` = ?",array($if, $id, $remote_device_id));
+ if (!$remote_port_id && is_numeric($id) && $id < 1000) {
+ $remote_port_id = dbFetchCell("SELECT `port_id` FROM `ports` WHERE (`ifIndex`= ? OR `ifName`= ?) AND `device_id` = ?", array(1000+$id, "1:$id", $remote_device_id));
+ }

Actually there is also a Case 4, where lldpRemSysName shows
a string "(none).(none)" instead of an empty string, when the
(optional) system name is not configured. To fix:

- if (empty($lldp['lldpRemSysName']) && $lldp['lldpRemChassisIdSubtype'] == 'macAddress')
+ if ((empty($lldp['lldpRemSysName']) || $lldp['lldpRemSysName'] == '(none).(none)') && $lldp['lldpRemChassisIdSubtype'] == 'macAddress')

The affected devices were: Summit Extreme X450 switches (Case #3),
a Quanta LY2R switch (the 'interfaceAlias' Case #2), and a
Networking Operating System RackSwitch G7052 (the 'local', Case #1).
Not sure what switch was connected there in Case 4, possibly a
smaller Quanta, identifies as "armv5tejl/Linux 2.6.17.WB_WPCM450.1.3".

The full patch is attached (I hope it survives the mailing list).
If more information is needed I'd be gladly providing.

Regards
Mark

_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium