Dear adam,

I’m no Observium expert, I don’t know how your product works. If this was actually explained somewhere, I would probably pushed debug a bit further.

Doesn’t matter anymore. I’m just happy we potentially found the issue.

Thank you for your help anyways.

Best regards.

De : observium <observium-bounces@observium.org> au nom de Adam Armstrong <adama@memetic.org>
Répondre à : Observium Network Observation System <observium@observium.org>
Date : vendredi 6 janvier 2017 16:06
À : "observium@observium.org" <observium@observium.org>
Objet : Re: [Observium] Possible bug with BGP IPv6 sessions state alert checker

This looks like yet-another-shitty-vendor-bug.

Returning "running" for a shutdown session is a vendor bug, and should probably be addressed with them.

There was never really any need to contact us about this at all. You needed to verify what the device was exposing via SNMP and contact the vendor directly.

https://en.wikipedia.org/wiki/The_Boy_Who_Cried_Wolf

adam.

On 06/01/2017 14:20:20, Youssef BENGELLOUN - ZAHR <ybzahr@prodware.fr> wrote:

Dear Adam,

I don’t know, I didn’t write the code. Hence me asking.

Could you please care to elaborate what “running” means and what kind of information it recovers ?

Still, I am confirming that on my devices, sessions are administratively disabled :

telnet@rr01-par01#sh ip bgp conf | i 2a00:bf40::5

neighbor 2a00:bf40::5 peer-group ER-PEER-V6

neighbor 2a00:bf40::5 shutdown

telnet@rr01-par01#sh ip bgp conf | i 2a00:bf40::6

neighbor 2a00:bf40::6 peer-group ER-PEER-V6

neighbor 2a00:bf40::6 shutdown

So, either this is some kind of bug on the brocade side (running NetIron 5.7e) or something on the software side.

I’m not pointing fingers here. I’m being open with you and trying to move forward with this.

Best regards.

Youssef BENGELLOUN - ZAHR - Consultant Expert
Prodware France
T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr

Web : prodware.fr

De : observium <observium-bounces@observium.org> au nom de Adam Armstrong <adama@memetic.org>
Répondre à : Observium Network Observation System <observium@observium.org>
Date : vendredi 6 janvier 2017 15:03
À : "observium@observium.org" <observium@observium.org>
Objet : Re: [Observium] Possible bug with BGP IPv6 sessions state alert checker

Well then, why is your device returning "running" for a session which is supposed to be administratively down?

adam.

On 06/01/2017 14:02:04, Youssef BENGELLOUN - ZAHR <ybzahr@prodware.fr> wrote:

There you go :

You don’t need to be rude.

Best regards.

Youssef BENGELLOUN - ZAHR - Consultant Expert
Prodware France
T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr

Web : prodware.fr

De : observium <observium-bounces@observium.org> au nom de Adam Armstrong <adama@memetic.org>
Répondre à : Observium Network Observation System <observium@observium.org>
Date : vendredi 6 janvier 2017 14:55
À : "observium@observium.org" <observium@observium.org>
Objet : Re: [Observium] Possible bug with BGP IPv6 sessions state alert checker

jesus fuck.

Does that look anything like the screenshot I posted?

adam.

On 06/01/2017 13:47:39, Youssef BENGELLOUN - ZAHR <ybzahr@prodware.fr> wrote:

Here it is :

It’s exactly the same output for both BGP sessions from two different RR routers (4 in total).

9 days ago is when we upgraded the latest stable release 8292 with support of BGPv6 sessions. That’s when the issue appeared.

Best regards.

Youssef BENGELLOUN - ZAHR - Consultant Expert
Prodware France
T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr

Web : prodware.fr

De : observium <observium-bounces@observium.org> au nom de Adam Armstrong <adama@memetic.org>
Répondre à : Observium Network Observation System <observium@observium.org>
Date : vendredi 6 janvier 2017 14:33
À : "observium@observium.org" <observium@observium.org>
Objet : Re: [Observium] Possible bug with BGP IPv6 sessions state alert checker

None of this is useful. All that's useful is what the device is returning in SNMP.

You can see what the device returned in SNMP either in the poller, or by putting your mouse over the (i) icon on an alert checker, like this :

http://omega.memetic.org/~adama/monosnap/Observium_Dev____-_Alerts_-_Google_Chrome_2017-01-06_13.31.17.png

The CLI is not SNMP. It's incredibly unhelpful to try to use the CLI as "proof" of anything, since on many platforms they will frequently disagree due to bugs or MIB standards.

adam.

On 06/01/2017 13:28:50, Youssef BENGELLOUN - ZAHR <ybzahr@prodware.fr> wrote:

Yes, it is :

«IPv6 addresses DEAD:BEEF:CAFE::5 and DEAD:BEEF:CAFE::6 are loopbacks used to be used by former PEs.

If I look at the BGP sessions state on the routers themselves, you can see that I have willingly administratively shutdown those particular sessions :

telnet@rr01-par01#sh ipv bgp summary

Neighbor Address AS# State Time Rt:Accepted Filtered Sent ToSend

DEAD:BEEF:CAFE::2 XYZ ESTAB 366d 8h29m 2 0 32985 0

DEAD:BEEF:CAFE::3 XYZ ESTAB 366d 8h18m 24074 0 8911 0

DEAD:BEEF:CAFE::4 XYZ ESTAB 366d 8h16m 1790 0 32360 0

DEAD:BEEF:CAFE::5 XYZ ADMDN 366d12h37m 0 0 0 0

DEAD:BEEF:CAFE::6 XYZ ADMDN 366d12h37m 0 0 0 0

DEAD:BEEF:CAFE::7 XYZ CONN 357d12h10m 0 0 0 32985

DEAD:BEEF:CAFE::8 XYZ ESTAB 366d12h37m 32453 0 24701 0

Rest of alerts checks for IPv4 and IPv6 sessions are fine.»

Are you expecting a different sort of feedbacks ? Maybe debug traces ?

Thank you.

Youssef BENGELLOUN - ZAHR - Consultant Expert
Prodware France
T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr

Web : prodware.fr

De : observium <observium-bounces@observium.org> au nom de Adam Armstrong <adama@memetic.org>
Répondre à : Observium Network Observation System <observium@observium.org>
Date : vendredi 6 janvier 2017 13:53
À : "observium@observium.org" <observium@observium.org>
Objet : Re: [Observium] Possible bug with BGP IPv6 sessions state alert checker

No, it's not in your initial email.

adam.

On 06/01/2017 12:52:15, Youssef BENGELLOUN - ZAHR <ybzahr@prodware.fr> wrote:

Dear Adam,

It’s actually all in my initial email. Target devices aren’t anymore, those PEs were unplugged.

I kept the configuration in the RRs routers in order to save some time when re-deploying in a near future. That’s how I discovered the “issue”.

Would like me to provide you with some debug traces ?

Best regards.

Youssef BENGELLOUN - ZAHR - Consultant Expert
Prodware France
T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr

Web : prodware.fr

De : observium <observium-bounces@observium.org> au nom de Adam Armstrong <adama@memetic.org>
Répondre à : Observium Network Observation System <observium@observium.org>
Date : vendredi 6 janvier 2017 12:51
À : "observium@observium.org" <observium@observium.org>
Objet : Re: [Observium] Possible bug with BGP IPv6 sessions state alert checker

You're not giving any actual useful information. What are those devices actually returning?

Put your mouse over the (i) icon.

adam.

On 06/01/2017 08:37:02, Youssef BENGELLOUN - ZAHR <ybzahr@prodware.fr> wrote:

Dear Adam,

Actually, I checked a lot before sending the email. If I can avoid noise, I will always choose to do so.

My check is pretty simple : I would like to detect as errors BGP sessions that are AdminEnabled (ie not shutdown) and that aren’t in Established state. Theese could be problematic for any kind of reason.

For this particular example I gave, same check works just fine for IPv4 for this old peers :

This has been confirmed on two routeurs acting as RRs :

telnet@rr01-par01#sh ip bgp summary

Neighbor Address AS# State Time Rt:Accepted Filtered Sent ToSend

           X.Y.Z.2 ABCDE ESTAB 369d 5h59m 3 0 621434 1

            X.Y.Z.3 ABCDE ESTAB 369d 6h 4m 406952 0 214471 12

           X.Y.Z.4 ABCDE ESTAB 369d 6h 2m 59865 0 597987 1

            X.Y.Z.5 ABCDE ADMDN 369d10h23m 0 0 0 0

            X.Y.Z.6 ABCDE ADMDN 369d10h23m 0 0 0 0

            X.Y.Z.7 ABCDE ESTAB> 369d 5h59m 195213 0 510924 1

            X.Y.Z.8 ABCDE ESTAB 369d 5h59m 397106 0 540911 1

But, for IPv6 counter-part, it’s not working.

Maybe I’m actually missing something. If so, enlighten me with your wisdom.

Respectfully.

Youssef BENGELLOUN - ZAHR - Consultant Expert
Prodware France
T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr

Web : prodware.fr

De : observium <observium-bounces@observium.org> au nom de Adam Armstrong <adama@memetic.org>
Répondre à : Observium Network Observation System <observium@observium.org>
Date : jeudi 5 janvier 2017 22:33
À : "observium@observium.org" <observium@observium.org>
Objet : Re: [Observium] Possible bug with BGP IPv6 sessions state alert checker

Does bgpPeerAdminStatus not equals "stop" and bgpPeerState not equals "established"?

What, you didn't check before emailing?

Why not?

adam.

On 05/01/2017 20:54:49, Youssef BENGELLOUN - ZAHR <ybzahr@prodware.fr> wrote:

Dear all,

I have configured a simple BGP checker alert based on the provided example in observium’s website :

This works just fine for IPv4, but something is clearly wrong with IPv6 as some IPv6 BGP sessions show up in error.

This is an example from two Brocade CER-RT routers acting BGP route reflectors for a bunch of PEs. They are showing the exact same error :

IPv6 addresses DEAD:BEEF:CAFE::5 and DEAD:BEEF:CAFE::6 are loopbacks used to used by former PEs.

If I look at the BGP sessions state on the routers themselves, you can that I have willingly administratively shutdown the sessions :

telnet@rr01-par01#sh ipv bgp summary

Neighbor Address AS# State Time Rt:Accepted Filtered Sent ToSend

DEAD:BEEF:CAFE::2 XYZ ESTAB 366d 8h29m 2 0 32985 0

DEAD:BEEF:CAFE::3 XYZ ESTAB 366d 8h18m 24074 0 8911 0

DEAD:BEEF:CAFE::4 XYZ ESTAB 366d 8h16m 1790 0 32360 0

DEAD:BEEF:CAFE::5 XYZ ADMDN 366d12h37m 0 0 0 0

DEAD:BEEF:CAFE::6 XYZ ADMDN 366d12h37m 0 0 0 0

DEAD:BEEF:CAFE::7 XYZ CONN 357d12h10m 0 0 0 32985

DEAD:BEEF:CAFE::8 XYZ ESTAB 366d12h37m 32453 0 24701 0

Rest of alert checks for IPv4 and IPv6 sessions are fine.

Any hints about this ? Any traces I could provide ?

Best regards.

Youssef BENGELLOUN - ZAHR - Consultant Expert
Prodware France
T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr

Web : prodware.fr