New subject: Possible bug with BGP IPv6 sessions state alert checker

5 Jan 2017

On 28/02/2017 15:11:41, Youssef BENGELLOUN - ZAHR
<ybzahr@prodware.fr> wrote:

Dear Community,

Latest update, this has been officially filed as a DEFECT and
addressed :

“The fix from engineering worked. I now correctly get returned “1”
when the peer is admin down and “2” when it isn’t (details below
if you are interested)

It will now be checked into each of the NI versions to be included
in the next patch. You can check for DEFECT000633962”

*********************
Admin Down

telnet@NetIron CER 2024F#sh ipv6 bgp summ
BGP4 Summary
Router ID: 10.2.24.1 Local AS Number: 13456
Confederation Identifier: not configured
Confederation Peers:
Maximum Number of IP ECMP Paths Supported for Load Sharing: 1
Number of Neighbors Configured: 1, UP: 0
Number of Routes Installed: 0
Number of Routes Advertising to All Neighbors: 0 (0 entries)
Number of Attribute Entries Installed: 0
'+': Data in InQueue '>': Data in OutQueue '-': Clearing
'*': Update Policy 'c': Group change 'p': Group change Pending
'r': Restarting 's': Stale '^': Up before Restart '<': eor=""
waiting="">
Neighbor Address AS# State Time Rt:Accepted Filtered Sent ToSend
cafe::1 12345 ADMDN 2h 0m18s 0 0 0 0

telnet@NetIron CER 2024F#Feb 27 14:55:19.089 : 172.29.198.48:
2046894425: SNMP-V2 request: Get PDU with 1 varbind(s):
bgp4V2PeerAdminStatus.1.2.16.202.254.0.0.0.0.0.0.0.0.0.0.0.0.0.2.2.16.202.254.0.0.0.0.0.0.0.0.0.0.0.0.0.1

Feb 27 14:55:19.090 : 172.29.198.48: 2046894425: SNMP response:
GetResponse PDU with 1 varbind(s):
bgp4V2PeerAdminStatus.1.2.16.202.254.0.0.0.0.0.0.0.0.0.0.0.0.0.2.2.16.202.254.0.0.0.0.0.0.0.0.0.0.0.0.0.1
= 1

Admin Up

telnet@NetIron CER 2024F#sh ipv6 bgp summ
BGP4 Summary
Router ID: 10.2.24.1 Local AS Number: 13456
Confederation Identifier: not configured
Confederation Peers:
Maximum Number of IP ECMP Paths Supported for Load Sharing: 1
Number of Neighbors Configured: 1, UP: 0
Number of Routes Installed: 0
Number of Routes Advertising to All Neighbors: 0 (0 entries)
Number of Attribute Entries Installed: 0
'+': Data in InQueue '>': Data in OutQueue '-': Clearing
'*': Update Policy 'c': Group change 'p': Group change Pending
'r': Restarting 's': Stale '^': Up before Restart '<': eor=""
waiting="">
Neighbor Address AS# State Time Rt:Accepted Filtered Sent ToSend
cafe::1 12345 CONN 2h 1m31s 0 0 0 0

telnet@NetIron CER 2024F#Feb 27 14:56:30.800 : 172.29.198.48:
2046894428: SNMP-V2 request: Get PDU with 1 varbind(s):
bgp4V2PeerAdminStatus.1.2.16.202.254.0.0.0.0.0.0.0.0.0.0.0.0.0.2.2.16.202.254.0.0.0.0.0.0.0.0.0.0.0.0.0.1

Feb 27 14:56:30.801 : 172.29.198.48: 2046894428: SNMP response:
GetResponse PDU with 1 varbind(s):
bgp4V2PeerAdminStatus.1.2.16.202.254.0.0.0.0.0.0.0.0.0.0.0.0.0.2.2.16.202.254.0.0.0.0.0.0.0.0.0.0.0.0.0.1
= 2

*********************

Best regards.

Le 14/02/2017 11:46, « Youssef BENGELLOUN - ZAHR » a écrit :

Dear community,

DEFECT confirmed but not filed yet :

« Just to let you know engineering have accepted this as a new
defect and are working on a fix. We will ask for this to be back
ported to the 5.7, 5.8 and 5.8 maintenance releases as well once
available. »

Now, it’s about waiting.

Best regards.

Le 10/02/2017 13:56, « Youssef BENGELLOUN - ZAHR » a écrit :

Dear community,

Just got an initial feedback from BTAC pointing to a DEFECT (very
likely) :

“I did a quick test on version 6 to check and I am still seeing
the response 2 (running when I have a peer ADMDN. So I have raised
an escalation case to get this checked as it looks like a defect.”

Now, it’s only a matter of time.

Best regards.

Le 10/02/2017 11:29, « Youssef BENGELLOUN - ZAHR » a écrit :

Dear Community,

Just to update you on this. I have created a case with BTAC and
initial findings confirms SNMP is indeed returning the wrong status.

As of today, no DEFECT matching this behavior has been found.

They are going to lab this, and revert back to me.

Best regards.

Le 06/01/2017 16:22, « observium au nom de Adam Armstrong » a écrit :

3 Rules of reporting stuff to us are :

#1 What is it reporting via SNMP?
#2 What is it reporting via SNMP?
#3 What is it reporting via SNMP?

adam.

On 2017-01-06 15:18, Youssef BENGELLOUN - ZAHR wrote:
> Dear adam,
>
> I’m no Observium expert, I don’t know how your product works. If
> this was actually explained somewhere, I would probably pushed
debug a
> bit further.
>
> Doesn’t matter anymore. I’m just happy we potentially found the
> issue.
>
> Thank you for your help anyways.
>
> Best regards.
>
> YOUSSEF BENGELLOUN - ZAHR - Consultant Expert
> Prodware France
> T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr
>
> -------------------------
>
> Web : prodware.fr [1]
>
> [2] [3] [4] [5] [6] [7]
>
> [8]
>
> DE : observium au nom de Adam
> Armstrong
> RÉPONDRE À : Observium Network Observation System
>
> DATE : vendredi 6 janvier 2017 16:06
> À : "observium@observium.org"
> OBJET : Re: [Observium] Possible bug with BGP IPv6 sessions state
> alert checker
>
> This looks like yet-another-shitty-vendor-bug.
>
> Returning "running" for a shutdown session is a vendor bug, and
should
> probably be addressed with them.
>
> There was never really any need to contact us about this at all.
You
> needed to verify what the device was exposing via SNMP and
contact the
> vendor directly.
>
> https://en.wikipedia.org/wiki/The_Boy_Who_Cried_Wolf
>
> adam.
>
>> On 06/01/2017 14:20:20, Youssef BENGELLOUN - ZAHR
>> wrote:
>>
>> Dear Adam,
>>
>> I don’t know, I didn’t write the code. Hence me asking.
>>
>> Could you please care to elaborate what “running” means and what
>> kind of information it recovers ?
>>
>> Still, I am confirming that on my devices, sessions are
>> administratively disabled :
>>
>> telnet@rr01-par01#sh ip bgp conf | i 2a00:bf40::5
>>
>> neighbor 2a00:bf40::5 peer-group ER-PEER-V6
>>
>> neighbor 2a00:bf40::5 shutdown
>>
>> telnet@rr01-par01#sh ip bgp conf | i 2a00:bf40::6
>>
>> neighbor 2a00:bf40::6 peer-group ER-PEER-V6
>>
>> neighbor 2a00:bf40::6 shutdown
>>
>> So, either this is some kind of bug on the brocade side (running
>> NetIron 5.7e) or something on the software side.
>>
>> I’m not pointing fingers here. I’m being open with you and
>> trying to move forward with this.
>>
>> Best regards.
>>
>> YOUSSEF BENGELLOUN - ZAHR - Consultant Expert
>> Prodware France
>> T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr
>>
>> -------------------------
>>
>> Web : prodware.fr [1]
>>
>> [2] [3] [4] [5] [6] [7]
>>
>> [8]
>>
>> DE : observium au nom de Adam
>> Armstrong
>> RÉPONDRE À : Observium Network Observation System
>>
>> DATE : vendredi 6 janvier 2017 15:03
>> À : "observium@observium.org"
>> OBJET : Re: [Observium] Possible bug with BGP IPv6 sessions state
>> alert checker
>>
>> Well then, why is your device returning "running" for a session
>> which is supposed to be administratively down?
>>
>> adam.
>>
>> On 06/01/2017 14:02:04, Youssef BENGELLOUN - ZAHR
>> wrote:
>>
>> There you go :
>>
>> You don’t need to be rude.
>>
>> Best regards.
>>
>> YOUSSEF BENGELLOUN - ZAHR - Consultant Expert
>> Prodware France
>> T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr
>>
>> -------------------------
>>
>> Web : prodware.fr [1]
>>
>> [2] [3] [4] [5] [6] [7]
>>
>> [8]
>>
>> DE : observium au nom de Adam
>> Armstrong
>> RÉPONDRE À : Observium Network Observation System
>>
>> DATE : vendredi 6 janvier 2017 14:55
>> À : "observium@observium.org"
>> OBJET : Re: [Observium] Possible bug with BGP IPv6 sessions state
>> alert checker
>>
>> jesus fuck.
>>
>> Does that look anything like the screenshot I posted?
>>
>> adam.
>>
>> On 06/01/2017 13:47:39, Youssef BENGELLOUN - ZAHR
>> wrote:
>>
>> Here it is :
>>
>> It’s exactly the same output for both BGP sessions from two
>> different RR routers (4 in total).
>>
>> 9 days ago is when we upgraded the latest stable release 8292 with
>> support of BGPv6 sessions. That’s when the issue appeared.
>>
>> Best regards.
>>
>> YOUSSEF BENGELLOUN - ZAHR - Consultant Expert
>> Prodware France
>> T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr
>>
>> -------------------------
>>
>> Web : prodware.fr [1]
>>
>> [2] [3] [4] [5] [6] [7]
>>
>> [8]
>>
>> DE : observium au nom de Adam
>> Armstrong
>> RÉPONDRE À : Observium Network Observation System
>>
>> DATE : vendredi 6 janvier 2017 14:33
>> À : "observium@observium.org"
>> OBJET : Re: [Observium] Possible bug with BGP IPv6 sessions state
>> alert checker
>>
>> None of this is useful. All that's useful is what the device is
>> returning in SNMP.
>>
>> You can see what the device returned in SNMP either in the poller,
>> or by putting your mouse over the (i) icon on an alert checker,
like
>> this :
>>
>>
>
http://omega.memetic.org/~adama/monosnap/Observium_Dev____-_Alerts_-_Google_Chrome_2017-01-06_13.31.17.png
>>
>>
>> The CLI is not SNMP. It's incredibly unhelpful to try to use
the CLI
>> as "proof" of anything, since on many platforms they will
frequently
>> disagree due to bugs or MIB standards.
>>
>> adam.
>>
>> On 06/01/2017 13:28:50, Youssef BENGELLOUN - ZAHR
>> wrote:
>>
>> Yes, it is :
>>
>> «IPv6 addresses DEAD:BEEF:CAFE::5 and DEAD:BEEF:CAFE::6 are
>> loopbacks used to be used by former PEs.
>>
>> If I look at the BGP sessions state on the routers themselves, you
>> can see that I have willingly administratively shutdown those
>> particular sessions :
>>
>> telnet@rr01-par01#sh ipv bgp summary
>>
>> Neighbor Address AS# State Time
>> Rt:Accepted Filtered Sent ToSend
>>
>> DEAD:BEEF:CAFE::2 XYZ ESTAB 366d 8h29m 2
>> 0 32985 0
>>
>> DEAD:BEEF:CAFE::3 XYZ ESTAB 366d 8h18m
>> 24074 0 8911 0
>>
>> DEAD:BEEF:CAFE::4 XYZ ESTAB 366d 8h16m
>> 1790 0 32360 0
>>
>> DEAD:BEEF:CAFE::5 XYZ ADMDN 366d12h37m 0
>> 0 0 0
>>
>> DEAD:BEEF:CAFE::6 XYZ ADMDN 366d12h37m 0
>> 0 0 0
>>
>> DEAD:BEEF:CAFE::7 XYZ CONN 357d12h10m 0
>> 0 0 32985
>>
>> DEAD:BEEF:CAFE::8 XYZ ESTAB 366d12h37m
>> 32453 0 24701 0
>>
>> Rest of alerts checks for IPv4 and IPv6 sessions are fine.»
>>
>> Are you expecting a different sort of feedbacks ? Maybe debug
traces
>> ?
>>
>> Thank you.
>>
>> YOUSSEF BENGELLOUN - ZAHR - Consultant Expert
>> Prodware France
>> T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr
>>
>> -------------------------
>>
>> Web : prodware.fr [1]
>>
>> [2] [3] [4] [5] [6] [7]
>>
>> [8]
>>
>> DE : observium au nom de Adam
>> Armstrong
>> RÉPONDRE À : Observium Network Observation System
>>
>> DATE : vendredi 6 janvier 2017 13:53
>> À : "observium@observium.org"
>> OBJET : Re: [Observium] Possible bug with BGP IPv6 sessions state
>> alert checker
>>
>> No, it's not in your initial email.
>>
>> adam.
>>
>> On 06/01/2017 12:52:15, Youssef BENGELLOUN - ZAHR
>> wrote:
>>
>> Dear Adam,
>>
>> It’s actually all in my initial email. Target devices aren’t
>> anymore, those PEs were unplugged.
>>
>> I kept the configuration in the RRs routers in order to save some
>> time when re-deploying in a near future. That’s how I discovered
>> the “issue”.
>>
>> Would like me to provide you with some debug traces ?
>>
>> Best regards.
>>
>> YOUSSEF BENGELLOUN - ZAHR - Consultant Expert
>> Prodware France
>> T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr
>>
>> -------------------------
>>
>> Web : prodware.fr [1]
>>
>> [2] [3] [4] [5] [6] [7]
>>
>> [8]
>>
>> DE : observium au nom de Adam
>> Armstrong
>> RÉPONDRE À : Observium Network Observation System
>>
>> DATE : vendredi 6 janvier 2017 12:51
>> À : "observium@observium.org"
>> OBJET : Re: [Observium] Possible bug with BGP IPv6 sessions state
>> alert checker
>>
>> You're not giving any actual useful information. What are those
>> devices actually returning?
>>
>> Put your mouse over the (i) icon.
>>
>> adam.
>>
>> On 06/01/2017 08:37:02, Youssef BENGELLOUN - ZAHR
>> wrote:
>>
>> Dear Adam,
>>
>> Actually, I checked a lot before sending the email. If I can avoid
>> noise, I will always choose to do so.
>>
>> My check is pretty simple : I would like to detect as errors BGP
>> sessions that are AdminEnabled (ie not shutdown) and that aren’t
>> in Established state. Theese could be problematic for any kind of
>> reason.
>>
>> For this particular example I gave, same check works just fine for
>> IPv4 for this old peers :
>>
>> This has been confirmed on two routeurs acting as RRs :
>>
>> telnet@rr01-par01#sh ip bgp summary
>>
>> Neighbor Address AS# State Time Rt:Accepted
>> Filtered Sent ToSend
>>
>> X.Y.Z.2 ABCDE ESTAB 369d 5h59m 3
>> 0 621434 1
>>
>> X.Y.Z.3 ABCDE ESTAB 369d 6h 4m 406952
>> 0 214471 12
>>
>> X.Y.Z.4 ABCDE ESTAB 369d 6h 2m 59865
>> 0 597987 1
>>
>> X.Y.Z.5 ABCDE ADMDN 369d10h23m 0
>> 0 0 0
>>
>> X.Y.Z.6 ABCDE ADMDN 369d10h23m 0
>> 0 0 0
>>
>> X.Y.Z.7 ABCDE ESTAB> 369d 5h59m 195213
>> 0 510924 1
>>
>> X.Y.Z.8 ABCDE ESTAB 369d 5h59m 397106
>> 0 540911 1
>>
>> But, for IPv6 counter-part, it’s not working.
>>
>> Maybe I’m actually missing something. If so, enlighten me with
>> your wisdom.
>>
>> Respectfully.
>>
>> YOUSSEF BENGELLOUN - ZAHR - Consultant Expert
>> Prodware France
>> T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr
>>
>> -------------------------
>>
>> Web : prodware.fr [1]
>>
>> [2] [3] [4] [5] [6] [7]
>>
>> [8]
>>
>> DE : observium au nom de Adam
>> Armstrong
>> RÉPONDRE À : Observium Network Observation System
>>
>> DATE : jeudi 5 janvier 2017 22:33
>> À : "observium@observium.org"
>> OBJET : Re: [Observium] Possible bug with BGP IPv6 sessions state
>> alert checker
>>
>> Does bgpPeerAdminStatus not equals "stop" and bgpPeerState not
>> equals "established"?
>>
>> What, you didn't check before emailing?
>>
>> Why not?
>>
>> adam.
>>
>> On 05/01/2017 20:54:49, Youssef BENGELLOUN - ZAHR
>> wrote:
>>
>> Dear all,
>>
>> I have configured a simple BGP checker alert based on the provided
>> example in observium’s website :
>>
>> This works just fine for IPv4, but something is clearly wrong with
>> IPv6 as some IPv6 BGP sessions show up in error.
>>
>> This is an example from two Brocade CER-RT routers acting BGP
route
>> reflectors for a bunch of PEs. They are showing the exact same
error
>> :
>>
>> IPv6 addresses DEAD:BEEF:CAFE::5 and DEAD:BEEF:CAFE::6 are
loopbacks
>> used to used by former PEs.
>>
>> If I look at the BGP sessions state on the routers themselves, you
>> can that I have willingly administratively shutdown the sessions :
>>
>> telnet@rr01-par01#sh ipv bgp summary
>>
>> Neighbor Address AS# State Time
>> Rt:Accepted Filtered Sent ToSend
>>
>> DEAD:BEEF:CAFE::2 XYZ ESTAB 366d 8h29m 2
>> 0 32985 0
>>
>> DEAD:BEEF:CAFE::3 XYZ ESTAB 366d 8h18m
>> 24074 0 8911 0
>>
>> DEAD:BEEF:CAFE::4 XYZ ESTAB 366d 8h16m
>> 1790 0 32360 0
>>
>> DEAD:BEEF:CAFE::5 XYZ ADMDN 366d12h37m 0
>> 0 0 0
>>
>> DEAD:BEEF:CAFE::6 XYZ ADMDN 366d12h37m 0
>> 0 0 0
>>
>> DEAD:BEEF:CAFE::7 XYZ CONN 357d12h10m 0
>> 0 0 32985
>>
>> DEAD:BEEF:CAFE::8 XYZ ESTAB 366d12h37m
>> 32453 0 24701 0
>>
>> Rest of alert checks for IPv4 and IPv6 sessions are fine.
>>
>> Any hints about this ? Any traces I could provide ?
>>
>> Best regards.
>>
>> YOUSSEF BENGELLOUN - ZAHR - Consultant Expert
>> Prodware France
>> T : +33 979 999 000 - F : +33 988 814 001 - ybzahr@prodware.fr
>>
>> -------------------------
>>
>> Web : prodware.fr [1]
>>
>> [2] [3] [4] [5] [6] [7]
>>
>> [8]
>
>
>
> Links:
> ------
> [1] http://www.prodware.fr
> [2] http://twitter.com/Prodware/
> [3] http://www.facebook.com/Prodware/
> [4] https://www.linkedin.com/company/prodwarefrance
> [5] https://www.youtube.com/c/ProdwareFrance
> [6] http://www.viadeo.com/fr/company/prodware
> [7] http://www.prodware.fr/social-network/
> [8] http://www.prodware.net/email/2017/wishes/fr.html
>

BENGELLOUN - ZAHR Youssef - Consultant Expert
Prodware France
T : +33 979 999 000 F : +33 988 814 001 - ybzahr@prodware.fr
Web : prodware.fr

_______________________________________________
> observium mailing list
> observium@observium.org
> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

Possible bug with BGP IPv6 sessions state alert checker

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Jason Lixfeld

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Links:

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Links:

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Michael

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

Youssef BENGELLOUN - ZAHR

tags (0)

participants (7)