Hello,
This morning 2 XR devices randomly flagged on observium as rebooted. Both devices asr9001 running XR 4.3.4 have 2 week uptimes and no events this morning.
Thanks,
Charlie Burns
Hi there.
I've seen this issue too, after doing a Discovery. It flag's some of my devices as if they have been rebooted.
When I connect to the devices and inspect the uptime I do not see that they have been rebooted.
Strange I think.
Regards, Søren
Fra: observium [mailto:observium-bounces@observium.org] På vegne af Charlie Burns Sendt: 28. april 2014 17:46 Til: observium@observium.org Emne: [Observium] device reboot
Hello,
This morning 2 XR devices randomly flagged on observium as rebooted. Both devices asr9001 running XR 4.3.4 have 2 week uptimes and no events this morning.
Thanks,
Charlie Burns
Discovery doesn't have anything to do with uptime at all.
This sounds like the poller process isn't receiving the correct uptime when the discovery process is running, probably due to some network congestion somewhere.
Probably firewall session related.
adam.
On 2014-04-28 11:55, Søren Friis Rosiak wrote:
Hi there.
I've seen this issue too, after doing a Discovery. It flag's some of my devices as if they have been rebooted.
When I connect to the devices and inspect the uptime I do not see that they have been rebooted.
Strange I think.
Regards, Søren
Fra: observium [mailto:observium-bounces@observium.org] På vegne af Charlie Burns Sendt: 28. april 2014 17:46 Til: observium@observium.org Emne: [Observium] device reboot
Hello,
This morning 2 XR devices randomly flagged on observium as rebooted. Both devices asr9001 running XR 4.3.4 have 2 week uptimes and no events this morning.
Thanks,
Charlie Burns
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
This is almost always related to poor network connectivity.
When it isn't related to poor network connectivity, it's because of poor SNMP server on the remote device.
adam.
On 2014-04-28 10:45, Charlie Burns wrote:
Hello,
This morning 2 XR devices randomly flagged on observium as rebooted. Both devices asr9001 running XR 4.3.4 have 2 week uptimes and no events this morning.
Thanks,
Charlie Burns _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Strange no BFD/OSPF/LDP/BGP hits. Also it lists the uptime of the device since this "poor snmp server/poor network connectivity" event. Thanks for following up Adam.
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: Monday, April 28, 2014 1:24 PM To: Observium Network Observation System Subject: Re: [Observium] device reboot
This is almost always related to poor network connectivity.
When it isn't related to poor network connectivity, it's because of poor SNMP server on the remote device.
adam.
On 2014-04-28 10:45, Charlie Burns wrote:
Hello,
This morning 2 XR devices randomly flagged on observium as rebooted. Both devices asr9001 running XR 4.3.4 have 2 week uptimes and no events this morning.
Thanks,
Charlie Burns _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
It's possible that no response comes back from the sysUptime query, in which case PHP may, depending on constellation status, solar flares and other similar unpredictabilities, PHP tries to interpret this as 0, in which case Observium flags it as "just rebooted" (as 0 is < than its previous uptime).
Next poll it'll receive the right amount of uptime seconds again, and all will be well.
So yea, I'd go with "somewhat higher load / congestion" -> UDP packet missing in action or late to the party.
Tom
On 28/04/2014 19:35, Charlie Burns wrote:
Strange no BFD/OSPF/LDP/BGP hits. Also it lists the uptime of the device since this "poor snmp server/poor network connectivity" event. Thanks for following up Adam.
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: Monday, April 28, 2014 1:24 PM To: Observium Network Observation System Subject: Re: [Observium] device reboot
This is almost always related to poor network connectivity.
When it isn't related to poor network connectivity, it's because of poor SNMP server on the remote device.
adam.
On 2014-04-28 10:45, Charlie Burns wrote:
Hello,
This morning 2 XR devices randomly flagged on observium as rebooted. Both devices asr9001 running XR 4.3.4 have 2 week uptimes and no events this morning.
Thanks,
Charlie Burns _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
If anyone else hits this issue on ios-xr it is an open bug in 4.3.4.
Bug: CSCum44940 snmpd crash with backend pooling with snmp view cfg changes
Thanks,
Charlie Burns
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Tom Laermans Sent: Monday, April 28, 2014 2:45 PM To: Observium Network Observation System Subject: Re: [Observium] device reboot
It's possible that no response comes back from the sysUptime query, in which case PHP may, depending on constellation status, solar flares and other similar unpredictabilities, PHP tries to interpret this as 0, in which case Observium flags it as "just rebooted" (as 0 is < than its previous uptime).
Next poll it'll receive the right amount of uptime seconds again, and all will be well.
So yea, I'd go with "somewhat higher load / congestion" -> UDP packet missing in action or late to the party.
Tom
On 28/04/2014 19:35, Charlie Burns wrote:
Strange no BFD/OSPF/LDP/BGP hits. Also it lists the uptime of the device since this "poor snmp server/poor network connectivity" event. Thanks for following up Adam.
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: Monday, April 28, 2014 1:24 PM To: Observium Network Observation System Subject: Re: [Observium] device reboot
This is almost always related to poor network connectivity.
When it isn't related to poor network connectivity, it's because of poor SNMP server on the remote device.
adam.
On 2014-04-28 10:45, Charlie Burns wrote:
Hello,
This morning 2 XR devices randomly flagged on observium as rebooted. Both devices asr9001 running XR 4.3.4 have 2 week uptimes and no events this morning.
Thanks,
Charlie Burns _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Thanks for the update!
On 26/05/2014 17:30, Charlie Burns wrote:
If anyone else hits this issue on ios-xr it is an open bug in 4.3.4.
Bug: CSCum44940 snmpd crash with backend pooling with snmp view cfg changes
Thanks,
Charlie Burns
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Tom Laermans Sent: Monday, April 28, 2014 2:45 PM To: Observium Network Observation System Subject: Re: [Observium] device reboot
It's possible that no response comes back from the sysUptime query, in which case PHP may, depending on constellation status, solar flares and other similar unpredictabilities, PHP tries to interpret this as 0, in which case Observium flags it as "just rebooted" (as 0 is < than its previous uptime).
Next poll it'll receive the right amount of uptime seconds again, and all will be well.
So yea, I'd go with "somewhat higher load / congestion" -> UDP packet missing in action or late to the party.
Tom
On 28/04/2014 19:35, Charlie Burns wrote:
Strange no BFD/OSPF/LDP/BGP hits. Also it lists the uptime of the device since this "poor snmp server/poor network connectivity" event. Thanks for following up Adam.
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: Monday, April 28, 2014 1:24 PM To: Observium Network Observation System Subject: Re: [Observium] device reboot
This is almost always related to poor network connectivity.
When it isn't related to poor network connectivity, it's because of poor SNMP server on the remote device.
adam.
On 2014-04-28 10:45, Charlie Burns wrote:
Hello,
This morning 2 XR devices randomly flagged on observium as rebooted. Both devices asr9001 running XR 4.3.4 have 2 week uptimes and no events this morning.
Thanks,
Charlie Burns _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
We have also experienced events similar to this when the remove device has been completely maxxed out (99% CPU). At this point, Cisco devices especially in my experience, put their last resources into backplane processing and not responding to SNMP.
Just a thought...
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: 28 April 2014 18:24 To: Observium Network Observation System Subject: Re: [Observium] device reboot
This is almost always related to poor network connectivity.
When it isn't related to poor network connectivity, it's because of poor SNMP server on the remote device.
adam.
On 2014-04-28 10:45, Charlie Burns wrote:
Hello,
This morning 2 XR devices randomly flagged on observium as rebooted. Both devices asr9001 running XR 4.3.4 have 2 week uptimes and no events this morning.
Thanks,
Charlie Burns _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (5)
-
Adam Armstrong
-
Charlie Burns
-
John Millington
-
Søren Friis Rosiak
-
Tom Laermans