We are seeing spurious peaks and troughs on some traces from a MX480 Juniper router. We are sure that these events are not actually occurring and are trying to track down the reason for them. I wonder if other users might be able to help?
Thanks
Terry
Hi, This is most probably due to counters not updating properly between polls.The counter is unchanged on the first poll resulting in a 0bytes/s-rate and then on the next poll the counter for the whole 10min-window is updated resulting in a spike of exactly 2x the current rate. As this seems to happen regulary every 6th hour I suspect this is related to discovery. Maybe the router has some SNMP-ratelimiting to the controlplane and during discovery Observium will fire of a lot of SNMP-polls causing the router to block them or something like that?
You can try to turn off discovery temporary (comment out the line in the cronjob) and wait 6h from the last spike, if the spike is gone then its pretty clear that the router does not like the large amount of SNMP-querys during discovery for some reason.
/Markus
Den mån 18 mars 2019 kl 15:26 skrev Terry Stone via observium < observium@observium.org>:
We are seeing spurious peaks and troughs on some traces from a MX480 Juniper router. We are sure that these events are not actually occurring and are trying to track down the reason for them. I wonder if other users might be able to help?
Thanks
Terry _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Yeah, that looks like exactly what this is. The peak is the data that was missing from the previous poll. Very odd.
Does that platform have known issues with slow updating long caching of queries?
adam.
Sent from Mailbird [http://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm...] On 2019-03-18 14:49:13, Markus Klock via observium observium@observium.org wrote: Hi, This is most probably due to counters not updating properly between polls.The counter is unchanged on the first poll resulting in a 0bytes/s-rate and then on the next poll the counter for the whole 10min-window is updated resulting in a spike of exactly 2x the current rate. As this seems to happen regulary every 6th hour I suspect this is related to discovery. Maybe the router has some SNMP-ratelimiting to the controlplane and during discovery Observium will fire of a lot of SNMP-polls causing the router to block them or something like that?
You can try to turn off discovery temporary (comment out the line in the cronjob) and wait 6h from the last spike, if the spike is gone then its pretty clear that the router does not like the large amount of SNMP-querys during discovery for some reason.
/Markus
Den mån 18 mars 2019 kl 15:26 skrev Terry Stone via observium <observium@observium.org [mailto:observium@observium.org]>:
We are seeing spurious peaks and troughs on some traces from a MX480 Juniper router. We are sure that these events are not actually occurring and are trying to track down the reason for them. I wonder if other users might be able to help?
Thanks
Terry _______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
This usually happens with older Juniper hardware (if you have an RE-2000) but only with faster polling rates (30s-1min).
It's hard to miss a 5 minute update window.
On Mon, Mar 18, 2019 at 1:24 PM Adam Armstrong via observium observium@observium.org wrote:
Yeah, that looks like exactly what this is. The peak is the data that was missing from the previous poll. Very odd.
Does that platform have known issues with slow updating long caching of queries?
adam.
Sent from Mailbird
On 2019-03-18 14:49:13, Markus Klock via observium observium@observium.org wrote:
Hi, This is most probably due to counters not updating properly between polls.The counter is unchanged on the first poll resulting in a 0bytes/s-rate and then on the next poll the counter for the whole 10min-window is updated resulting in a spike of exactly 2x the current rate. As this seems to happen regulary every 6th hour I suspect this is related to discovery. Maybe the router has some SNMP-ratelimiting to the controlplane and during discovery Observium will fire of a lot of SNMP-polls causing the router to block them or something like that?
You can try to turn off discovery temporary (comment out the line in the cronjob) and wait 6h from the last spike, if the spike is gone then its pretty clear that the router does not like the large amount of SNMP-querys during discovery for some reason.
/Markus
Den mån 18 mars 2019 kl 15:26 skrev Terry Stone via observium observium@observium.org:
We are seeing spurious peaks and troughs on some traces from a MX480 Juniper router. We are sure that these events are not actually occurring and are trying to track down the reason for them. I wonder if other users might be able to help?
Thanks
Terry _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Thanks for that. We have just upgraded to the MX480 from a MX80, we were seeing a very similar pattern on the MX80 so it seems to me it might be a Juniper issue rather than a discrete platform issue. I will talk to our Juniper expert to get his thoughts. On 3/18/2019 12:37:37 PM, Luis Balbinot via observium observium@observium.org wrote: This usually happens with older Juniper hardware (if you have an RE-2000) but only with faster polling rates (30s-1min).
It's hard to miss a 5 minute update window.
On Mon, Mar 18, 2019 at 1:24 PM Adam Armstrong via observium wrote:
Yeah, that looks like exactly what this is. The peak is the data that was missing from the previous poll. Very odd.
Does that platform have known issues with slow updating long caching of queries?
adam.
Sent from Mailbird
On 2019-03-18 14:49:13, Markus Klock via observium wrote:
Hi, This is most probably due to counters not updating properly between polls.The counter is unchanged on the first poll resulting in a 0bytes/s-rate and then on the next poll the counter for the whole 10min-window is updated resulting in a spike of exactly 2x the current rate. As this seems to happen regulary every 6th hour I suspect this is related to discovery. Maybe the router has some SNMP-ratelimiting to the controlplane and during discovery Observium will fire of a lot of SNMP-polls causing the router to block them or something like that?
You can try to turn off discovery temporary (comment out the line in the cronjob) and wait 6h from the last spike, if the spike is gone then its pretty clear that the router does not like the large amount of SNMP-querys during discovery for some reason.
/Markus
Den mån 18 mars 2019 kl 15:26 skrev Terry Stone via observium :
We are seeing spurious peaks and troughs on some traces from a MX480 Juniper router. We are sure that these events are not actually occurring and are trying to track down the reason for them. I wonder if other users might be able to help?
Thanks
Terry _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (4)
-
Adam Armstrong -
Luis Balbinot -
Markus Klock -
Terry Stone