Alert ifInOctets_perc doesn't match port ifInOctets_perc?
Had a couple of utilisation alerts come through today reporting a 10Gig port as over 80% utilisation, when in reality it doesnt appear to be that busy?.
The alert email shows this graph, showing over 100% utilisation and a solid 80+% most of the day:
[image: Inline images 1]
When you click through the embedded link to the 'real' graph in Observium itself, it reports the correct utilisation figures I would epxect to see based on the traffic graph:
[image: Inline images 2]
Traffic graph for the port:
[image: Inline images 3]
Are the alerts reading the information from a different rrd? Doesn't seem like it as the port would be alarming constantly based on the graph in the email as it shows over 80% for most of the day?
Happy to accept that the port may have spiked to 8Gbps, but I would have thought this would have been reflected in the graphs somewhere but if anything traffic was tailing off?
Is there possibly an issue with the alerting on ifInOctets_perc or do I need to be looking more closely at what the device is reporting, which on face value _seems_ OK looking at a debug of poller.php?
Thanks, Tim C
The only way this is likely to happen is if the ifSpeed/ifHighSpeed value of the device changes.
adam.
On 2016-09-13 11:39, Mailing Lists wrote:
Had a couple of utilisation alerts come through today reporting a 10Gig port as over 80% utilisation, when in reality it doesnt appear to be that busy?.
The alert email shows this graph, showing over 100% utilisation and a solid 80+% most of the day:
When you click through the embedded link to the 'real' graph in Observium itself, it reports the correct utilisation figures I would epxect to see based on the traffic graph:
Traffic graph for the port:
Are the alerts reading the information from a different rrd? Doesn't seem like it as the port would be alarming constantly based on the graph in the email as it shows over 80% for most of the day?
Happy to accept that the port may have spiked to 8Gbps, but I would have thought this would have been reflected in the graphs somewhere but if anything traffic was tailing off?
Is there possibly an issue with the alerting on ifInOctets_perc or do I need to be looking more closely at what the device is reporting, which on face value _seems_ OK looking at a debug of poller.php?
Thanks, Tim C _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On 13 Sep 2016, at 17:54, Adam Armstrong adama@memetic.org wrote:
The only way this is likely to happen is if the ifSpeed/ifHighSpeed value of the device changes.
adam.
Thanks Adam. Yep, ifSpeed seems to be changing between 4294967295 and 10000000000 intermittently. :/
Just to save the first bit of digging, is this likely to be the device mis-reporting this specific value, or being set to a special value by observium because of missing or invalid response?
On 2016-09-13 11:39, Mailing Lists wrote: Had a couple of utilisation alerts come through today reporting a 10Gig port as over 80% utilisation, when in reality it doesnt appear to be that busy?. The alert email shows this graph, showing over 100% utilisation and a solid 80+% most of the day: When you click through the embedded link to the 'real' graph in Observium itself, it reports the correct utilisation figures I would epxect to see based on the traffic graph: Traffic graph for the port: Are the alerts reading the information from a different rrd? Doesn't seem like it as the port would be alarming constantly based on the graph in the email as it shows over 80% for most of the day? Happy to accept that the port may have spiked to 8Gbps, but I would have thought this would have been reflected in the graphs somewhere but if anything traffic was tailing off? Is there possibly an issue with the alerting on ifInOctets_perc or do I need to be looking more closely at what the device is reporting, which on face value _seems_ OK looking at a debug of poller.php? Thanks, Tim C _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
4294967295 is the maximum value of a 32bit value, this is likely to be being returned by the device. Not sure why.
adam. On 13/09/2016 13:38:53, Tim Cooper lists@coop3r.com wrote:
On 13 Sep 2016, at 17:54, Adam Armstrong <adama@memetic.org [mailto:adama@memetic.org]> wrote:
The only way this is likely to happen is if the ifSpeed/ifHighSpeed value of the device changes.
adam.
Thanks Adam. Yep, ifSpeed seems to be changing between 4294967295 and 10000000000 intermittently. :/
Just to save the first bit of digging, is this likely to be the device mis-reporting this specific value, or being set to a special value by observium because of missing or invalid response?
On 2016-09-13 11:39, Mailing Lists wrote:
Had a couple of utilisation alerts come through today reporting a
10Gig port as over 80% utilisation, when in reality it doesnt appear
to be that busy?.
The alert email shows this graph, showing over 100% utilisation and a
solid 80+% most of the day:
When you click through the embedded link to the 'real' graph in
Observium itself, it reports the correct utilisation figures I would
epxect to see based on the traffic graph:
Traffic graph for the port:
Are the alerts reading the information from a different rrd? Doesn't
seem like it as the port would be alarming constantly based on the
graph in the email as it shows over 80% for most of the day?
Happy to accept that the port may have spiked to 8Gbps, but I would
have thought this would have been reflected in the graphs somewhere
but if anything traffic was tailing off?
Is there possibly an issue with the alerting on ifInOctets_perc or do
I need to be looking more closely at what the device is reporting,
which on face value _seems_ OK looking at a debug of poller.php?
Thanks,
Tim C
_______________________________________________
observium mailing list
observium@observium.org [mailto:observium@observium.org]
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
On 13 Sep 2016, at 19:40, Adam Armstrong adama@memetic.org wrote:
4294967295 is the maximum value of a 32bit value, this is likely to be being returned by the device. Not sure why.
adam.
Looks like ifSpeed is supposed to report back 4294967295 if the interface is faster that 1gig. Thinking back I remember seeing something about this on the mailing list a while back? Question is why the device is switching to 10000000000 on ifSpeed intermittently when it should be a 32bit limit...
Will have to set something up to try and capture it happening and raise a TAC case :/
Thanks for your help,
Tim C
On 13/09/2016 13:38:53, Tim Cooper lists@coop3r.com wrote:
On 13 Sep 2016, at 17:54, Adam Armstrong adama@memetic.org wrote:
The only way this is likely to happen is if the ifSpeed/ifHighSpeed value of the device changes.
adam.
Thanks Adam. Yep, ifSpeed seems to be changing between 4294967295 and 10000000000 intermittently. :/
Just to save the first bit of digging, is this likely to be the device mis-reporting this specific value, or being set to a special value by observium because of missing or invalid response?
On 2016-09-13 11:39, Mailing Lists wrote: Had a couple of utilisation alerts come through today reporting a 10Gig port as over 80% utilisation, when in reality it doesnt appear to be that busy?. The alert email shows this graph, showing over 100% utilisation and a solid 80+% most of the day: When you click through the embedded link to the 'real' graph in Observium itself, it reports the correct utilisation figures I would epxect to see based on the traffic graph: Traffic graph for the port: Are the alerts reading the information from a different rrd? Doesn't seem like it as the port would be alarming constantly based on the graph in the email as it shows over 80% for most of the day? Happy to accept that the port may have spiked to 8Gbps, but I would have thought this would have been reflected in the graphs somewhere but if anything traffic was tailing off? Is there possibly an issue with the alerting on ifInOctets_perc or do I need to be looking more closely at what the device is reporting, which on face value _seems_ OK looking at a debug of poller.php? Thanks, Tim C _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
the larger number comes from ifHighSpeed. If that's occasionally missing from the output, it'll revert to the ifSpeed value.
adam. On 13/09/2016 14:38:54, Tim Cooper lists@coop3r.com wrote:
On 13 Sep 2016, at 19:40, Adam Armstrong <adama@memetic.org [mailto:adama@memetic.org]> wrote:
4294967295 is the maximum value of a 32bit value, this is likely to be being returned by the device. Not sure why.
adam.
Looks like ifSpeed is supposed to report back 4294967295 if the interface is faster that 1gig. Thinking back I remember seeing something about this on the mailing list a while back? Question is why the device is switching to 10000000000 on ifSpeed intermittently when it should be a 32bit limit...
Will have to set something up to try and capture it happening and raise a TAC case :/
Thanks for your help,
Tim C
On 13/09/2016 13:38:53, Tim Cooper <lists@coop3r.com [mailto:lists@coop3r.com]> wrote:
On 13 Sep 2016, at 17:54, Adam Armstrong <adama@memetic.org [mailto:adama@memetic.org]> wrote:
The only way this is likely to happen is if the ifSpeed/ifHighSpeed value of the device changes.
adam.
Thanks Adam. Yep, ifSpeed seems to be changing between 4294967295 and 10000000000 intermittently. :/
Just to save the first bit of digging, is this likely to be the device mis-reporting this specific value, or being set to a special value by observium because of missing or invalid response?
On 2016-09-13 11:39, Mailing Lists wrote:
Had a couple of utilisation alerts come through today reporting a
10Gig port as over 80% utilisation, when in reality it doesnt appear
to be that busy?.
The alert email shows this graph, showing over 100% utilisation and a
solid 80+% most of the day:
When you click through the embedded link to the 'real' graph in
Observium itself, it reports the correct utilisation figures I would
epxect to see based on the traffic graph:
Traffic graph for the port:
Are the alerts reading the information from a different rrd? Doesn't
seem like it as the port would be alarming constantly based on the
graph in the email as it shows over 80% for most of the day?
Happy to accept that the port may have spiked to 8Gbps, but I would
have thought this would have been reflected in the graphs somewhere
but if anything traffic was tailing off?
Is there possibly an issue with the alerting on ifInOctets_perc or do
I need to be looking more closely at what the device is reporting,
which on face value _seems_ OK looking at a debug of poller.php?
Thanks,
Tim C
_______________________________________________
observium mailing list
observium@observium.org [mailto:observium@observium.org]
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
Indeed, Observium itself replaces ifSpeed with the contents of ifHighSpeed if that is being sent. So if we're not getting ifHighSpeed in for some reason the Speed will be set at 2^32 ...
Maybe we should handle 2^32 as "the interface is high speed and we need ifHighSpeed - if we don't receive it, stay at old value" ?
Tom
On 13/09/2016 22:13, Adam Armstrong wrote:
the larger number comes from ifHighSpeed. If that's occasionally missing from the output, it'll revert to the ifSpeed value.
adam.
On 13/09/2016 14:38:54, Tim Cooper lists@coop3r.com wrote:
On 13 Sep 2016, at 19:40, Adam Armstrong <adama@memetic.org mailto:adama@memetic.org> wrote:
4294967295 is the maximum value of a 32bit value, this is likely to be being returned by the device. Not sure why.
adam.
Looks like ifSpeed is supposed to report back 4294967295 if the interface is faster that 1gig. Thinking back I remember seeing something about this on the mailing list a while back? Question is why the device is switching to 10000000000 on ifSpeed intermittently when it should be a 32bit limit...
Will have to set something up to try and capture it happening and raise a TAC case :/
Thanks for your help,
Tim C
On 13/09/2016 13:38:53, Tim Cooper <lists@coop3r.com mailto:lists@coop3r.com> wrote:
On 13 Sep 2016, at 17:54, Adam Armstrong <adama@memetic.org mailto:adama@memetic.org> wrote:
The only way this is likely to happen is if the ifSpeed/ifHighSpeed value of the device changes.
adam.
Thanks Adam. Yep, ifSpeed seems to be changing between 4294967295 and 10000000000 intermittently. :/
Just to save the first bit of digging, is this likely to be the device mis-reporting this specific value, or being set to a special value by observium because of missing or invalid response?
On 2016-09-13 11:39, Mailing Lists wrote:
Had a couple of utilisation alerts come through today reporting a 10Gig port as over 80% utilisation, when in reality it doesnt appear to be that busy?. The alert email shows this graph, showing over 100% utilisation and a solid 80+% most of the day: When you click through the embedded link to the 'real' graph in Observium itself, it reports the correct utilisation figures I would epxect to see based on the traffic graph: Traffic graph for the port: Are the alerts reading the information from a different rrd? Doesn't seem like it as the port would be alarming constantly based on the graph in the email as it shows over 80% for most of the day? Happy to accept that the port may have spiked to 8Gbps, but I would have thought this would have been reflected in the graphs somewhere but if anything traffic was tailing off? Is there possibly an issue with the alerting on ifInOctets_perc or do I need to be looking more closely at what the device is reporting, which on face value _seems_ OK looking at a debug of poller.php? Thanks, Tim C _______________________________________________ observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On 14 Sep 2016, at 08:47, Tom Laermans tom.laermans@powersource.cx wrote:
Indeed, Observium itself replaces ifSpeed with the contents of ifHighSpeed if that is being sent. So if we're not getting ifHighSpeed in for some reason the Speed will be set at 2^32 ...
Maybe we should handle 2^32 as "the interface is high speed and we need ifHighSpeed - if we don't receive it, stay at old value" ?
Tom
Well for me, I'm going to try and focus on why the device is crapping out, and try to work around that to stop it flapping in the first place, as that seems the best fix ;)
I'll leave you guys to decide what's best on the software design. One could argue there is only so much you should do to work around the device itself not behaving correctly, which is essentially the issue here :)
On 13/09/2016 22:13, Adam Armstrong wrote: the larger number comes from ifHighSpeed. If that's occasionally missing from the output, it'll revert to the ifSpeed value.
adam.
On 13/09/2016 14:38:54, Tim Cooper lists@coop3r.com wrote:
On 13 Sep 2016, at 19:40, Adam Armstrong adama@memetic.org wrote:
4294967295 is the maximum value of a 32bit value, this is likely to be being returned by the device. Not sure why.
adam.
Looks like ifSpeed is supposed to report back 4294967295 if the interface is faster that 1gig. Thinking back I remember seeing something about this on the mailing list a while back? Question is why the device is switching to 10000000000 on ifSpeed intermittently when it should be a 32bit limit...
Will have to set something up to try and capture it happening and raise a TAC case :/
Thanks for your help,
Tim C
On 13/09/2016 13:38:53, Tim Cooper lists@coop3r.com wrote:
On 13 Sep 2016, at 17:54, Adam Armstrong adama@memetic.org wrote:
The only way this is likely to happen is if the ifSpeed/ifHighSpeed value of the device changes.
adam.
Thanks Adam. Yep, ifSpeed seems to be changing between 4294967295 and 10000000000 intermittently. :/
Just to save the first bit of digging, is this likely to be the device mis-reporting this specific value, or being set to a special value by observium because of missing or invalid response?
> On 2016-09-13 11:39, Mailing Lists wrote: > Had a couple of utilisation alerts come through today reporting a > 10Gig port as over 80% utilisation, when in reality it doesnt appear > to be that busy?. > The alert email shows this graph, showing over 100% utilisation and a > solid 80+% most of the day: > When you click through the embedded link to the 'real' graph in > Observium itself, it reports the correct utilisation figures I would > epxect to see based on the traffic graph: > Traffic graph for the port: > Are the alerts reading the information from a different rrd? Doesn't > seem like it as the port would be alarming constantly based on the > graph in the email as it shows over 80% for most of the day? > Happy to accept that the port may have spiked to 8Gbps, but I would > have thought this would have been reflected in the graphs somewhere > but if anything traffic was tailing off? > Is there possibly an issue with the alerting on ifInOctets_perc or do > I need to be looking more closely at what the device is reporting, > which on face value _seems_ OK looking at a debug of poller.php? > Thanks, > Tim C > _______________________________________________ > observium mailing list > observium@observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Yeah, I'm not a fan of "assuming" too much.
adam. On 14/09/2016 07:44:06, Tim Cooper lists@coop3r.com wrote:
On 14 Sep 2016, at 08:47, Tom Laermans <tom.laermans@powersource.cx [mailto:tom.laermans@powersource.cx]> wrote:
Indeed, Observium itself replaces ifSpeed with the contents of ifHighSpeed if that is being sent. So if we're not getting ifHighSpeed in for some reason the Speed will be set at 2^32 ...
Maybe we should handle 2^32 as "the interface is high speed and we need ifHighSpeed - if we don't receive it, stay at old value" ?
Tom
Well for me, I'm going to try and focus on why the device is crapping out, and try to work around that to stop it flapping in the first place, as that seems the best fix ;)
I'll leave you guys to decide what's best on the software design. One could argue there is only so much you should do to work around the device itself not behaving correctly, which is essentially the issue here :)
On 13/09/2016 22:13, Adam Armstrong wrote:
the larger number comes from ifHighSpeed. If that's occasionally missing from the output, it'll revert to the ifSpeed value.
adam. On 13/09/2016 14:38:54, Tim Cooper lists@coop3r.com [mailto:lists@coop3r.com] wrote:
On 13 Sep 2016, at 19:40, Adam Armstrong <adama@memetic.org [mailto:adama@memetic.org]> wrote:
4294967295 is the maximum value of a 32bit value, this is likely to be being returned by the device. Not sure why.
adam.
Looks like ifSpeed is supposed to report back 4294967295 if the interface is faster that 1gig. Thinking back I remember seeing something about this on the mailing list a while back? Question is why the device is switching to 10000000000 on ifSpeed intermittently when it should be a 32bit limit...
Will have to set something up to try and capture it happening and raise a TAC case :/
Thanks for your help,
Tim C
On 13/09/2016 13:38:53, Tim Cooper <lists@coop3r.com [mailto:lists@coop3r.com]> wrote:
On 13 Sep 2016, at 17:54, Adam Armstrong <adama@memetic.org [mailto:adama@memetic.org]> wrote:
The only way this is likely to happen is if the ifSpeed/ifHighSpeed value of the device changes.
adam.
Thanks Adam. Yep, ifSpeed seems to be changing between 4294967295 and 10000000000 intermittently. :/
Just to save the first bit of digging, is this likely to be the device mis-reporting this specific value, or being set to a special value by observium because of missing or invalid response?
On 2016-09-13 11:39, Mailing Lists wrote:
Had a couple of utilisation alerts come through today reporting a
10Gig port as over 80% utilisation, when in reality it doesnt appear
to be that busy?.
The alert email shows this graph, showing over 100% utilisation and a
solid 80+% most of the day:
When you click through the embedded link to the 'real' graph in
Observium itself, it reports the correct utilisation figures I would
epxect to see based on the traffic graph:
Traffic graph for the port:
Are the alerts reading the information from a different rrd? Doesn't
seem like it as the port would be alarming constantly based on the
graph in the email as it shows over 80% for most of the day?
Happy to accept that the port may have spiked to 8Gbps, but I would
have thought this would have been reflected in the graphs somewhere
but if anything traffic was tailing off?
Is there possibly an issue with the alerting on ifInOctets_perc or do
I need to be looking more closely at what the device is reporting,
which on face value _seems_ OK looking at a debug of poller.php?
Thanks,
Tim C
_______________________________________________
observium mailing list
observium@observium.org [mailto:observium@observium.org]
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
Agreed, but - It is highly unlikely anyone will develop a new interface that does exactly 4294967295bps - If it is indeed defined somewhere that this is the value that says "look at ifHighSpeed instead", that is an interesting datapoint.
I think the latter bullet point is something worth finding out and if it is the case, act upon it.
Tom
On 14/09/2016 17:21, Adam Armstrong wrote:
Yeah, I'm not a fan of "assuming" too much.
adam.
On 14/09/2016 07:44:06, Tim Cooper lists@coop3r.com wrote:
On 14 Sep 2016, at 08:47, Tom Laermans <tom.laermans@powersource.cx mailto:tom.laermans@powersource.cx> wrote:
Indeed, Observium itself replaces ifSpeed with the contents of ifHighSpeed if that is being sent. So if we're not getting ifHighSpeed in for some reason the Speed will be set at 2^32 ...
Maybe we should handle 2^32 as "the interface is high speed and we need ifHighSpeed - if we don't receive it, stay at old value" ?
Tom
Well for me, I'm going to try and focus on why the device is crapping out, and try to work around that to stop it flapping in the first place, as that seems the best fix ;)
I'll leave you guys to decide what's best on the software design. One could argue there is only so much you should do to work around the device itself not behaving correctly, which is essentially the issue here :)
On 13/09/2016 22:13, Adam Armstrong wrote:
the larger number comes from ifHighSpeed. If that's occasionally missing from the output, it'll revert to the ifSpeed value.
adam.
On 13/09/2016 14:38:54, Tim Cooper lists@coop3r.com wrote:
On 13 Sep 2016, at 19:40, Adam Armstrong <adama@memetic.org mailto:adama@memetic.org> wrote:
4294967295 is the maximum value of a 32bit value, this is likely to be being returned by the device. Not sure why.
adam.
Looks like ifSpeed is supposed to report back 4294967295 if the interface is faster that 1gig. Thinking back I remember seeing something about this on the mailing list a while back? Question is why the device is switching to 10000000000 on ifSpeed intermittently when it should be a 32bit limit...
Will have to set something up to try and capture it happening and raise a TAC case :/
Thanks for your help,
Tim C
> On 13/09/2016 13:38:53, Tim Cooper <lists@coop3r.com > mailto:lists@coop3r.com> wrote: > > > > On 13 Sep 2016, at 17:54, Adam Armstrong <adama@memetic.org > mailto:adama@memetic.org> wrote: > >> The only way this is likely to happen is if the >> ifSpeed/ifHighSpeed value of the device changes. >> >> adam. > > Thanks Adam. Yep, ifSpeed seems to be changing between > 4294967295 and 10000000000 intermittently. :/ > > Just to save the first bit of digging, is this likely to be the > device mis-reporting this specific value, or being set to a > special value by observium because of missing or invalid response? > >> >> On 2016-09-13 11:39, Mailing Lists wrote: >>> Had a couple of utilisation alerts come through today reporting a >>> 10Gig port as over 80% utilisation, when in reality it doesnt >>> appear >>> to be that busy?. >>> The alert email shows this graph, showing over 100% >>> utilisation and a >>> solid 80+% most of the day: >>> When you click through the embedded link to the 'real' graph in >>> Observium itself, it reports the correct utilisation figures I >>> would >>> epxect to see based on the traffic graph: >>> Traffic graph for the port: >>> Are the alerts reading the information from a different rrd? >>> Doesn't >>> seem like it as the port would be alarming constantly based on the >>> graph in the email as it shows over 80% for most of the day? >>> Happy to accept that the port may have spiked to 8Gbps, but I >>> would >>> have thought this would have been reflected in the graphs >>> somewhere >>> but if anything traffic was tailing off? >>> Is there possibly an issue with the alerting on >>> ifInOctets_perc or do >>> I need to be looking more closely at what the device is reporting, >>> which on face value _seems_ OK looking at a debug of poller.php? >>> Thanks, >>> Tim C >>> _______________________________________________ >>> observium mailing list >>> observium@observium.org mailto:observium@observium.org >>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium >> _______________________________________________ >> observium mailing list >> observium@observium.org mailto:observium@observium.org >> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On 14 Sep 2016, at 17:41, Tom Laermans tom.laermans@powersource.cx wrote:
Agreed, but
- It is highly unlikely anyone will develop a new interface that does exactly 4294967295bps
- If it is indeed defined somewhere that this is the value that says "look at ifHighSpeed instead", that is an interesting datapoint.
I think the latter bullet point is something worth finding out and if it is the case, act upon it.
Tom
That did appear to be the case in the RFC I found yesterday. Will try and dig out the paragraph and a link.
On 14/09/2016 17:21, Adam Armstrong wrote: Yeah, I'm not a fan of "assuming" too much.
adam.
On 14/09/2016 07:44:06, Tim Cooper lists@coop3r.com wrote:
On 14 Sep 2016, at 08:47, Tom Laermans tom.laermans@powersource.cx wrote:
Indeed, Observium itself replaces ifSpeed with the contents of ifHighSpeed if that is being sent. So if we're not getting ifHighSpeed in for some reason the Speed will be set at 2^32 ...
Maybe we should handle 2^32 as "the interface is high speed and we need ifHighSpeed - if we don't receive it, stay at old value" ?
Tom
Well for me, I'm going to try and focus on why the device is crapping out, and try to work around that to stop it flapping in the first place, as that seems the best fix ;)
I'll leave you guys to decide what's best on the software design. One could argue there is only so much you should do to work around the device itself not behaving correctly, which is essentially the issue here :)
On 13/09/2016 22:13, Adam Armstrong wrote: the larger number comes from ifHighSpeed. If that's occasionally missing from the output, it'll revert to the ifSpeed value.
adam.
On 13/09/2016 14:38:54, Tim Cooper lists@coop3r.com wrote:
On 13 Sep 2016, at 19:40, Adam Armstrong adama@memetic.org wrote:
> 4294967295 is the maximum value of a 32bit value, this is likely to be being returned by the device. Not sure why. > > adam.
Looks like ifSpeed is supposed to report back 4294967295 if the interface is faster that 1gig. Thinking back I remember seeing something about this on the mailing list a while back? Question is why the device is switching to 10000000000 on ifSpeed intermittently when it should be a 32bit limit...
Will have to set something up to try and capture it happening and raise a TAC case :/
Thanks for your help,
Tim C
>> On 13/09/2016 13:38:53, Tim Cooper lists@coop3r.com wrote: >> >> >> >> On 13 Sep 2016, at 17:54, Adam Armstrong adama@memetic.org wrote: >> >>> The only way this is likely to happen is if the ifSpeed/ifHighSpeed value of the device changes. >>> >>> adam. >> >> Thanks Adam. Yep, ifSpeed seems to be changing between 4294967295 and 10000000000 intermittently. :/ >> >> Just to save the first bit of digging, is this likely to be the device mis-reporting this specific value, or being set to a special value by observium because of missing or invalid response? >> >>> >>>> On 2016-09-13 11:39, Mailing Lists wrote: >>>> Had a couple of utilisation alerts come through today reporting a >>>> 10Gig port as over 80% utilisation, when in reality it doesnt appear >>>> to be that busy?. >>>> The alert email shows this graph, showing over 100% utilisation and a >>>> solid 80+% most of the day: >>>> When you click through the embedded link to the 'real' graph in >>>> Observium itself, it reports the correct utilisation figures I would >>>> epxect to see based on the traffic graph: >>>> Traffic graph for the port: >>>> Are the alerts reading the information from a different rrd? Doesn't >>>> seem like it as the port would be alarming constantly based on the >>>> graph in the email as it shows over 80% for most of the day? >>>> Happy to accept that the port may have spiked to 8Gbps, but I would >>>> have thought this would have been reflected in the graphs somewhere >>>> but if anything traffic was tailing off? >>>> Is there possibly an issue with the alerting on ifInOctets_perc or do >>>> I need to be looking more closely at what the device is reporting, >>>> which on face value _seems_ OK looking at a debug of poller.php? >>>> Thanks, >>>> Tim C >>>> _______________________________________________ >>>> observium mailing list >>>> observium@observium.org >>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium >>> _______________________________________________ >>> observium mailing list >>> observium@observium.org >>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > _______________________________________________ > observium mailing list > observium@observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
It looks like RFC 2863 https://tools.ietf.org/html/rfc2863 says that:
ifSpeed OBJECT-TYPE
McCloghrie & Kastenholz Standards Track [Page 30]
------------------------------------------------------------------------
https://tools.ietf.org/html/rfc2863#page-31 RFC 2863 https://tools.ietf.org/html/rfc2863 The Interfaces Group MIB June 2000
SYNTAX Gauge32 MAX-ACCESS read-only STATUS current DESCRIPTION "An estimate of the interface's current bandwidth in bits per second. For interfaces which do not vary in bandwidth or for those where no accurate estimation can be made, this object should contain the nominal bandwidth.*If the bandwidth of the interface is greater than the maximum value reportable by this object then this object should report its maximum value (4,294,967,295) and ifHighSpeed must be used to report the interace's speed.* For a sub-layer which has no concept of bandwidth, this object should be zero." ::= { ifEntry 5 }
So ifSpeed "should" be reported as 2^32, but ifHighSpeed "must" be used for the value. That seems to jive with the logic of "if Observium gets 2^32 reported for ifSpeed, but nothing for ifHighSpeed, then stick with the previous value (presumably from a properly received ifHighSpeed value)" that Tom suggested.
On 09/14/2016 02:14 PM, Tim Cooper wrote:
On 14 Sep 2016, at 17:41, Tom Laermans <tom.laermans@powersource.cx mailto:tom.laermans@powersource.cx> wrote:
Agreed, but
- It is highly unlikely anyone will develop a new interface that does
exactly 4294967295bps
- If it is indeed defined somewhere that this is the value that says
"look at ifHighSpeed instead", that is an interesting datapoint.
I think the latter bullet point is something worth finding out and if it is the case, act upon it.
Tom
That did appear to be the case in the RFC I found yesterday. Will try and dig out the paragraph and a link.
On 14/09/2016 17:21, Adam Armstrong wrote:
Yeah, I'm not a fan of "assuming" too much.
adam.
On 14/09/2016 07:44:06, Tim Cooper lists@coop3r.com wrote:
On 14 Sep 2016, at 08:47, Tom Laermans <tom.laermans@powersource.cx mailto:tom.laermans@powersource.cx> wrote:
Indeed, Observium itself replaces ifSpeed with the contents of ifHighSpeed if that is being sent. So if we're not getting ifHighSpeed in for some reason the Speed will be set at 2^32 ...
Maybe we should handle 2^32 as "the interface is high speed and we need ifHighSpeed - if we don't receive it, stay at old value" ?
Tom
Well for me, I'm going to try and focus on why the device is crapping out, and try to work around that to stop it flapping in the first place, as that seems the best fix ;)
I'll leave you guys to decide what's best on the software design. One could argue there is only so much you should do to work around the device itself not behaving correctly, which is essentially the issue here :)
On 13/09/2016 22:13, Adam Armstrong wrote:
the larger number comes from ifHighSpeed. If that's occasionally missing from the output, it'll revert to the ifSpeed value.
adam. > > On 13/09/2016 14:38:54, Tim Cooper lists@coop3r.com wrote: > > > > On 13 Sep 2016, at 19:40, Adam Armstrong <adama@memetic.org > mailto:adama@memetic.org> wrote: > >> 4294967295 is the maximum value of a 32bit value, this is >> likely to be being returned by the device. Not sure why. >> >> adam. > > Looks like ifSpeed is supposed to report back 4294967295 if the > interface is faster that 1gig. Thinking back I remember seeing > something about this on the mailing list a while back? Question > is why the device is switching to 10000000000 on ifSpeed > intermittently when it should be a 32bit limit... > > Will have to set something up to try and capture it happening > and raise a TAC case :/ > > Thanks for your help, > > Tim C > >>> On 13/09/2016 13:38:53, Tim Cooper <lists@coop3r.com >>> mailto:lists@coop3r.com> wrote: >>> >>> >>> >>> On 13 Sep 2016, at 17:54, Adam Armstrong <adama@memetic.org >>> mailto:adama@memetic.org> wrote: >>> >>>> The only way this is likely to happen is if the >>>> ifSpeed/ifHighSpeed value of the device changes. >>>> >>>> adam. >>> >>> Thanks Adam. Yep, ifSpeed seems to be changing between >>> 4294967295 and 10000000000 intermittently. :/ >>> >>> Just to save the first bit of digging, is this likely to be >>> the device mis-reporting this specific value, or being set to >>> a special value by observium because of missing or invalid >>> response? >>> >>>> >>>> On 2016-09-13 11:39, Mailing Lists wrote: >>>>> Had a couple of utilisation alerts come through today >>>>> reporting a >>>>> 10Gig port as over 80% utilisation, when in reality it >>>>> doesnt appear >>>>> to be that busy?. >>>>> The alert email shows this graph, showing over 100% >>>>> utilisation and a >>>>> solid 80+% most of the day: >>>>> When you click through the embedded link to the 'real' graph in >>>>> Observium itself, it reports the correct utilisation figures >>>>> I would >>>>> epxect to see based on the traffic graph: >>>>> Traffic graph for the port: >>>>> Are the alerts reading the information from a different rrd? >>>>> Doesn't >>>>> seem like it as the port would be alarming constantly based >>>>> on the >>>>> graph in the email as it shows over 80% for most of the day? >>>>> Happy to accept that the port may have spiked to 8Gbps, but >>>>> I would >>>>> have thought this would have been reflected in the graphs >>>>> somewhere >>>>> but if anything traffic was tailing off? >>>>> Is there possibly an issue with the alerting on >>>>> ifInOctets_perc or do >>>>> I need to be looking more closely at what the device is >>>>> reporting, >>>>> which on face value _seems_ OK looking at a debug of poller.php? >>>>> Thanks, >>>>> Tim C >>>>> _______________________________________________ >>>>> observium mailing list >>>>> observium@observium.org mailto:observium@observium.org >>>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium >>>> _______________________________________________ >>>> observium mailing list >>>> observium@observium.org mailto:observium@observium.org >>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium >> _______________________________________________ >> observium mailing list >> observium@observium.org mailto:observium@observium.org >> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (5)
-
Adam Armstrong -
Joseph Zancocchio -
Mailing Lists -
Tim Cooper
-
Tom Laermans