I've seen a couple other entries where there is a burst that doesn't fit with the interface counters, however none as consistent as the one I originally emailed on.
The below bill/interface just had a burst small burst inbound that doesn't show up in the counters, but does in the billing graph (bill1-accurate.png).
20, 2011-11-15 08:15:08, 299, 749963913, 2972, 749960941
20, 2011-11-15 08:20:07, 300, 1020775706, 1885, 1020773821
20, 2011-11-15 08:25:09, 302, 813072182, 1806, 813070376
20, 2011-11-15 08:30:08, 299, 803081017, 4671, 803076346
20, 2011-11-15 08:35:07, 299, 1182257940, 2472, 1182255468
20, 2011-11-15 08:40:08, 301, 65126192730628, 65124905329746, 1287400882
20, 2011-11-15 08:45:07, 300, 65126107460041, 65124905329746, 1202130295
20, 2011-11-15 08:50:08, 301, 1268579798, 3388, 1268576410
20, 2011-11-15 08:55:08, 300, 1223124545, 3515, 1223121030
20, 2011-11-15 09:00:07, 299, 1459063239, 2204, 1459061035
Which stands out not only because of the massive increase, but also being exactly the same.
bill_history:
20, XXXXX, cdr, 10000, 1, 0, 0, 62454, 62454, out, 134360224, 130251020, 4109204, 0, 0, 1032722, 2011-11-22 12:23:34, , , , 0
The original interface (index 1908) has been showing a continual 8Gb/s of inbound delta (bill2-accurate.png), even though if you look at the ifHCInOctets there is actually zero change. And it doesn't show up in the interface graphs (bill2-interface.png)
Tue Nov 22 14:27:45 EST 2011
berant@observium:/opt/observium$ date ; snmpget -c Not4SnMpP -v 2c X.X.X.X 1.3.6.1.2.1.31.1.1.1.6.1908
Tue Nov 22 14:28:01 EST 2011
IF-MIB::ifHCInOctets.1908 = Counter64: 644220606
berant@observium:/opt/observium$ date ; snmpget -c Not4SnMpP -v 2c X.X.X.X 1.3.6.1.2.1.31.1.1.1.6.1908
Tue Nov 22 14:35:04 EST 2011
IF-MIB::ifHCInOctets.1908 = Counter64: 644220606
berant@observium:/opt/observium$ date ; snmpget -c Not4SnMpP -v 2c X.X.X.X 1.3.6.1.2.1.31.1.1.1.6.1908
Tue Nov 22 14:42:39 EST 2011
IF-MIB::ifHCInOctets.1908 = Counter64: 644220606
berant@observium:/opt/observium$ date ; snmpget -c Not4SnMpP -v 2c X.X.X.X 1.3.6.1.2.1.31.1.1.1.6.1908
Tue Nov 22 14:47:39 EST 2011
IF-MIB::ifHCInOctets.1908 = Counter64: 644220606
All of the bills I've seen this issue on have extremely minimal or in the above case zero traffic in one direction or another.
Anything else I can provide?
Thanks,
Berant
On Wed, Nov 16, 2011 at 9:34 AM, Berant Lemmenes
<berant@lemmenes.com> wrote:
IF-MIB::ifHCOutOctets.1908 = Counter64: 1302403640
IF-MIB::ifHCInOctets.1908 = Counter64: 644218878
On Wed, Nov 16, 2011 at 9:18 AM, Adam Armstrong
<adama@memetic.org> wrote:
Ahh shit, that table doesn't store counters, only deltas.
What is the current SNMP counter for that port?
adam.
On Wed, 16 Nov 2011 09:12:21 -0500, Berant Lemmenes <berant@lemmenes.com>
wrote:
> On Wed, Nov 16, 2011 at 8:16 AM, Adam Armstrong <
adama@memetic.org>
wrote:
>
>>
>> Same thing, I should imagine. I've never seen this bug present anywhere
>> else.
>>
>> Is the device being polled quite far away? Or attached on the end of
some
>> string?
>>
>> What is the measurement directly before the spike?
>>
>> adam.
>>
>>
> It's actually diversely routed bailing wire. :) It's about ~300 miles
away
> or 6ms as the packet flies.
>
> Looking at the data it looks like in_delta is looking suspicious, it's
> either 0, 64, 896, or some crazy large value during the spike. There is
> essentially 0 inbound traffic on this interface and barely any outbound,
> could that play into it?
>
> Here are the values before and at the start of the large blocks of high
> values:
>
> 31, 2011-11-10 16:55:07, 300, 400, 64, 336
> 31, 2011-11-10 17:00:06, 299, 309525032052119, 309525032051591, 528
>
>
> 31, 2011-11-14 07:40:09, 299, 2608, 64, 2544
> 31, 2011-11-14 07:45:11, 302, 321074668856640, 321074668848076, 8564