Hi,
Thanks for that - however I find this interesting, the RRDs are a different size now compared to the restored versions:
For a normal port:
1,078,896 port-10148.rrd.restore (the backup/restored version)
1,739,144 port-10148.rrd (the empty created at midnight version)
For netstats-icmp:
1,218,600 netstats-icmp.rrd.restore (original)
3,013,472 netstats-icmp.rrd (new empty one)
So an increase for every file occurred at midnight, we have nothing else running on this server, so no reason for something to go and pick out all similar hardware and delete their rrds.
I then checked the rrd content and found a large difference in the rra/row sizes and the number of actual rra’s:
# rrdtool info port-10148.rrd.restore | grep .rows <- the smaller restored version from yesterday
rra[0].rows = 2016
rra[1].rows = 2976
rra[2].rows = 1440
rra[3].rows = 1440
rra[4].rows = 1440
rra[5].rows = 360
rra[6].rows = 1440
rra[7].rows = 1440
rra[8].rows = 360
rra[9].rows = 1440
# rrdtool info port-10148.rrd | grep .rows <- the larger created-at-midnight version
rra[0].rows = 2400
rra[1].rows = 1200
rra[2].rows = 1200
rra[3].rows = 1200
rra[4].rows = 600
rra[5].rows = 700
rra[6].rows = 775
rra[7].rows = 797
I’ve checked further back in time and the files have been the same (smaller) size for as far back as our backups go (several months) – so something caused it to change the layout of the rrd file, only for these specific hosts, overnight.
I’m a bit lost with this one, any ideas?
From: observium [
mailto:observium-bounces@observium.org]
On Behalf Of Adam Armstrong
Sent: 12 January 2014 11:51
To: Observium Network Observation System
Subject: Re: [Observium] Sudden loss of history
Observium doesn't have the ability to do this. Once the RRDs exist, it can't recreate them. Either they were removed or deleted, and were then recreated
by observium.
Adam.
Robert Williams
Custodian Data Centre
Email: Robert@CustodianDC.com