Hi,
Appears we have lost all RRD contents (for every possible sensor/data type) although the eventlog data is all still there, with most
recent entires from the 9th.
I have checked all devices now and it specifically affected only
2960G switches and Debian linux (interestingly not Ubuntu linux) hosts.
It’s the subscription version, and currently it’s at r4878. Since it automatically SVNs I will assume that it was at 4877 at midnight
and upgraded to 4878 then.
The cron includes an SVN-up followed by a discovery.php –h all - a few minutes later.
I’ve restored the RRDs from yesterday for all the affected hosts and all the history is back, but I’d love to know why it suddenly
did this? Let me know what other information I can supply to assist.
Cheers!
From: observium [mailto:observium-bounces@observium.org]
On Behalf Of Mike Stupalov
Sent: 12 January 2014 11:21
To: Observium Network Observation System
Subject: Re: [Observium] Sudden loss of history
Hi,
with which revision you updated?
Lost only eventlogs or eventlogs with rrd (graph) datas?
On Sun, Jan 12, 2014 at 1:48 PM, Robert Williams <Robert@custodiandc.com> wrote:
Hi Guys,
At midnight last night during the normal scheduled re-discovery we appear to have lost all historic data for all ports on a large number of our switches. I’m still checking them all but it appears to have affected all Cisco 2960G units (24 or 48 port).
Data collection is working fine, but all history has gone, including all CPU, RAM, temperatures, fans, RPS Power etc.
There are no Eventlog entries for the devices for the last 24 hours, but there are some from a couple of days ago which read like this:
However, as I mentioned these events are from the 9th so a few days ago.
Any ideas please? I can restore the RRDs from backup but I’d like to know why it’s done this otherwise I’ll just be restoring them again tomorrow morning J
Cheers!
Robert Williams
Custodian Data Centre
Email: Robert@CustodianDC.com
http://www.CustodianDC.com