The rrd files contain the historical data. If you delete them, the historical data will go away.

This is the only thing that would make the historical data go away, the rrds being removed and then automatically recreated.

As for that log entry, it seems like your system’s clock is going backwards sometimes :

1603020715 when last update time is 1603021928

You were attempting to insert data with the timestamp:

Sunday, 18 October 2020, 11:31:55 AM GMT

But the previous data inserted was at time:

Sunday, 18 October 2020, 11:52:08 AM GMT

There’s some super weird stuff going on with that system.

Adam.

From: Gordon Cheng (gocheng) <gocheng@cisco.com>
Sent: 22 October 2020 18:33
To: Observium <observium@observium.org>; Adam Armstrong <adama@observium.org>
Subject: Graphs keep reset randomly and losing previous history

Hi Adam and Observium team:

We recently started having an issue with our Observium (20.9.10749) that the graphs for different devices would restart and all its previous history would be gone:

And we usually see the following ‘messages’ under /var/log around that time:

Oct 18 05:07:13 sjc-observium-1 rrdcached[1495]: queue_thread_main: rrd_update_r (/opt/observium/rrd/atl-wan04/status.rrd) failed with status -1. (/opt/observium/rrd/atl-wan04/status.rrd: illegal attempt to update using time 1603020715 when last update time is 1603021928 (minimum one second step))

Oct 19 05:12:05 sjc-observium-1 rrdcached[14841]: queue_thread_main: rrd_update_r (/opt/observium/rrd/atl-wan04/status.rrd) failed with status -1. (/opt/observium/rrd/atl-wan04/status.rrd: illegal attempt to update using time 1603106221 when last update time is 1603108322 (minimum one second step))

Oct 20 05:32:47 sjc-observium-1 rrdcached[14471]: queue_thread_main: rrd_update_r (/opt/observium/rrd/atl-wan04/status.rrd) failed with status -1. (/opt/observium/rrd/atl-wan04/status.rrd: illegal attempt to update using time 1603194156 when last update time is 1603194737 (minimum one second step))

We have tried restarting the rrdcached process, and deleting the rrdcached files which are not updated for some time (with the command “find * -type f -mtime +5 -delete” under the rrd directory).  But they don’t help much.

Do you have any suggestions how we can further investigate and troubleshoot this?

Thanks.

- Gordon