You'll probably see in the eventlog the interfaces being created and destroyed.
And it might even be obvious from the live interface list.
I saw this recently with an appliance, creating hundreds of temporary interfaces.
Adam.
Sent from BlueMail
On 6 Nov 2019, 15:34, at 15:34, Edvinas Kairys via observium observium@observium.org wrote:
as i managed to see the type of that files, so yeah, they're all alert-*.rrd ones. The housekeeping scripts didn't delete them (because i assume the scripts sees them as the more recent than the retention time)
infra [root@observ001prvitx rrd]# find jaynet01b -type f | sort | uniq -c | awk '{print $2}' | awk -F"/" '{print $2}' | awk -F"-" '{print $1}' | sort | uniq -c | sort
938 port
993164 alert
i think i will try to delete them. Btw, how to know which interfaces caused them to be recreated to be able to disable ? i'm attaching the graph which shows how the space was increasing in 4months period:
[image: image.png]
On Wed, Nov 6, 2019 at 5:22 PM Adam Armstrong adama@memetic.org wrote:
They're just the availability data for an alert entry.
If your device is creating lots of alert entries, I assume they're
port
entries and it's creating and destroying lots of temporary interfaces
for
some reason.
If this is happening, you should block these interfaces from being discovered by filtering them out using the relevant config options.
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=15726 On 6 Nov 2019, at 14:24, Edvinas Kairys via observium < observium@observium.org> wrote:
would i be safe to delete alert-*.rrd files ?
afaik these files are not related to interfaces traffic graph
history ?
Thank you
On Wed, Nov 6, 2019 at 4:00 PM Edvinas Kairys <
edvinas.email@gmail.com>
wrote:
managed to see count of files:
so, in mirrored devices there are this count of files:
infra [root@observ001prvitx rrd]# find jaynet01a -type f | wc -l 13249
and in the problematic one:
infra [root@observ001prvitx rrd]# find jaynet01b -type f | wc -l 995349
On Wed, Nov 6, 2019 at 3:53 PM Edvinas Kairys <
edvinas.email@gmail.com>
wrote:
thank you,
Something is very strange space in that folder started to increase rapidly ~3months ago.
i even can't to a list of files: of that particular folder.
Now i ran the housekeeping script separately in a screen mode, and
it's
already running ~3hours in that particular folder and still
deleting items:
[2019/11/06 13:52:21 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873343.rrd modification time
is
2019-10-07 08:22:09 - deleting [2019/11/06 13:52:24 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871338.rrd modification time
is
2019-10-07 07:21:54 - deleting [2019/11/06 13:52:26 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6878336.rrd modification time
is
2019-10-07 08:21:50 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877497.rrd modification time
is
2019-10-07 08:22:02 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867753.rrd modification time
is
2019-10-07 04:22:05 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871233.rrd modification time
is
2019-10-07 07:21:48 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867316.rrd modification time
is
2019-10-07 04:22:09 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873301.rrd modification time
is
2019-10-07 08:22:06 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6865505.rrd modification time
is
2019-10-07 04:22:00 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873140.rrd modification time
is
2019-10-07 08:21:58 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877296.rrd modification time
is
2019-10-07 08:21:52 - deleting
etc, etc..
On Wed, Nov 6, 2019 at 1:25 PM Markus Klock <
markus@best-practice.se>
wrote:
Hey, Well your housekeeping settings are to not delete RRDs unless
they are
not updated for 2 months. Did this one device with 200GB RRDs have files older than 2
months?
If not then none of those RRDs will be deleted for now. /Markus
Den ons 6 nov. 2019 kl 10:45 skrev Edvinas Kairys via observium < observium@observium.org>:
> Hello, > > I noticed that Observium started to use up lots of storage
space.
> From ~200Gb it started to use ~400Gbs. Found interested thing,
that one
> device RRD folder tooks ~200GBs instead of ~4-5Gbs. I'm always
running
> housekeeping scripts in cron: > > # Run housekeeping script daily for syslog, eventlog and alert
log
> 13 5 * * * root /opt/observium/housekeeping.php -ysel >>
/dev/null
> 2>&1 > > # Run housekeeping script daily for rrds, ports, orphaned
entries in
> the database and performance data > 47 4 * * * root /opt/observium/housekeeping.php -yrptib >>
/dev/null
> 2>&1 > > But seems it doesnt help, and housekeeping log doesn't show
errors.
> > Could someone advice how to clear that files in RRD folder
according
> to housekeeping settings which are here: > > infra [root@observ001prvitx logs]# cat > /opt/observium/includes/defaults.inc.php | grep housekeep > > $config['housekeeping']['syslog']['age'] = '1M'; //
Maximum
> age of syslog entries; 0 to disable > $config['housekeeping']['eventlog']['age'] = '3M'; //
Maximum
> age of event log entries; 0 to disable > $config['housekeeping']['alertlog']['age'] = '3M'; //
Maximum
> age of alert log entries; 0 to disable > $config['housekeeping']['authlog']['age'] = '6M'; //
Maximum
> age of authlog entries; 0 to disable > $config['housekeeping']['inventory']['age'] = '1M'; //
Maximum
> age of deleted inventory entries; 0 to disable > $config['housekeeping']['deleted_ports']['age'] = '1M'; //
Maximum
> age of deleted ports before automatically purging; 0 to disable > $config['housekeeping']['rrd']['age'] = '2M'; //
Maximum
> age of unused rrd files before automatically purging; 0 to
disable
> $config['housekeeping']['rrd']['invalid'] = TRUE; //
Delete
> .rrd files that are not valid RRD files (eg created with a full
disk)
> > afaik M = month ? > _______________________________________________ > observium mailing list > observium@observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium