Hello,
I noticed that Observium started to use up lots of storage space. From ~200Gb it started to use ~400Gbs. Found interested thing, that one device RRD folder tooks ~200GBs instead of ~4-5Gbs. I'm always running housekeeping scripts in cron:
# Run housekeeping script daily for syslog, eventlog and alert log 13 5 * * * root /opt/observium/housekeeping.php -ysel >> /dev/null 2>&1
# Run housekeeping script daily for rrds, ports, orphaned entries in the database and performance data 47 4 * * * root /opt/observium/housekeeping.php -yrptib >> /dev/null 2>&1
But seems it doesnt help, and housekeeping log doesn't show errors.
Could someone advice how to clear that files in RRD folder according to housekeeping settings which are here:
infra [root@observ001prvitx logs]# cat /opt/observium/includes/defaults.inc.php | grep housekeep
$config['housekeeping']['syslog']['age'] = '1M'; // Maximum age of syslog entries; 0 to disable $config['housekeeping']['eventlog']['age'] = '3M'; // Maximum age of event log entries; 0 to disable $config['housekeeping']['alertlog']['age'] = '3M'; // Maximum age of alert log entries; 0 to disable $config['housekeeping']['authlog']['age'] = '6M'; // Maximum age of authlog entries; 0 to disable $config['housekeeping']['inventory']['age'] = '1M'; // Maximum age of deleted inventory entries; 0 to disable $config['housekeeping']['deleted_ports']['age'] = '1M'; // Maximum age of deleted ports before automatically purging; 0 to disable $config['housekeeping']['rrd']['age'] = '2M'; // Maximum age of unused rrd files before automatically purging; 0 to disable $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
afaik M = month ?
Hey, Well your housekeeping settings are to not delete RRDs unless they are not updated for 2 months. Did this one device with 200GB RRDs have files older than 2 months? If not then none of those RRDs will be deleted for now. /Markus
Den ons 6 nov. 2019 kl 10:45 skrev Edvinas Kairys via observium < observium@observium.org>:
Hello,
I noticed that Observium started to use up lots of storage space. From ~200Gb it started to use ~400Gbs. Found interested thing, that one device RRD folder tooks ~200GBs instead of ~4-5Gbs. I'm always running housekeeping scripts in cron:
# Run housekeeping script daily for syslog, eventlog and alert log 13 5 * * * root /opt/observium/housekeeping.php -ysel >> /dev/null 2>&1
# Run housekeeping script daily for rrds, ports, orphaned entries in the database and performance data 47 4 * * * root /opt/observium/housekeeping.php -yrptib >> /dev/null 2>&1
But seems it doesnt help, and housekeeping log doesn't show errors.
Could someone advice how to clear that files in RRD folder according to housekeeping settings which are here:
infra [root@observ001prvitx logs]# cat /opt/observium/includes/defaults.inc.php | grep housekeep
$config['housekeeping']['syslog']['age'] = '1M'; // Maximum age of syslog entries; 0 to disable $config['housekeeping']['eventlog']['age'] = '3M'; // Maximum age of event log entries; 0 to disable $config['housekeeping']['alertlog']['age'] = '3M'; // Maximum age of alert log entries; 0 to disable $config['housekeeping']['authlog']['age'] = '6M'; // Maximum age of authlog entries; 0 to disable $config['housekeeping']['inventory']['age'] = '1M'; // Maximum age of deleted inventory entries; 0 to disable $config['housekeeping']['deleted_ports']['age'] = '1M'; // Maximum age of deleted ports before automatically purging; 0 to disable $config['housekeeping']['rrd']['age'] = '2M'; // Maximum age of unused rrd files before automatically purging; 0 to disable $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
afaik M = month ? _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
thank you,
Something is very strange space in that folder started to increase rapidly ~3months ago.
i even can't to a list of files: of that particular folder.
Now i ran the housekeeping script separately in a screen mode, and it's already running ~3hours in that particular folder and still deleting items:
[2019/11/06 13:52:21 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873343.rrd modification time is 2019-10-07 08:22:09 - deleting [2019/11/06 13:52:24 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871338.rrd modification time is 2019-10-07 07:21:54 - deleting [2019/11/06 13:52:26 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6878336.rrd modification time is 2019-10-07 08:21:50 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877497.rrd modification time is 2019-10-07 08:22:02 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867753.rrd modification time is 2019-10-07 04:22:05 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871233.rrd modification time is 2019-10-07 07:21:48 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867316.rrd modification time is 2019-10-07 04:22:09 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873301.rrd modification time is 2019-10-07 08:22:06 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6865505.rrd modification time is 2019-10-07 04:22:00 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873140.rrd modification time is 2019-10-07 08:21:58 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877296.rrd modification time is 2019-10-07 08:21:52 - deleting
etc, etc..
On Wed, Nov 6, 2019 at 1:25 PM Markus Klock markus@best-practice.se wrote:
Hey, Well your housekeeping settings are to not delete RRDs unless they are not updated for 2 months. Did this one device with 200GB RRDs have files older than 2 months? If not then none of those RRDs will be deleted for now. /Markus
Den ons 6 nov. 2019 kl 10:45 skrev Edvinas Kairys via observium < observium@observium.org>:
Hello,
I noticed that Observium started to use up lots of storage space. From ~200Gb it started to use ~400Gbs. Found interested thing, that one device RRD folder tooks ~200GBs instead of ~4-5Gbs. I'm always running housekeeping scripts in cron:
# Run housekeeping script daily for syslog, eventlog and alert log 13 5 * * * root /opt/observium/housekeeping.php -ysel >> /dev/null 2>&1
# Run housekeeping script daily for rrds, ports, orphaned entries in the database and performance data 47 4 * * * root /opt/observium/housekeeping.php -yrptib >> /dev/null 2>&1
But seems it doesnt help, and housekeeping log doesn't show errors.
Could someone advice how to clear that files in RRD folder according to housekeeping settings which are here:
infra [root@observ001prvitx logs]# cat /opt/observium/includes/defaults.inc.php | grep housekeep
$config['housekeeping']['syslog']['age'] = '1M'; // Maximum age of syslog entries; 0 to disable $config['housekeeping']['eventlog']['age'] = '3M'; // Maximum age of event log entries; 0 to disable $config['housekeeping']['alertlog']['age'] = '3M'; // Maximum age of alert log entries; 0 to disable $config['housekeeping']['authlog']['age'] = '6M'; // Maximum age of authlog entries; 0 to disable $config['housekeeping']['inventory']['age'] = '1M'; // Maximum age of deleted inventory entries; 0 to disable $config['housekeeping']['deleted_ports']['age'] = '1M'; // Maximum age of deleted ports before automatically purging; 0 to disable $config['housekeeping']['rrd']['age'] = '2M'; // Maximum age of unused rrd files before automatically purging; 0 to disable $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
afaik M = month ? _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
managed to see count of files:
so, in mirrored devices there are this count of files:
infra [root@observ001prvitx rrd]# find jaynet01a -type f | wc -l 13249
and in the problematic one:
infra [root@observ001prvitx rrd]# find jaynet01b -type f | wc -l 995349
On Wed, Nov 6, 2019 at 3:53 PM Edvinas Kairys edvinas.email@gmail.com wrote:
thank you,
Something is very strange space in that folder started to increase rapidly ~3months ago.
i even can't to a list of files: of that particular folder.
Now i ran the housekeeping script separately in a screen mode, and it's already running ~3hours in that particular folder and still deleting items:
[2019/11/06 13:52:21 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873343.rrd modification time is 2019-10-07 08:22:09 - deleting [2019/11/06 13:52:24 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871338.rrd modification time is 2019-10-07 07:21:54 - deleting [2019/11/06 13:52:26 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6878336.rrd modification time is 2019-10-07 08:21:50 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877497.rrd modification time is 2019-10-07 08:22:02 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867753.rrd modification time is 2019-10-07 04:22:05 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871233.rrd modification time is 2019-10-07 07:21:48 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867316.rrd modification time is 2019-10-07 04:22:09 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873301.rrd modification time is 2019-10-07 08:22:06 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6865505.rrd modification time is 2019-10-07 04:22:00 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873140.rrd modification time is 2019-10-07 08:21:58 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877296.rrd modification time is 2019-10-07 08:21:52 - deleting
etc, etc..
On Wed, Nov 6, 2019 at 1:25 PM Markus Klock markus@best-practice.se wrote:
Hey, Well your housekeeping settings are to not delete RRDs unless they are not updated for 2 months. Did this one device with 200GB RRDs have files older than 2 months? If not then none of those RRDs will be deleted for now. /Markus
Den ons 6 nov. 2019 kl 10:45 skrev Edvinas Kairys via observium < observium@observium.org>:
Hello,
I noticed that Observium started to use up lots of storage space. From ~200Gb it started to use ~400Gbs. Found interested thing, that one device RRD folder tooks ~200GBs instead of ~4-5Gbs. I'm always running housekeeping scripts in cron:
# Run housekeeping script daily for syslog, eventlog and alert log 13 5 * * * root /opt/observium/housekeeping.php -ysel >> /dev/null 2>&1
# Run housekeeping script daily for rrds, ports, orphaned entries in the database and performance data 47 4 * * * root /opt/observium/housekeeping.php -yrptib >> /dev/null 2>&1
But seems it doesnt help, and housekeeping log doesn't show errors.
Could someone advice how to clear that files in RRD folder according to housekeeping settings which are here:
infra [root@observ001prvitx logs]# cat /opt/observium/includes/defaults.inc.php | grep housekeep
$config['housekeeping']['syslog']['age'] = '1M'; // Maximum age of syslog entries; 0 to disable $config['housekeeping']['eventlog']['age'] = '3M'; // Maximum age of event log entries; 0 to disable $config['housekeeping']['alertlog']['age'] = '3M'; // Maximum age of alert log entries; 0 to disable $config['housekeeping']['authlog']['age'] = '6M'; // Maximum age of authlog entries; 0 to disable $config['housekeeping']['inventory']['age'] = '1M'; // Maximum age of deleted inventory entries; 0 to disable $config['housekeeping']['deleted_ports']['age'] = '1M'; // Maximum age of deleted ports before automatically purging; 0 to disable $config['housekeeping']['rrd']['age'] = '2M'; // Maximum age of unused rrd files before automatically purging; 0 to disable $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
afaik M = month ? _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
would i be safe to delete alert-*.rrd files ?
afaik these files are not related to interfaces traffic graph history ?
Thank you
On Wed, Nov 6, 2019 at 4:00 PM Edvinas Kairys edvinas.email@gmail.com wrote:
managed to see count of files:
so, in mirrored devices there are this count of files:
infra [root@observ001prvitx rrd]# find jaynet01a -type f | wc -l 13249
and in the problematic one:
infra [root@observ001prvitx rrd]# find jaynet01b -type f | wc -l 995349
On Wed, Nov 6, 2019 at 3:53 PM Edvinas Kairys edvinas.email@gmail.com wrote:
thank you,
Something is very strange space in that folder started to increase rapidly ~3months ago.
i even can't to a list of files: of that particular folder.
Now i ran the housekeeping script separately in a screen mode, and it's already running ~3hours in that particular folder and still deleting items:
[2019/11/06 13:52:21 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873343.rrd modification time is 2019-10-07 08:22:09 - deleting [2019/11/06 13:52:24 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871338.rrd modification time is 2019-10-07 07:21:54 - deleting [2019/11/06 13:52:26 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6878336.rrd modification time is 2019-10-07 08:21:50 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877497.rrd modification time is 2019-10-07 08:22:02 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867753.rrd modification time is 2019-10-07 04:22:05 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871233.rrd modification time is 2019-10-07 07:21:48 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867316.rrd modification time is 2019-10-07 04:22:09 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873301.rrd modification time is 2019-10-07 08:22:06 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6865505.rrd modification time is 2019-10-07 04:22:00 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873140.rrd modification time is 2019-10-07 08:21:58 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877296.rrd modification time is 2019-10-07 08:21:52 - deleting
etc, etc..
On Wed, Nov 6, 2019 at 1:25 PM Markus Klock markus@best-practice.se wrote:
Hey, Well your housekeeping settings are to not delete RRDs unless they are not updated for 2 months. Did this one device with 200GB RRDs have files older than 2 months? If not then none of those RRDs will be deleted for now. /Markus
Den ons 6 nov. 2019 kl 10:45 skrev Edvinas Kairys via observium < observium@observium.org>:
Hello,
I noticed that Observium started to use up lots of storage space. From ~200Gb it started to use ~400Gbs. Found interested thing, that one device RRD folder tooks ~200GBs instead of ~4-5Gbs. I'm always running housekeeping scripts in cron:
# Run housekeeping script daily for syslog, eventlog and alert log 13 5 * * * root /opt/observium/housekeeping.php -ysel >> /dev/null 2>&1
# Run housekeeping script daily for rrds, ports, orphaned entries in the database and performance data 47 4 * * * root /opt/observium/housekeeping.php -yrptib >> /dev/null 2>&1
But seems it doesnt help, and housekeeping log doesn't show errors.
Could someone advice how to clear that files in RRD folder according to housekeeping settings which are here:
infra [root@observ001prvitx logs]# cat /opt/observium/includes/defaults.inc.php | grep housekeep
$config['housekeeping']['syslog']['age'] = '1M'; // Maximum age of syslog entries; 0 to disable $config['housekeeping']['eventlog']['age'] = '3M'; // Maximum age of event log entries; 0 to disable $config['housekeeping']['alertlog']['age'] = '3M'; // Maximum age of alert log entries; 0 to disable $config['housekeeping']['authlog']['age'] = '6M'; // Maximum age of authlog entries; 0 to disable $config['housekeeping']['inventory']['age'] = '1M'; // Maximum age of deleted inventory entries; 0 to disable $config['housekeeping']['deleted_ports']['age'] = '1M'; // Maximum age of deleted ports before automatically purging; 0 to disable $config['housekeeping']['rrd']['age'] = '2M'; // Maximum age of unused rrd files before automatically purging; 0 to disable $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
afaik M = month ? _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
They're just the availability data for an alert entry.
If your device is creating lots of alert entries, I assume they're port entries and it's creating and destroying lots of temporary interfaces for some reason.
If this is happening, you should block these interfaces from being discovered by filtering them out using the relevant config options.
Adam.
Sent from BlueMail
On 6 Nov 2019, 14:24, at 14:24, Edvinas Kairys via observium observium@observium.org wrote:
would i be safe to delete alert-*.rrd files ?
afaik these files are not related to interfaces traffic graph history ?
Thank you
On Wed, Nov 6, 2019 at 4:00 PM Edvinas Kairys edvinas.email@gmail.com wrote:
managed to see count of files:
so, in mirrored devices there are this count of files:
infra [root@observ001prvitx rrd]# find jaynet01a -type f | wc -l 13249
and in the problematic one:
infra [root@observ001prvitx rrd]# find jaynet01b -type f | wc -l 995349
On Wed, Nov 6, 2019 at 3:53 PM Edvinas Kairys
wrote:
thank you,
Something is very strange space in that folder started to increase rapidly ~3months ago.
i even can't to a list of files: of that particular folder.
Now i ran the housekeeping script separately in a screen mode, and
it's
already running ~3hours in that particular folder and still deleting
items:
[2019/11/06 13:52:21 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873343.rrd modification time is 2019-10-07 08:22:09 - deleting [2019/11/06 13:52:24 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871338.rrd modification time is 2019-10-07 07:21:54 - deleting [2019/11/06 13:52:26 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6878336.rrd modification time is 2019-10-07 08:21:50 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877497.rrd modification time is 2019-10-07 08:22:02 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867753.rrd modification time is 2019-10-07 04:22:05 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871233.rrd modification time is 2019-10-07 07:21:48 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867316.rrd modification time is 2019-10-07 04:22:09 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873301.rrd modification time is 2019-10-07 08:22:06 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6865505.rrd modification time is 2019-10-07 04:22:00 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873140.rrd modification time is 2019-10-07 08:21:58 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877296.rrd modification time is 2019-10-07 08:21:52 - deleting
etc, etc..
On Wed, Nov 6, 2019 at 1:25 PM Markus Klock
wrote:
Hey, Well your housekeeping settings are to not delete RRDs unless they
are
not updated for 2 months. Did this one device with 200GB RRDs have files older than 2 months? If not then none of those RRDs will be deleted for now. /Markus
Den ons 6 nov. 2019 kl 10:45 skrev Edvinas Kairys via observium < observium@observium.org>:
Hello,
I noticed that Observium started to use up lots of storage space.
From
~200Gb it started to use ~400Gbs. Found interested thing, that one
device
RRD folder tooks ~200GBs instead of ~4-5Gbs. I'm always running housekeeping scripts in cron:
# Run housekeeping script daily for syslog, eventlog and alert log 13 5 * * * root /opt/observium/housekeeping.php -ysel >> /dev/null
2>&1
# Run housekeeping script daily for rrds, ports, orphaned entries
in
the database and performance data 47 4 * * * root /opt/observium/housekeeping.php -yrptib >>
/dev/null
2>&1
But seems it doesnt help, and housekeeping log doesn't show
errors.
Could someone advice how to clear that files in RRD folder
according to
housekeeping settings which are here:
infra [root@observ001prvitx logs]# cat /opt/observium/includes/defaults.inc.php | grep housekeep
$config['housekeeping']['syslog']['age'] = '1M'; //
Maximum age
of syslog entries; 0 to disable $config['housekeeping']['eventlog']['age'] = '3M'; //
Maximum age
of event log entries; 0 to disable $config['housekeeping']['alertlog']['age'] = '3M'; //
Maximum age
of alert log entries; 0 to disable $config['housekeeping']['authlog']['age'] = '6M'; //
Maximum age
of authlog entries; 0 to disable $config['housekeeping']['inventory']['age'] = '1M'; //
Maximum age
of deleted inventory entries; 0 to disable $config['housekeeping']['deleted_ports']['age'] = '1M'; //
Maximum age
of deleted ports before automatically purging; 0 to disable $config['housekeeping']['rrd']['age'] = '2M'; //
Maximum age
of unused rrd files before automatically purging; 0 to disable $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete
.rrd
files that are not valid RRD files (eg created with a full disk)
afaik M = month ? _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
as i managed to see the type of that files, so yeah, they're all alert-*.rrd ones. The housekeeping scripts didn't delete them (because i assume the scripts sees them as the more recent than the retention time)
infra [root@observ001prvitx rrd]# find jaynet01b -type f | sort | uniq -c | awk '{print $2}' | awk -F"/" '{print $2}' | awk -F"-" '{print $1}' | sort | uniq -c | sort
938 port
993164 alert
i think i will try to delete them. Btw, how to know which interfaces caused them to be recreated to be able to disable ? i'm attaching the graph which shows how the space was increasing in 4months period:
[image: image.png]
On Wed, Nov 6, 2019 at 5:22 PM Adam Armstrong adama@memetic.org wrote:
They're just the availability data for an alert entry.
If your device is creating lots of alert entries, I assume they're port entries and it's creating and destroying lots of temporary interfaces for some reason.
If this is happening, you should block these interfaces from being discovered by filtering them out using the relevant config options.
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=15726 On 6 Nov 2019, at 14:24, Edvinas Kairys via observium < observium@observium.org> wrote:
would i be safe to delete alert-*.rrd files ?
afaik these files are not related to interfaces traffic graph history ?
Thank you
On Wed, Nov 6, 2019 at 4:00 PM Edvinas Kairys < edvinas.email@gmail.com> wrote:
managed to see count of files:
so, in mirrored devices there are this count of files:
infra [root@observ001prvitx rrd]# find jaynet01a -type f | wc -l 13249
and in the problematic one:
infra [root@observ001prvitx rrd]# find jaynet01b -type f | wc -l 995349
On Wed, Nov 6, 2019 at 3:53 PM Edvinas Kairys < edvinas.email@gmail.com> wrote:
thank you,
Something is very strange space in that folder started to increase rapidly ~3months ago.
i even can't to a list of files: of that particular folder.
Now i ran the housekeeping script separately in a screen mode, and it's already running ~3hours in that particular folder and still deleting items:
[2019/11/06 13:52:21 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873343.rrd modification time is 2019-10-07 08:22:09 - deleting [2019/11/06 13:52:24 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871338.rrd modification time is 2019-10-07 07:21:54 - deleting [2019/11/06 13:52:26 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6878336.rrd modification time is 2019-10-07 08:21:50 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877497.rrd modification time is 2019-10-07 08:22:02 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867753.rrd modification time is 2019-10-07 04:22:05 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871233.rrd modification time is 2019-10-07 07:21:48 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867316.rrd modification time is 2019-10-07 04:22:09 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873301.rrd modification time is 2019-10-07 08:22:06 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6865505.rrd modification time is 2019-10-07 04:22:00 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873140.rrd modification time is 2019-10-07 08:21:58 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877296.rrd modification time is 2019-10-07 08:21:52 - deleting
etc, etc..
On Wed, Nov 6, 2019 at 1:25 PM Markus Klock < markus@best-practice.se> wrote:
Hey, Well your housekeeping settings are to not delete RRDs unless they are not updated for 2 months. Did this one device with 200GB RRDs have files older than 2 months? If not then none of those RRDs will be deleted for now. /Markus
Den ons 6 nov. 2019 kl 10:45 skrev Edvinas Kairys via observium < observium@observium.org>:
Hello,
I noticed that Observium started to use up lots of storage space. From ~200Gb it started to use ~400Gbs. Found interested thing, that one device RRD folder tooks ~200GBs instead of ~4-5Gbs. I'm always running housekeeping scripts in cron:
# Run housekeeping script daily for syslog, eventlog and alert log 13 5 * * * root /opt/observium/housekeeping.php -ysel >> /dev/null 2>&1
# Run housekeeping script daily for rrds, ports, orphaned entries in the database and performance data 47 4 * * * root /opt/observium/housekeeping.php -yrptib >> /dev/null 2>&1
But seems it doesnt help, and housekeeping log doesn't show errors.
Could someone advice how to clear that files in RRD folder according to housekeeping settings which are here:
infra [root@observ001prvitx logs]# cat /opt/observium/includes/defaults.inc.php | grep housekeep
$config['housekeeping']['syslog']['age'] = '1M'; // Maximum age of syslog entries; 0 to disable $config['housekeeping']['eventlog']['age'] = '3M'; // Maximum age of event log entries; 0 to disable $config['housekeeping']['alertlog']['age'] = '3M'; // Maximum age of alert log entries; 0 to disable $config['housekeeping']['authlog']['age'] = '6M'; // Maximum age of authlog entries; 0 to disable $config['housekeeping']['inventory']['age'] = '1M'; // Maximum age of deleted inventory entries; 0 to disable $config['housekeeping']['deleted_ports']['age'] = '1M'; // Maximum age of deleted ports before automatically purging; 0 to disable $config['housekeeping']['rrd']['age'] = '2M'; // Maximum age of unused rrd files before automatically purging; 0 to disable $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
afaik M = month ? _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
You'll probably see in the eventlog the interfaces being created and destroyed.
And it might even be obvious from the live interface list.
I saw this recently with an appliance, creating hundreds of temporary interfaces.
Adam.
Sent from BlueMail
On 6 Nov 2019, 15:34, at 15:34, Edvinas Kairys via observium observium@observium.org wrote:
as i managed to see the type of that files, so yeah, they're all alert-*.rrd ones. The housekeeping scripts didn't delete them (because i assume the scripts sees them as the more recent than the retention time)
infra [root@observ001prvitx rrd]# find jaynet01b -type f | sort | uniq -c | awk '{print $2}' | awk -F"/" '{print $2}' | awk -F"-" '{print $1}' | sort | uniq -c | sort
938 port
993164 alert
i think i will try to delete them. Btw, how to know which interfaces caused them to be recreated to be able to disable ? i'm attaching the graph which shows how the space was increasing in 4months period:
[image: image.png]
On Wed, Nov 6, 2019 at 5:22 PM Adam Armstrong adama@memetic.org wrote:
They're just the availability data for an alert entry.
If your device is creating lots of alert entries, I assume they're
port
entries and it's creating and destroying lots of temporary interfaces
for
some reason.
If this is happening, you should block these interfaces from being discovered by filtering them out using the relevant config options.
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=15726 On 6 Nov 2019, at 14:24, Edvinas Kairys via observium < observium@observium.org> wrote:
would i be safe to delete alert-*.rrd files ?
afaik these files are not related to interfaces traffic graph
history ?
Thank you
On Wed, Nov 6, 2019 at 4:00 PM Edvinas Kairys <
edvinas.email@gmail.com>
wrote:
managed to see count of files:
so, in mirrored devices there are this count of files:
infra [root@observ001prvitx rrd]# find jaynet01a -type f | wc -l 13249
and in the problematic one:
infra [root@observ001prvitx rrd]# find jaynet01b -type f | wc -l 995349
On Wed, Nov 6, 2019 at 3:53 PM Edvinas Kairys <
edvinas.email@gmail.com>
wrote:
thank you,
Something is very strange space in that folder started to increase rapidly ~3months ago.
i even can't to a list of files: of that particular folder.
Now i ran the housekeeping script separately in a screen mode, and
it's
already running ~3hours in that particular folder and still
deleting items:
[2019/11/06 13:52:21 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873343.rrd modification time
is
2019-10-07 08:22:09 - deleting [2019/11/06 13:52:24 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871338.rrd modification time
is
2019-10-07 07:21:54 - deleting [2019/11/06 13:52:26 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6878336.rrd modification time
is
2019-10-07 08:21:50 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877497.rrd modification time
is
2019-10-07 08:22:02 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867753.rrd modification time
is
2019-10-07 04:22:05 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871233.rrd modification time
is
2019-10-07 07:21:48 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867316.rrd modification time
is
2019-10-07 04:22:09 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873301.rrd modification time
is
2019-10-07 08:22:06 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6865505.rrd modification time
is
2019-10-07 04:22:00 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873140.rrd modification time
is
2019-10-07 08:21:58 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877296.rrd modification time
is
2019-10-07 08:21:52 - deleting
etc, etc..
On Wed, Nov 6, 2019 at 1:25 PM Markus Klock <
markus@best-practice.se>
wrote:
Hey, Well your housekeeping settings are to not delete RRDs unless
they are
not updated for 2 months. Did this one device with 200GB RRDs have files older than 2
months?
If not then none of those RRDs will be deleted for now. /Markus
Den ons 6 nov. 2019 kl 10:45 skrev Edvinas Kairys via observium < observium@observium.org>:
> Hello, > > I noticed that Observium started to use up lots of storage
space.
> From ~200Gb it started to use ~400Gbs. Found interested thing,
that one
> device RRD folder tooks ~200GBs instead of ~4-5Gbs. I'm always
running
> housekeeping scripts in cron: > > # Run housekeeping script daily for syslog, eventlog and alert
log
> 13 5 * * * root /opt/observium/housekeeping.php -ysel >>
/dev/null
> 2>&1 > > # Run housekeeping script daily for rrds, ports, orphaned
entries in
> the database and performance data > 47 4 * * * root /opt/observium/housekeeping.php -yrptib >>
/dev/null
> 2>&1 > > But seems it doesnt help, and housekeeping log doesn't show
errors.
> > Could someone advice how to clear that files in RRD folder
according
> to housekeeping settings which are here: > > infra [root@observ001prvitx logs]# cat > /opt/observium/includes/defaults.inc.php | grep housekeep > > $config['housekeeping']['syslog']['age'] = '1M'; //
Maximum
> age of syslog entries; 0 to disable > $config['housekeeping']['eventlog']['age'] = '3M'; //
Maximum
> age of event log entries; 0 to disable > $config['housekeeping']['alertlog']['age'] = '3M'; //
Maximum
> age of alert log entries; 0 to disable > $config['housekeeping']['authlog']['age'] = '6M'; //
Maximum
> age of authlog entries; 0 to disable > $config['housekeeping']['inventory']['age'] = '1M'; //
Maximum
> age of deleted inventory entries; 0 to disable > $config['housekeeping']['deleted_ports']['age'] = '1M'; //
Maximum
> age of deleted ports before automatically purging; 0 to disable > $config['housekeeping']['rrd']['age'] = '2M'; //
Maximum
> age of unused rrd files before automatically purging; 0 to
disable
> $config['housekeeping']['rrd']['invalid'] = TRUE; //
Delete
> .rrd files that are not valid RRD files (eg created with a full
disk)
> > afaik M = month ? > _______________________________________________ > observium mailing list > observium@observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Seems like increasing default snmp timeout value from 1s to 3s helped (atleast for now).
Afaik the global value for it is 1s. Where is the default setting of timeout ? I would like to increase the timeout value globally from 1s to 5s.
On Wed, Nov 6, 2019 at 5:53 PM Adam Armstrong adama@memetic.org wrote:
You'll probably see in the eventlog the interfaces being created and destroyed.
And it might even be obvious from the live interface list.
I saw this recently with an appliance, creating hundreds of temporary interfaces.
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=15726 On 6 Nov 2019, at 15:34, Edvinas Kairys via observium < observium@observium.org> wrote:
as i managed to see the type of that files, so yeah, they're all alert-*.rrd ones. The housekeeping scripts didn't delete them (because i assume the scripts sees them as the more recent than the retention time)
infra [root@observ001prvitx rrd]# find jaynet01b -type f | sort | uniq -c | awk '{print $2}' | awk -F"/" '{print $2}' | awk -F"-" '{print $1}' | sort | uniq -c | sort
938 port993164 alert
i think i will try to delete them. Btw, how to know which interfaces caused them to be recreated to be able to disable ? i'm attaching the graph which shows how the space was increasing in 4months period:
[image: image.png]
On Wed, Nov 6, 2019 at 5:22 PM Adam Armstrong < adama@memetic.org> wrote:
They're just the availability data for an alert entry.
If your device is creating lots of alert entries, I assume they're port entries and it's creating and destroying lots of temporary interfaces for some reason.
If this is happening, you should block these interfaces from being discovered by filtering them out using the relevant config options.
Adam.
Sent from BlueMail http://www.bluemail.me/r?b=15726 On 6 Nov 2019, at 14:24, Edvinas Kairys via observium < observium@observium.org> wrote:
would i be safe to delete alert-*.rrd files ?
afaik these files are not related to interfaces traffic graph history ?
Thank you
On Wed, Nov 6, 2019 at 4:00 PM Edvinas Kairys < edvinas.email@gmail.com> wrote:
managed to see count of files:
so, in mirrored devices there are this count of files:
infra [root@observ001prvitx rrd]# find jaynet01a -type f | wc -l 13249
and in the problematic one:
infra [root@observ001prvitx rrd]# find jaynet01b -type f | wc -l 995349
On Wed, Nov 6, 2019 at 3:53 PM Edvinas Kairys < edvinas.email@gmail.com> wrote:
thank you,
Something is very strange space in that folder started to increase rapidly ~3months ago.
i even can't to a list of files: of that particular folder.
Now i ran the housekeeping script separately in a screen mode, and it's already running ~3hours in that particular folder and still deleting items:
[2019/11/06 13:52:21 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873343.rrd modification time is 2019-10-07 08:22:09 - deleting [2019/11/06 13:52:24 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871338.rrd modification time is 2019-10-07 07:21:54 - deleting [2019/11/06 13:52:26 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6878336.rrd modification time is 2019-10-07 08:21:50 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877497.rrd modification time is 2019-10-07 08:22:02 - deleting [2019/11/06 13:52:28 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867753.rrd modification time is 2019-10-07 04:22:05 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6871233.rrd modification time is 2019-10-07 07:21:48 - deleting [2019/11/06 13:52:29 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6867316.rrd modification time is 2019-10-07 04:22:09 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873301.rrd modification time is 2019-10-07 08:22:06 - deleting [2019/11/06 13:52:31 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6865505.rrd modification time is 2019-10-07 04:22:00 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6873140.rrd modification time is 2019-10-07 08:21:58 - deleting [2019/11/06 13:52:32 +0000] housekeeping.php(30530): File /opt/observium/rrd/jaynet01b/alert-6877296.rrd modification time is 2019-10-07 08:21:52 - deleting
etc, etc..
On Wed, Nov 6, 2019 at 1:25 PM Markus Klock < markus@best-practice.se> wrote:
> Hey, > Well your housekeeping settings are to not delete RRDs unless they > are not updated for 2 months. > Did this one device with 200GB RRDs have files older than 2 months? > If not then none of those RRDs will be deleted for now. > /Markus > > Den ons 6 nov. 2019 kl 10:45 skrev Edvinas Kairys via observium < > observium@observium.org>: > >> Hello, >> >> I noticed that Observium started to use up lots of storage space. >> From ~200Gb it started to use ~400Gbs. Found interested thing, that one >> device RRD folder tooks ~200GBs instead of ~4-5Gbs. I'm always running >> housekeeping scripts in cron: >> >> # Run housekeeping script daily for syslog, eventlog and alert log >> 13 5 * * * root /opt/observium/housekeeping.php -ysel >> /dev/null >> 2>&1 >> >> # Run housekeeping script daily for rrds, ports, orphaned entries >> in the database and performance data >> 47 4 * * * root /opt/observium/housekeeping.php -yrptib >> >> /dev/null 2>&1 >> >> But seems it doesnt help, and housekeeping log doesn't show errors. >> >> Could someone advice how to clear that files in RRD folder >> according to housekeeping settings which are here: >> >> infra [root@observ001prvitx logs]# cat >> /opt/observium/includes/defaults.inc.php | grep housekeep >> >> $config['housekeeping']['syslog']['age'] = '1M'; // Maximum >> age of syslog entries; 0 to disable >> $config['housekeeping']['eventlog']['age'] = '3M'; // Maximum >> age of event log entries; 0 to disable >> $config['housekeeping']['alertlog']['age'] = '3M'; // Maximum >> age of alert log entries; 0 to disable >> $config['housekeeping']['authlog']['age'] = '6M'; // Maximum >> age of authlog entries; 0 to disable >> $config['housekeeping']['inventory']['age'] = '1M'; // Maximum >> age of deleted inventory entries; 0 to disable >> $config['housekeeping']['deleted_ports']['age'] = '1M'; // Maximum >> age of deleted ports before automatically purging; 0 to disable >> $config['housekeeping']['rrd']['age'] = '2M'; // Maximum >> age of unused rrd files before automatically purging; 0 to disable >> $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete >> .rrd files that are not valid RRD files (eg created with a full disk) >> >> afaik M = month ? >> _______________________________________________ >> observium mailing list >> observium@observium.org >> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium >> >
------------------------------observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (3)
-
Adam Armstrong -
Edvinas Kairys -
Markus Klock