It's not the MySQL logs growing, those are on another disk altogether.  100% of the growth is in the /opt/observium/rrds directory.
I'm trying to implement some system that will let us track individual file growth and/or creation so we can figure out where the space is going... (`find -ls`  + git is as good is the best I can come up with so far)

We don't run the housekeeping script, because we need historical data to stick around for other (non-Observium) processes to parse and make pretty graphs from.
However, running housekeeping with the "-r" and "-T" flags only finds six old no-longer-existing hosts, but those amount to only 10GB (out of 1+TB).

Further to the above, we're OK with using massive amounts of disk space, we just need it to me [more] deterministic.

Actually... is there a way to calculate how big any given RRD file should​ be?  They should have a fully-deterministic upper bound, right?

-Adam


Adam Thompson
Consultant, Infrastructure Services
1593169877849
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athompson@merlin.mb.ca
www.merlin.mb.ca
From: observium <observium-bounces@observium.org> on behalf of Daniel Johansson via observium <observium@observium.org>
Sent: September 27, 2021 09:38
To: observium <observium@observium.org>
Cc: Daniel Johansson <daz@voodoo-people.com>
Subject: Re: [Observium] unending disk space growth?
 
Hi

IF you don't need mysql transaction logs you can use these values as I use them in our production of observium:
binlog_expire_logs_seconds      = 300
max_binlog_size   = 1000M 


If you really don't need them you could probable disable it with setting binlogs value to 1 and log size to 1M or so.

Regards
Daniel


From: "observium" <observium@observium.org>
To: "observium" <observium@observium.org>
Cc: "Flesch, Philipp" <philipp.flesch@gv.mpg.de>
Sent: Monday, September 27, 2021 4:12:01 PM
Subject: Re: [Observium] unending disk space growth?

And – of course – don’t forget to run the Observium housekeeping cron-jobs

 

Regards,

 

Philipp

 

Von: observium <observium-bounces@observium.org> Im Auftrag von Timothy Stoddard via observium
Gesendet: Montag, 27. September 2021 16:10
An: Observium <observium@observium.org>
Cc: Timothy Stoddard <testoddard@ualr.edu>
Betreff: Re: [Observium] unending disk space growth?

 

This is most likely my-sql logs growing.  Check your database directory and look for old logs.  

 

Thanks

--
Tim

 

 

On Fri, Sep 24, 2021 at 10:38 AM Adam Armstrong via observium <observium@observium.org> wrote:

Can you pinpoint the large size to any particular device or group of devices?

 

Rrd dir should not grow indefinitely unless some devices are continually adding and removing entities, and you aren’t running housekeeping to remove old RRDs.

 

Adam.

Sent from my iPhone



On 24 Sep 2021, at 16:20, Adam Thompson via observium <observium@observium.org> wrote:

Since we last rebuilt our Observium instance in late 2019, we've noticed that it's chewing up disk space at a prodigious rate, that we didn't see before.

 

We're seeing a steady ~300BG/month increase in storage needs.  Observium, of course, monitors itself, and the line on the storage graph is a nearly-perfect slope (after adjusting for disk size increases).

 

This came to my attention this morning (again), when we ran out of disk space (again) and I had to expand the volume (again) to continue operating.   (So much for our alerting... oops.)

 

We have added a few new devices, but not 1TB worth, and not consistently and constantly since late 2019.

 

Since RRD files are - in theory - fixed-size, why does /opt/obvservium/rrd keep growing without bound? And how can I predict future growth?  Will it ever stop?

 

Help?  Anyone?

 

-Adam

 

Adam Thompson
Consultant, Infrastructure Services
1593169877849
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athompson@merlin.mb.ca
www.merlin.mb.ca

_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium


_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium