unending disk space growth?
Since we last rebuilt our Observium instance in late 2019, we've noticed that it's chewing up disk space at a prodigious rate, that we didn't see before.
We're seeing a steady ~300BG/month increase in storage needs. Observium, of course, monitors itself, and the line on the storage graph is a nearly-perfect slope (after adjusting for disk size increases).
This came to my attention this morning (again), when we ran out of disk space (again) and I had to expand the volume (again) to continue operating. (So much for our alerting... oops.)
We have added a few new devices, but not 1TB worth, and not consistently and constantly since late 2019.
Since RRD files are - in theory - fixed-size, why does /opt/obvservium/rrd keep growing without bound? And how can I predict future growth? Will it ever stop?
Help? Anyone?
-Adam
Adam Thompson Consultant, Infrastructure Services [1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.camailto:athompson@merlin.mb.ca www.merlin.mb.cahttp://www.merlin.mb.ca/
Can you pinpoint the large size to any particular device or group of devices?
Rrd dir should not grow indefinitely unless some devices are continually adding and removing entities, and you aren’t running housekeeping to remove old RRDs.
Adam.
Sent from my iPhone
On 24 Sep 2021, at 16:20, Adam Thompson via observium observium@observium.org wrote:
Since we last rebuilt our Observium instance in late 2019, we've noticed that it's chewing up disk space at a prodigious rate, that we didn't see before.
We're seeing a steady ~300BG/month increase in storage needs. Observium, of course, monitors itself, and the line on the storage graph is a nearly-perfect slope (after adjusting for disk size increases).
This came to my attention this morning (again), when we ran out of disk space (again) and I had to expand the volume (again) to continue operating. (So much for our alerting... oops.)
We have added a few new devices, but not 1TB worth, and not consistently and constantly since late 2019.
Since RRD files are - in theory - fixed-size, why does /opt/obvservium/rrd keep growing without bound? And how can I predict future growth? Will it ever stop?
Help? Anyone?
-Adam
Adam Thompson Consultant, Infrastructure Services
100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.ca www.merlin.mb.ca _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
This is most likely my-sql logs growing. Check your database directory and look for old logs.
Thanks -- Tim
On Fri, Sep 24, 2021 at 10:38 AM Adam Armstrong via observium < observium@observium.org> wrote:
Can you pinpoint the large size to any particular device or group of devices?
Rrd dir should not grow indefinitely unless some devices are continually adding and removing entities, and you aren’t running housekeeping to remove old RRDs.
Adam.
Sent from my iPhone
On 24 Sep 2021, at 16:20, Adam Thompson via observium < observium@observium.org> wrote:
Since we last rebuilt our Observium instance in late 2019, we've noticed that it's chewing up disk space at a prodigious rate, that we didn't see before.
We're seeing a steady ~300BG/month increase in storage needs. Observium, of course, monitors itself, and the line on the storage graph is a nearly-perfect slope (after adjusting for disk size increases).
This came to my attention this morning (again), when we ran out of disk space (again) and I had to expand the volume (again) to continue operating. (So much for our alerting... oops.)
We have added a few new devices, but not 1TB worth, and not consistently and constantly since late 2019.
Since RRD files are - in theory - fixed-size, why does /opt/obvservium/rrd keep growing without bound? And how can I predict future growth? Will it ever stop?
Help? Anyone?
-Adam
*Adam Thompson* Consultant, Infrastructure Services [image: 1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.ca www.merlin.mb.ca _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
And – of course – don’t forget to run the Observium housekeeping cron-jobs
Regards,
Philipp
Von: observium observium-bounces@observium.org Im Auftrag von Timothy Stoddard via observium Gesendet: Montag, 27. September 2021 16:10 An: Observium observium@observium.org Cc: Timothy Stoddard testoddard@ualr.edu Betreff: Re: [Observium] unending disk space growth?
This is most likely my-sql logs growing. Check your database directory and look for old logs.
Thanks
-- Tim
On Fri, Sep 24, 2021 at 10:38 AM Adam Armstrong via observium <observium@observium.org mailto:observium@observium.org > wrote:
Can you pinpoint the large size to any particular device or group of devices?
Rrd dir should not grow indefinitely unless some devices are continually adding and removing entities, and you aren’t running housekeeping to remove old RRDs.
Adam.
Sent from my iPhone
On 24 Sep 2021, at 16:20, Adam Thompson via observium <observium@observium.org mailto:observium@observium.org > wrote:
Since we last rebuilt our Observium instance in late 2019, we've noticed that it's chewing up disk space at a prodigious rate, that we didn't see before.
We're seeing a steady ~300BG/month increase in storage needs. Observium, of course, monitors itself, and the line on the storage graph is a nearly-perfect slope (after adjusting for disk size increases).
This came to my attention this morning (again), when we ran out of disk space (again) and I had to expand the volume (again) to continue operating. (So much for our alerting... oops.)
We have added a few new devices, but not 1TB worth, and not consistently and constantly since late 2019.
Since RRD files are - in theory - fixed-size, why does /opt/obvservium/rrd keep growing without bound? And how can I predict future growth? Will it ever stop?
Help? Anyone?
-Adam
Adam Thompson Consultant, Infrastructure Services
100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.ca mailto:athompson@merlin.mb.ca www.merlin.mb.ca http://www.merlin.mb.ca/
_______________________________________________ observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi
IF you don't need mysql transaction logs you can use these values as I use them in our production of observium: binlog_expire_logs_seconds = 300 max_binlog_size = 1000M
If you really don't need them you could probable disable it with setting binlogs value to 1 and log size to 1M or so.
Regards Daniel
From: "observium" observium@observium.org To: "observium" observium@observium.org Cc: "Flesch, Philipp" philipp.flesch@gv.mpg.de Sent: Monday, September 27, 2021 4:12:01 PM Subject: Re: [Observium] unending disk space growth?
And – of course – don’t forget to run the Observium housekeeping cron-jobs
Regards,
Philipp
Von: observium observium-bounces@observium.org Im Auftrag von Timothy Stoddard via observium Gesendet: Montag, 27. September 2021 16:10 An: Observium observium@observium.org Cc: Timothy Stoddard testoddard@ualr.edu Betreff: Re: [Observium] unending disk space growth?
This is most likely my-sql logs growing. Check your database directory and look for old logs.
Thanks
It's not the MySQL logs growing, those are on another disk altogether. 100% of the growth is in the /opt/observium/rrds directory. I'm trying to implement some system that will let us track individual file growth and/or creation so we can figure out where the space is going... (`find -ls` + git is as good is the best I can come up with so far)
We don't run the housekeeping script, because we need historical data to stick around for other (non-Observium) processes to parse and make pretty graphs from. However, running housekeeping with the "-r" and "-T" flags only finds six old no-longer-existing hosts, but those amount to only 10GB (out of 1+TB).
Further to the above, we're OK with using massive amounts of disk space, we just need it to me [more] deterministic.
Actually... is there a way to calculate how big any given RRD file should be? They should have a fully-deterministic upper bound, right?
-Adam
Adam Thompson Consultant, Infrastructure Services [1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.camailto:athompson@merlin.mb.ca www.merlin.mb.cahttp://www.merlin.mb.ca/ ________________________________ From: observium observium-bounces@observium.org on behalf of Daniel Johansson via observium observium@observium.org Sent: September 27, 2021 09:38 To: observium observium@observium.org Cc: Daniel Johansson daz@voodoo-people.com Subject: Re: [Observium] unending disk space growth?
Hi
IF you don't need mysql transaction logs you can use these values as I use them in our production of observium: binlog_expire_logs_seconds = 300 max_binlog_size = 1000M
If you really don't need them you could probable disable it with setting binlogs value to 1 and log size to 1M or so.
Regards Daniel
________________________________ From: "observium" observium@observium.org To: "observium" observium@observium.org Cc: "Flesch, Philipp" philipp.flesch@gv.mpg.de Sent: Monday, September 27, 2021 4:12:01 PM Subject: Re: [Observium] unending disk space growth?
And – of course – don’t forget to run the Observium housekeeping cron-jobs
Regards,
Philipp
Von: observium observium-bounces@observium.org Im Auftrag von Timothy Stoddard via observium Gesendet: Montag, 27. September 2021 16:10 An: Observium observium@observium.org Cc: Timothy Stoddard testoddard@ualr.edu Betreff: Re: [Observium] unending disk space growth?
This is most likely my-sql logs growing. Check your database directory and look for old logs.
Thanks
-- Tim
On Fri, Sep 24, 2021 at 10:38 AM Adam Armstrong via observium <observium@observium.orgmailto:observium@observium.org> wrote:
Can you pinpoint the large size to any particular device or group of devices?
Rrd dir should not grow indefinitely unless some devices are continually adding and removing entities, and you aren’t running housekeeping to remove old RRDs.
Adam.
Sent from my iPhone
On 24 Sep 2021, at 16:20, Adam Thompson via observium <observium@observium.orgmailto:observium@observium.org> wrote:
Since we last rebuilt our Observium instance in late 2019, we've noticed that it's chewing up disk space at a prodigious rate, that we didn't see before.
We're seeing a steady ~300BG/month increase in storage needs. Observium, of course, monitors itself, and the line on the storage graph is a nearly-perfect slope (after adjusting for disk size increases).
This came to my attention this morning (again), when we ran out of disk space (again) and I had to expand the volume (again) to continue operating. (So much for our alerting... oops.)
We have added a few new devices, but not 1TB worth, and not consistently and constantly since late 2019.
Since RRD files are - in theory - fixed-size, why does /opt/obvservium/rrd keep growing without bound? And how can I predict future growth? Will it ever stop?
Help? Anyone?
-Adam
Adam Thompson Consultant, Infrastructure Services [1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson@merlin.mb.camailto:athompson@merlin.mb.ca www.merlin.mb.cahttp://www.merlin.mb.ca/
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (5)
-
Adam Armstrong -
Adam Thompson -
Daniel Johansson -
Flesch, Philipp -
Timothy Stoddard