Hi everyone,
Yesterday I committed a housekeeping script to clean up after Observium.
If you have a nice number of devices combined with a long running time of Observium, you're sure to have a gigantic database of event log, syslog and performance timing information. Some of you have resorted to periodically truncating tables - fear no more, housekeeping.php is here!
You have a choice of specifying either -a for all modules or s, e, r, p and/or t for specific things. Apart from that, there are options in the configuration to set before most of these things actually do anything.
What it does: -s Cleans up syslog data, based on the age of the entries.
Configured by $config['housekeeping']['syslog']['age'] = 0; // Maximum age of syslog entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
-e Cleans up eventlog data, based on the age of the entries.
Configured by $config['housekeeping']['eventlog']['age'] = 0; // Maximum age of eventlog entries in seconds; 0 to disable i.e. 30*86400 for 30 days.
-r Cleans up RRD files, there are 2 options here:
Cleanup of RRD files with a modification time of a certain age. Ports that are gone, sensors that are gone, erroneaously created RRD files due to bugs or hardware changes, etc.
Configured by $config['housekeeping']['rrd']['age'] = 0; // Maximum age of unused rrd files in seconds before automatically purging; 0 to disable i.e. 90*86400 for 90 days.
Cleanup of invalid RRD files. Requires $config['file'] to be set correctly to the path of the "file" magic utility. Runs over all .rrd files to see if they are identified as RRDTool Databases. They can sometimes get corrupted (0 bytes, half a file) when your disk/ramdrive is full. When they exist, they are not recreated even if they're invalid. Consequently RRDtool refuses to add data into them as they are not valid RRD database files. This scripts deletes any .rrd file not identified as RRDTool.
Configured by $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
Note: this defaults to true currently. Handle with care.
-p Cleans up deleted ports. You can do this manually by using the purge button in the web interface, but one could probably safely assume if a port has not returned after 30 days it really is gone. Shorter could be useful in your environment, mine is set to 7 days.
Configured by $config['housekeeping']['deleted_ports']['age'] = 0; // Maximum age of deleted ports in seconds before automatically purging; 0 to disable i.e. 30*86400 for 30 days.
-t: Cleans up perf time information. Observium keeps performance times per module, per device and per run. This quickly accumulates into a giant number of rows in the database.
Configured by $config['housekeeping']['timing']['age'] = 0; // Maximum age of timing (discovery and poll time) entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
Housekeeping logs into housekeeping.log in the log_dir. (*)
I recommend putting this into cron, daily or weekly (yes, even though the age is set in seconds, I don't recommend running it every minute ;), but only after you've run it manually once. It'll take quite a while the first time to clean up large databases. The "check for invalid RRD files" is probably quite I/O heavy.
i'll be documenting this on the wiki soonish.
(*) log_dir was also added as a new configuration directive, it defaults to ${installdir}/logs which will be created automatically for you if possible.
Tom
Nice work, Tom. This should definitely go in the wiki docs if it's not already there.
PS - Jira #757 is mostly redundant now, although my script had an option to report what it was going to do and would tell you how many records and how much disk space could be reclaimed if the maintenance was executed. :) -- Cameron Moore
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Tom Laermans Sent: Thursday, July 17, 2014 6:49 AM To: Observium Network Observation System Subject: [Observium] Housekeeping script
Hi everyone,
Yesterday I committed a housekeeping script to clean up after Observium.
If you have a nice number of devices combined with a long running time of Observium, you're sure to have a gigantic database of event log, syslog and performance timing information. Some of you have resorted to periodically truncating tables - fear no more, housekeeping.php is here!
You have a choice of specifying either -a for all modules or s, e, r, p and/or t for specific things. Apart from that, there are options in the configuration to set before most of these things actually do anything.
What it does: -s Cleans up syslog data, based on the age of the entries.
Configured by $config['housekeeping']['syslog']['age'] = 0; // Maximum age of syslog entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
-e Cleans up eventlog data, based on the age of the entries.
Configured by $config['housekeeping']['eventlog']['age'] = 0; // Maximum age of eventlog entries in seconds; 0 to disable i.e. 30*86400 for 30 days.
-r Cleans up RRD files, there are 2 options here:
Cleanup of RRD files with a modification time of a certain age. Ports that are gone, sensors that are gone, erroneaously created RRD files due to bugs or hardware changes, etc.
Configured by $config['housekeeping']['rrd']['age'] = 0; // Maximum age of unused rrd files in seconds before automatically purging; 0 to disable i.e. 90*86400 for 90 days.
Cleanup of invalid RRD files. Requires $config['file'] to be set correctly to the path of the "file" magic utility. Runs over all .rrd files to see if they are identified as RRDTool Databases. They can sometimes get corrupted (0 bytes, half a file) when your disk/ramdrive is full. When they exist, they are not recreated even if they're invalid. Consequently RRDtool refuses to add data into them as they are not valid RRD database files. This scripts deletes any .rrd file not identified as RRDTool.
Configured by $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
Note: this defaults to true currently. Handle with care.
-p Cleans up deleted ports. You can do this manually by using the purge button in the web interface, but one could probably safely assume if a port has not returned after 30 days it really is gone. Shorter could be useful in your environment, mine is set to 7 days.
Configured by $config['housekeeping']['deleted_ports']['age'] = 0; // Maximum age of deleted ports in seconds before automatically purging; 0 to disable i.e. 30*86400 for 30 days.
-t: Cleans up perf time information. Observium keeps performance times per module, per device and per run. This quickly accumulates into a giant number of rows in the database.
Configured by $config['housekeeping']['timing']['age'] = 0; // Maximum age of timing (discovery and poll time) entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
Housekeeping logs into housekeeping.log in the log_dir. (*)
I recommend putting this into cron, daily or weekly (yes, even though the age is set in seconds, I don't recommend running it every minute ;), but only after you've run it manually once. It'll take quite a while the first time to clean up large databases. The "check for invalid RRD files" is probably quite I/O heavy.
i'll be documenting this on the wiki soonish.
(*) log_dir was also added as a new configuration directive, it defaults to ${installdir}/logs which will be created automatically for you if possible.
Tom _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Heya,
Yeah, I still want to take a look at your script - it was on my note with all the ideas I had myself to do this. Mike is reworking some parts, it'll tell you what it's going to do when he commits.
I'll take a look later, it's why I haven't closed it yet :)
Tom
On 07/17/2014 04:14 PM, Moore, Cameron wrote:
Nice work, Tom. This should definitely go in the wiki docs if it's not already there.
PS - Jira #757 is mostly redundant now, although my script had an option to report what it was going to do and would tell you how many records and how much disk space could be reclaimed if the maintenance was executed. :)
Cameron Moore
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Tom Laermans Sent: Thursday, July 17, 2014 6:49 AM To: Observium Network Observation System Subject: [Observium] Housekeeping script
Hi everyone,
Yesterday I committed a housekeeping script to clean up after Observium.
If you have a nice number of devices combined with a long running time of Observium, you're sure to have a gigantic database of event log, syslog and performance timing information. Some of you have resorted to periodically truncating tables - fear no more, housekeeping.php is here!
You have a choice of specifying either -a for all modules or s, e, r, p and/or t for specific things. Apart from that, there are options in the configuration to set before most of these things actually do anything.
What it does: -s Cleans up syslog data, based on the age of the entries.
Configured by $config['housekeeping']['syslog']['age'] = 0; // Maximum age of syslog entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
-e Cleans up eventlog data, based on the age of the entries.
Configured by $config['housekeeping']['eventlog']['age'] = 0; // Maximum age of eventlog entries in seconds; 0 to disable i.e. 30*86400 for 30 days.
-r Cleans up RRD files, there are 2 options here:
Cleanup of RRD files with a modification time of a certain age. Ports that are gone, sensors that are gone, erroneaously created RRD files due to bugs or hardware changes, etc.
Configured by $config['housekeeping']['rrd']['age'] = 0; // Maximum age of unused rrd files in seconds before automatically purging; 0 to disable i.e. 90*86400 for 90 days.
Cleanup of invalid RRD files. Requires $config['file'] to be set correctly to the path of the "file" magic utility. Runs over all .rrd files to see if they are identified as RRDTool Databases. They can sometimes get corrupted (0 bytes, half a file) when your disk/ramdrive is full. When they exist, they are not recreated even if they're invalid. Consequently RRDtool refuses to add data into them as they are not valid RRD database files. This scripts deletes any .rrd file not identified as RRDTool.
Configured by $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
Note: this defaults to true currently. Handle with care.
-p Cleans up deleted ports. You can do this manually by using the purge button in the web interface, but one could probably safely assume if a port has not returned after 30 days it really is gone. Shorter could be useful in your environment, mine is set to 7 days.
Configured by $config['housekeeping']['deleted_ports']['age'] = 0; // Maximum age of deleted ports in seconds before automatically purging; 0 to disable i.e. 30*86400 for 30 days.
-t: Cleans up perf time information. Observium keeps performance times per module, per device and per run. This quickly accumulates into a giant number of rows in the database.
Configured by $config['housekeeping']['timing']['age'] = 0; // Maximum age of timing (discovery and poll time) entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
Housekeeping logs into housekeeping.log in the log_dir. (*)
I recommend putting this into cron, daily or weekly (yes, even though the age is set in seconds, I don't recommend running it every minute ;), but only after you've run it manually once. It'll take quite a while the first time to clean up large databases. The "check for invalid RRD files" is probably quite I/O heavy.
i'll be documenting this on the wiki soonish.
(*) log_dir was also added as a new configuration directive, it defaults to ${installdir}/logs which will be created automatically for you if possible.
Tom _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi,
Housekeeping script updated.
Changes: - By default asks 'Are you sure want to delete (y/N)?', for bypass promts use '-y' in command line. Also crontab detects automaticaly (not necessarily add '-y' in cron); - Allowed use human readable ages like 3w, 2.5d (see comments in defaults.inc.php); - More checks for ages (minimum allowed 24h in most modules, except 'deleted_ports'); - Fixed 'deleted_ports' module, use more accurate port deletetion function.
enjoy :)
On 17.07.2014 18:30, Tom Laermans wrote:
Heya,
Yeah, I still want to take a look at your script - it was on my note with all the ideas I had myself to do this. Mike is reworking some parts, it'll tell you what it's going to do when he commits.
I'll take a look later, it's why I haven't closed it yet :)
Tom
On 07/17/2014 04:14 PM, Moore, Cameron wrote:
Nice work, Tom. This should definitely go in the wiki docs if it's not already there.
PS - Jira #757 is mostly redundant now, although my script had an option to report what it was going to do and would tell you how many records and how much disk space could be reclaimed if the maintenance was executed. :) -- Cameron Moore
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Tom Laermans Sent: Thursday, July 17, 2014 6:49 AM To: Observium Network Observation System Subject: [Observium] Housekeeping script
Hi everyone,
Yesterday I committed a housekeeping script to clean up after Observium.
If you have a nice number of devices combined with a long running time of Observium, you're sure to have a gigantic database of event log, syslog and performance timing information. Some of you have resorted to periodically truncating tables - fear no more, housekeeping.php is here!
You have a choice of specifying either -a for all modules or s, e, r, p and/or t for specific things. Apart from that, there are options in the configuration to set before most of these things actually do anything.
What it does: -s Cleans up syslog data, based on the age of the entries.
Configured by $config['housekeeping']['syslog']['age'] = 0; // Maximum age of syslog entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
-e Cleans up eventlog data, based on the age of the entries.
Configured by $config['housekeeping']['eventlog']['age'] = 0; // Maximum age of eventlog entries in seconds; 0 to disable i.e. 30*86400 for 30 days.
-r Cleans up RRD files, there are 2 options here:
Cleanup of RRD files with a modification time of a certain age. Ports that are gone, sensors that are gone, erroneaously created RRD files due to bugs or hardware changes, etc.
Configured by $config['housekeeping']['rrd']['age'] = 0; // Maximum age of unused rrd files in seconds before automatically purging; 0 to disable i.e. 90*86400 for 90 days.
Cleanup of invalid RRD files. Requires $config['file'] to be set correctly to the path of the "file" magic utility. Runs over all .rrd files to see if they are identified as RRDTool Databases. They can sometimes get corrupted (0 bytes, half a file) when your disk/ramdrive is full. When they exist, they are not recreated even if they're invalid. Consequently RRDtool refuses to add data into them as they are not valid RRD database files. This scripts deletes any .rrd file not identified as RRDTool.
Configured by $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
Note: this defaults to true currently. Handle with care.
-p Cleans up deleted ports. You can do this manually by using the purge button in the web interface, but one could probably safely assume if a port has not returned after 30 days it really is gone. Shorter could be useful in your environment, mine is set to 7 days.
Configured by $config['housekeeping']['deleted_ports']['age'] = 0; // Maximum age of deleted ports in seconds before automatically purging; 0 to disable i.e. 30*86400 for 30 days.
-t: Cleans up perf time information. Observium keeps performance times per module, per device and per run. This quickly accumulates into a giant number of rows in the database.
Configured by $config['housekeeping']['timing']['age'] = 0; // Maximum age of timing (discovery and poll time) entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
Housekeeping logs into housekeeping.log in the log_dir. (*)
I recommend putting this into cron, daily or weekly (yes, even though the age is set in seconds, I don't recommend running it every minute ;), but only after you've run it manually once. It'll take quite a while the first time to clean up large databases. The "check for invalid RRD files" is probably quite I/O heavy.
i'll be documenting this on the wiki soonish.
(*) log_dir was also added as a new configuration directive, it defaults to ${installdir}/logs which will be created automatically for you if possible.
Tom _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Woohoo, finally. Thanks Tom. Am I suppose to enter integers in seconds or "30*86400" as value do same trick?
On 17 июля 2014 г., at 15:49, Tom Laermans tom.laermans@powersource.cx wrote:
Hi everyone,
Yesterday I committed a housekeeping script to clean up after Observium.
If you have a nice number of devices combined with a long running time of Observium, you're sure to have a gigantic database of event log, syslog and performance timing information. Some of you have resorted to periodically truncating tables - fear no more, housekeeping.php is here!
You have a choice of specifying either -a for all modules or s, e, r, p and/or t for specific things. Apart from that, there are options in the configuration to set before most of these things actually do anything.
What it does: -s Cleans up syslog data, based on the age of the entries.
Configured by $config['housekeeping']['syslog']['age'] = 0; // Maximum age of syslog entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
-e Cleans up eventlog data, based on the age of the entries.
Configured by $config['housekeeping']['eventlog']['age'] = 0; // Maximum age of eventlog entries in seconds; 0 to disable i.e.30*86400 for 30 days.
-r Cleans up RRD files, there are 2 options here:
Cleanup of RRD files with a modification time of a certain age. Ports that are gone, sensors that are gone, erroneaously created RRD files due to bugs or hardware changes, etc.
Configured by $config['housekeeping']['rrd']['age'] = 0; // Maximum age of unused rrd files in seconds before automatically purging; 0 to disable i.e. 90*86400 for 90 days.
Cleanup of invalid RRD files. Requires $config['file'] to be set correctly to the path of the "file" magic utility. Runs over all .rrd files to see if they are identified as RRDTool Databases. They can sometimes get corrupted (0 bytes, half a file) when your disk/ramdrive is full. When they exist, they are not recreated even if they're invalid. Consequently RRDtool refuses to add data into them as they are not valid RRD database files. This scripts deletes any .rrd file not identified as RRDTool.
Configured by $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
Note: this defaults to true currently. Handle with care.
-p Cleans up deleted ports. You can do this manually by using the purge button in the web interface, but one could probably safely assume if a port has not returned after 30 days it really is gone. Shorter could be useful in your environment, mine is set to 7 days.
Configured by $config['housekeeping']['deleted_ports']['age'] = 0; // Maximum age of deleted ports in seconds before automatically purging; 0 to disable i.e. 30*86400 for 30 days.
-t: Cleans up perf time information. Observium keeps performance times per module, per device and per run. This quickly accumulates into a giant number of rows in the database.
Configured by $config['housekeeping']['timing']['age'] = 0; // Maximum age of timing (discovery and poll time) entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
Housekeeping logs into housekeeping.log in the log_dir. (*)
I recommend putting this into cron, daily or weekly (yes, even though the age is set in seconds, I don't recommend running it every minute ;), but only after you've run it manually once. It'll take quite a while the first time to clean up large databases. The "check for invalid RRD files" is probably quite I/O heavy.
i'll be documenting this on the wiki soonish.
(*) log_dir was also added as a new configuration directive, it defaults to ${installdir}/logs which will be created automatically for you if possible.
Tom _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
It's PHP - PHP can do math. 30*86400 works ;) But Mike patched it so 30d should work, if I understand correctly :)
Tom
On 17/07/2014 18:21, Nikolay Shopik wrote:
Woohoo, finally. Thanks Tom. Am I suppose to enter integers in seconds or "30*86400" as value do same trick?
On 17 июля 2014 г., at 15:49, Tom Laermans tom.laermans@powersource.cx wrote:
Hi everyone,
Yesterday I committed a housekeeping script to clean up after Observium.
If you have a nice number of devices combined with a long running time of Observium, you're sure to have a gigantic database of event log, syslog and performance timing information. Some of you have resorted to periodically truncating tables - fear no more, housekeeping.php is here!
You have a choice of specifying either -a for all modules or s, e, r, p and/or t for specific things. Apart from that, there are options in the configuration to set before most of these things actually do anything.
What it does: -s Cleans up syslog data, based on the age of the entries.
Configured by $config['housekeeping']['syslog']['age'] = 0; // Maximum age of syslog entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
-e Cleans up eventlog data, based on the age of the entries.
Configured by $config['housekeeping']['eventlog']['age'] = 0; // Maximum age of eventlog entries in seconds; 0 to disable i.e.30*86400 for 30 days.
-r Cleans up RRD files, there are 2 options here:
Cleanup of RRD files with a modification time of a certain age. Ports that are gone, sensors that are gone, erroneaously created RRD files due to bugs or hardware changes, etc.
Configured by $config['housekeeping']['rrd']['age'] = 0; // Maximum age of unused rrd files in seconds before automatically purging; 0 to disable i.e. 90*86400 for 90 days.
Cleanup of invalid RRD files. Requires $config['file'] to be set correctly to the path of the "file" magic utility. Runs over all .rrd files to see if they are identified as RRDTool Databases. They can sometimes get corrupted (0 bytes, half a file) when your disk/ramdrive is full. When they exist, they are not recreated even if they're invalid. Consequently RRDtool refuses to add data into them as they are not valid RRD database files. This scripts deletes any .rrd file not identified as RRDTool.
Configured by $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
Note: this defaults to true currently. Handle with care.
-p Cleans up deleted ports. You can do this manually by using the purge button in the web interface, but one could probably safely assume if a port has not returned after 30 days it really is gone. Shorter could be useful in your environment, mine is set to 7 days.
Configured by $config['housekeeping']['deleted_ports']['age'] = 0; // Maximum age of deleted ports in seconds before automatically purging; 0 to disable i.e. 30*86400 for 30 days.
-t: Cleans up perf time information. Observium keeps performance times per module, per device and per run. This quickly accumulates into a giant number of rows in the database.
Configured by $config['housekeeping']['timing']['age'] = 0; // Maximum age of timing (discovery and poll time) entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
Housekeeping logs into housekeeping.log in the log_dir. (*)
I recommend putting this into cron, daily or weekly (yes, even though the age is set in seconds, I don't recommend running it every minute ;), but only after you've run it manually once. It'll take quite a while the first time to clean up large databases. The "check for invalid RRD files" is probably quite I/O heavy.
i'll be documenting this on the wiki soonish.
(*) log_dir was also added as a new configuration directive, it defaults to ${installdir}/logs which will be created automatically for you if possible.
Tom _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
"Will be deleted 21974152 eventlog entries from before 2014-01-15 21:38:46 (y/N):"
...welp. Guess this came at a good time. ;-)
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Tom Laermans Sent: Thursday, July 17, 2014 9:27 AM To: Observium Network Observation System Subject: Re: [Observium] Housekeeping script
It's PHP - PHP can do math. 30*86400 works ;) But Mike patched it so 30d should work, if I understand correctly :)
Tom
On 17/07/2014 18:21, Nikolay Shopik wrote:
Woohoo, finally. Thanks Tom. Am I suppose to enter integers in seconds or "30*86400" as value do same trick?
On 17 июля 2014 г., at 15:49, Tom Laermans tom.laermans@powersource.cx wrote:
Hi everyone,
Yesterday I committed a housekeeping script to clean up after Observium.
If you have a nice number of devices combined with a long running time of Observium, you're sure to have a gigantic database of event log, syslog and performance timing information. Some of you have resorted to periodically truncating tables - fear no more, housekeeping.php is here!
You have a choice of specifying either -a for all modules or s, e, r, p and/or t for specific things. Apart from that, there are options in the configuration to set before most of these things actually do anything.
What it does: -s Cleans up syslog data, based on the age of the entries.
Configured by $config['housekeeping']['syslog']['age'] = 0; // Maximum age of syslog entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
-e Cleans up eventlog data, based on the age of the entries.
Configured by $config['housekeeping']['eventlog']['age'] = 0; // Maximum age of eventlog entries in seconds; 0 to disable i.e.30*86400 for 30 days.
-r Cleans up RRD files, there are 2 options here:
Cleanup of RRD files with a modification time of a certain age. Ports that are gone, sensors that are gone, erroneaously created RRD files due to bugs or hardware changes, etc.
Configured by $config['housekeeping']['rrd']['age'] = 0; // Maximum age of unused rrd files in seconds before automatically purging; 0 to disable i.e. 90*86400 for 90 days.
Cleanup of invalid RRD files. Requires $config['file'] to be set correctly to the path of the "file" magic utility. Runs over all .rrd files to see if they are identified as RRDTool Databases. They can sometimes get corrupted (0 bytes, half a file) when your disk/ramdrive is full. When they exist, they are not recreated even if they're invalid. Consequently RRDtool refuses to add data into them as they are not valid RRD database files. This scripts deletes any .rrd file not identified as RRDTool.
Configured by $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
Note: this defaults to true currently. Handle with care.
-p Cleans up deleted ports. You can do this manually by using the purge button in the web interface, but one could probably safely assume if a port has not returned after 30 days it really is gone. Shorter could be useful in your environment, mine is set to 7 days.
Configured by $config['housekeeping']['deleted_ports']['age'] = 0; // Maximum age of deleted ports in seconds before automatically purging; 0 to disable i.e. 30*86400 for 30 days.
-t: Cleans up perf time information. Observium keeps performance times per module, per device and per run. This quickly accumulates into a giant number of rows in the database.
Configured by $config['housekeeping']['timing']['age'] = 0; // Maximum age of timing (discovery and poll time) entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
Housekeeping logs into housekeeping.log in the log_dir. (*)
I recommend putting this into cron, daily or weekly (yes, even though the age is set in seconds, I don't recommend running it every minute ;), but only after you've run it manually once. It'll take quite a while the first time to clean up large databases. The "check for invalid RRD files" is probably quite I/O heavy.
i'll be documenting this on the wiki soonish.
(*) log_dir was also added as a new configuration directive, it defaults to ${installdir}/logs which will be created automatically for you if possible.
Tom _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Founded in 2007, IO is a worldwide leader in software defined data center technology, services and solutions that enable businesses and governments to intelligently control their information.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
When using -p, I'm receiving a "sh: 1: port_id: not found" error on every port the script identifies. Turning on debug mode shows the script attempting to look up a blank port_id.
SQL[SELECT `port_id` FROM `ports` WHERE `deleted` = 1 AND UNIX_TIMESTAMP(`ifLastChange`) < 1403027038;] Will be deleted 85 ports marked as 'DELETED' from before 2014-06-17 11:43:58 (y/N): y
...
SQL[SELECT * FROM `ports` AS P, `devices` AS D WHERE P.`port_id` = '' AND D.`device_id` = P.`device_id`]
SQL[DELETE FROM `bill_ports` WHERE `port_id` = '']
SQL[DELETE FROM `eigrp_ports` WHERE `port_id` = '']
SQL[DELETE FROM `ipv4_addresses` WHERE `port_id` = '']
SQL[DELETE FROM `ipv6_addresses` WHERE `port_id` = '']
SQL[DELETE FROM `ip_mac` WHERE `port_id` = '']
SQL[DELETE FROM `juniAtmVp` WHERE `port_id` = '']
SQL[DELETE FROM `mac_accounting` WHERE `port_id` = '']
SQL[DELETE FROM `ospf_nb` WHERE `port_id` = '']
SQL[DELETE FROM `ospf_ports` WHERE `port_id` = '']
SQL[DELETE FROM `ports_adsl` WHERE `port_id` = '']
SQL[DELETE FROM `ports_cbqos` WHERE `port_id` = '']
SQL[DELETE FROM `ports_vlans` WHERE `port_id` = '']
SQL[DELETE FROM `pseudowires` WHERE `port_id` = '']
SQL[DELETE FROM `vlans_fdb` WHERE `port_id` = '']
SQL[DELETE FROM `ports` WHERE `port_id` = '']
SQL[DELETE FROM `ports_stack` WHERE `port_id_high` = '' OR `port_id_low` = '']
SQL[DELETE FROM `links` WHERE `local_port_id` = '' OR `remote_port_id` = '']
SQL[DELETE FROM `entity_permissions` WHERE `entity_type` = 'port' AND `entity_id` = '']
SQL[SELECT * FROM `devices` WHERE `device_id` = ''] sh: 1: port_id: not found!
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Pedersen, Sean Sent: Thursday, July 17, 2014 9:42 AM To: Observium Network Observation System Subject: Re: [Observium] Housekeeping script
"Will be deleted 21974152 eventlog entries from before 2014-01-15 21:38:46 (y/N):"
...welp. Guess this came at a good time. ;-)
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Tom Laermans Sent: Thursday, July 17, 2014 9:27 AM To: Observium Network Observation System Subject: Re: [Observium] Housekeeping script
It's PHP - PHP can do math. 30*86400 works ;) But Mike patched it so 30d should work, if I understand correctly :)
Tom
On 17/07/2014 18:21, Nikolay Shopik wrote:
Woohoo, finally. Thanks Tom. Am I suppose to enter integers in seconds or "30*86400" as value do same trick?
On 17 июля 2014 г., at 15:49, Tom Laermans tom.laermans@powersource.cx wrote:
Hi everyone,
Yesterday I committed a housekeeping script to clean up after Observium.
If you have a nice number of devices combined with a long running time of Observium, you're sure to have a gigantic database of event log, syslog and performance timing information. Some of you have resorted to periodically truncating tables - fear no more, housekeeping.php is here!
You have a choice of specifying either -a for all modules or s, e, r, p and/or t for specific things. Apart from that, there are options in the configuration to set before most of these things actually do anything.
What it does: -s Cleans up syslog data, based on the age of the entries.
Configured by $config['housekeeping']['syslog']['age'] = 0; // Maximum age of syslog entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
-e Cleans up eventlog data, based on the age of the entries.
Configured by $config['housekeeping']['eventlog']['age'] = 0; // Maximum age of eventlog entries in seconds; 0 to disable i.e.30*86400 for 30 days.
-r Cleans up RRD files, there are 2 options here:
Cleanup of RRD files with a modification time of a certain age. Ports that are gone, sensors that are gone, erroneaously created RRD files due to bugs or hardware changes, etc.
Configured by $config['housekeeping']['rrd']['age'] = 0; // Maximum age of unused rrd files in seconds before automatically purging; 0 to disable i.e. 90*86400 for 90 days.
Cleanup of invalid RRD files. Requires $config['file'] to be set correctly to the path of the "file" magic utility. Runs over all .rrd files to see if they are identified as RRDTool Databases. They can sometimes get corrupted (0 bytes, half a file) when your disk/ramdrive is full. When they exist, they are not recreated even if they're invalid. Consequently RRDtool refuses to add data into them as they are not valid RRD database files. This scripts deletes any .rrd file not identified as RRDTool.
Configured by $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
Note: this defaults to true currently. Handle with care.
-p Cleans up deleted ports. You can do this manually by using the purge button in the web interface, but one could probably safely assume if a port has not returned after 30 days it really is gone. Shorter could be useful in your environment, mine is set to 7 days.
Configured by $config['housekeeping']['deleted_ports']['age'] = 0; // Maximum age of deleted ports in seconds before automatically purging; 0 to disable i.e. 30*86400 for 30 days.
-t: Cleans up perf time information. Observium keeps performance times per module, per device and per run. This quickly accumulates into a giant number of rows in the database.
Configured by $config['housekeeping']['timing']['age'] = 0; // Maximum age of timing (discovery and poll time) entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
Housekeeping logs into housekeeping.log in the log_dir. (*)
I recommend putting this into cron, daily or weekly (yes, even though the age is set in seconds, I don't recommend running it every minute ;), but only after you've run it manually once. It'll take quite a while the first time to clean up large databases. The "check for invalid RRD files" is probably quite I/O heavy.
i'll be documenting this on the wiki soonish.
(*) log_dir was also added as a new configuration directive, it defaults to ${installdir}/logs which will be created automatically for you if possible.
Tom _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Founded in 2007, IO is a worldwide leader in software defined data center technology, services and solutions that enable businesses and governments to intelligently control their information.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Founded in 2007, IO is a worldwide leader in software defined data center technology, services and solutions that enable businesses and governments to intelligently control their information.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
Fixed in r5661.
On Thu, Jul 17, 2014 at 10:37 PM, Pedersen, Sean spedersen@io.com wrote:
When using -p, I'm receiving a "sh: 1: port_id: not found" error on every port the script identifies. Turning on debug mode shows the script attempting to look up a blank port_id.
SQL[SELECT `port_id` FROM `ports` WHERE `deleted` = 1 AND UNIX_TIMESTAMP(`ifLastChange`) < 1403027038;] Will be deleted 85 ports marked as 'DELETED' from before 2014-06-17 11:43:58 (y/N): y
...
SQL[SELECT * FROM `ports` AS P, `devices` AS D WHERE P.`port_id` = '' AND D.`device_id` = P.`device_id`]
SQL[DELETE FROM `bill_ports` WHERE `port_id` = '']
SQL[DELETE FROM `eigrp_ports` WHERE `port_id` = '']
SQL[DELETE FROM `ipv4_addresses` WHERE `port_id` = '']
SQL[DELETE FROM `ipv6_addresses` WHERE `port_id` = '']
SQL[DELETE FROM `ip_mac` WHERE `port_id` = '']
SQL[DELETE FROM `juniAtmVp` WHERE `port_id` = '']
SQL[DELETE FROM `mac_accounting` WHERE `port_id` = '']
SQL[DELETE FROM `ospf_nb` WHERE `port_id` = '']
SQL[DELETE FROM `ospf_ports` WHERE `port_id` = '']
SQL[DELETE FROM `ports_adsl` WHERE `port_id` = '']
SQL[DELETE FROM `ports_cbqos` WHERE `port_id` = '']
SQL[DELETE FROM `ports_vlans` WHERE `port_id` = '']
SQL[DELETE FROM `pseudowires` WHERE `port_id` = '']
SQL[DELETE FROM `vlans_fdb` WHERE `port_id` = '']
SQL[DELETE FROM `ports` WHERE `port_id` = '']
SQL[DELETE FROM `ports_stack` WHERE `port_id_high` = '' OR `port_id_low` = '']
SQL[DELETE FROM `links` WHERE `local_port_id` = '' OR `remote_port_id` = '']
SQL[DELETE FROM `entity_permissions` WHERE `entity_type` = 'port' AND `entity_id` = '']
SQL[SELECT * FROM `devices` WHERE `device_id` = ''] sh: 1: port_id: not found!
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Pedersen, Sean Sent: Thursday, July 17, 2014 9:42 AM To: Observium Network Observation System Subject: Re: [Observium] Housekeeping script
"Will be deleted 21974152 eventlog entries from before 2014-01-15 21:38:46 (y/N):"
...welp. Guess this came at a good time. ;-)
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Tom Laermans Sent: Thursday, July 17, 2014 9:27 AM To: Observium Network Observation System Subject: Re: [Observium] Housekeeping script
It's PHP - PHP can do math. 30*86400 works ;) But Mike patched it so 30d should work, if I understand correctly :)
Tom
On 17/07/2014 18:21, Nikolay Shopik wrote:
Woohoo, finally. Thanks Tom. Am I suppose to enter integers in seconds
or "30*86400" as value do same trick?
On 17 июля 2014 г., at 15:49, Tom Laermans tom.laermans@powersource.cx
wrote:
Hi everyone,
Yesterday I committed a housekeeping script to clean up after Observium.
If you have a nice number of devices combined with a long running time
of Observium, you're sure to have a gigantic database of event log, syslog and performance timing information. Some of you have resorted to periodically truncating tables - fear no more, housekeeping.php is here!
You have a choice of specifying either -a for all modules or s, e, r, p
and/or t for specific things.
Apart from that, there are options in the configuration to set before
most of these things actually do anything.
What it does: -s Cleans up syslog data, based on the age of the entries.
Configured by $config['housekeeping']['syslog']['age'] = 0; // Maximum age of syslog entries in seconds; 0 to disable i.e. 7*86400 for
7 days.
-e Cleans up eventlog data, based on the age of the entries.
Configured by $config['housekeeping']['eventlog']['age'] = 0; // Maximum age of eventlog entries in seconds; 0 to disable i.e.30*86400 for 30 days.
-r Cleans up RRD files, there are 2 options here:
Cleanup of RRD files with a modification time of a certain age. Ports
that are gone, sensors that are gone, erroneaously created RRD files due to bugs or hardware changes, etc.
Configured by $config['housekeeping']['rrd']['age'] = 0; // Maximum age of unused rrd files in seconds before automatically purging; 0 to
disable i.e. 90*86400 for 90 days.
Cleanup of invalid RRD files. Requires $config['file'] to be set
correctly to the path of the "file" magic utility.
Runs over all .rrd files to see if they are identified as RRDTool
Databases. They can sometimes get corrupted (0 bytes, half a file) when your disk/ramdrive is full. When they exist, they are not recreated even if they're invalid. Consequently RRDtool refuses to add data into them as they are not valid RRD database files. This scripts deletes any .rrd file not identified as RRDTool.
Configured by $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
Note: this defaults to true currently. Handle with care.
-p Cleans up deleted ports. You can do this manually by using the purge
button in the web interface, but one could probably safely assume if a port has not returned after 30 days it really is gone. Shorter could be useful in your environment, mine is set to 7 days.
Configured by $config['housekeeping']['deleted_ports']['age'] = 0; // Maximum age of deleted ports in seconds before automatically purging; 0
to disable i.e. 30*86400 for 30 days.
-t: Cleans up perf time information. Observium keeps performance times
per module, per device and per run. This quickly accumulates into a giant number of rows in the database.
Configured by $config['housekeeping']['timing']['age'] = 0; // Maximum age of timing (discovery and poll time) entries in seconds; 0
to disable i.e. 7*86400 for 7 days.
Housekeeping logs into housekeeping.log in the log_dir. (*)
I recommend putting this into cron, daily or weekly (yes, even though
the age is set in seconds, I don't recommend running it every minute ;), but only after you've run it manually once. It'll take quite a while the first time to clean up large databases.
The "check for invalid RRD files" is probably quite I/O heavy.
i'll be documenting this on the wiki soonish.
(*) log_dir was also added as a new configuration directive, it
defaults to ${installdir}/logs which will be created automatically for you if possible.
Tom _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Founded in 2007, IO is a worldwide leader in software defined data center technology, services and solutions that enable businesses and governments to intelligently control their information.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Founded in 2007, IO is a worldwide leader in software defined data center technology, services and solutions that enable businesses and governments to intelligently control their information.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hello, just a question. The Housekeeping Script will only be available on the paid version?
On 17 July 2014 21:11, Mike Stupalov mike@observium.org wrote:
Fixed in r5661.
On Thu, Jul 17, 2014 at 10:37 PM, Pedersen, Sean spedersen@io.com wrote:
When using -p, I'm receiving a "sh: 1: port_id: not found" error on every port the script identifies. Turning on debug mode shows the script attempting to look up a blank port_id.
SQL[SELECT `port_id` FROM `ports` WHERE `deleted` = 1 AND UNIX_TIMESTAMP(`ifLastChange`) < 1403027038;] Will be deleted 85 ports marked as 'DELETED' from before 2014-06-17 11:43:58 (y/N): y
...
SQL[SELECT * FROM `ports` AS P, `devices` AS D WHERE P.`port_id` = '' AND D.`device_id` = P.`device_id`]
SQL[DELETE FROM `bill_ports` WHERE `port_id` = '']
SQL[DELETE FROM `eigrp_ports` WHERE `port_id` = '']
SQL[DELETE FROM `ipv4_addresses` WHERE `port_id` = '']
SQL[DELETE FROM `ipv6_addresses` WHERE `port_id` = '']
SQL[DELETE FROM `ip_mac` WHERE `port_id` = '']
SQL[DELETE FROM `juniAtmVp` WHERE `port_id` = '']
SQL[DELETE FROM `mac_accounting` WHERE `port_id` = '']
SQL[DELETE FROM `ospf_nb` WHERE `port_id` = '']
SQL[DELETE FROM `ospf_ports` WHERE `port_id` = '']
SQL[DELETE FROM `ports_adsl` WHERE `port_id` = '']
SQL[DELETE FROM `ports_cbqos` WHERE `port_id` = '']
SQL[DELETE FROM `ports_vlans` WHERE `port_id` = '']
SQL[DELETE FROM `pseudowires` WHERE `port_id` = '']
SQL[DELETE FROM `vlans_fdb` WHERE `port_id` = '']
SQL[DELETE FROM `ports` WHERE `port_id` = '']
SQL[DELETE FROM `ports_stack` WHERE `port_id_high` = '' OR `port_id_low` = '']
SQL[DELETE FROM `links` WHERE `local_port_id` = '' OR `remote_port_id` = '']
SQL[DELETE FROM `entity_permissions` WHERE `entity_type` = 'port' AND `entity_id` = '']
SQL[SELECT * FROM `devices` WHERE `device_id` = ''] sh: 1: port_id: not found!
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Pedersen, Sean Sent: Thursday, July 17, 2014 9:42 AM To: Observium Network Observation System Subject: Re: [Observium] Housekeeping script
"Will be deleted 21974152 eventlog entries from before 2014-01-15 21:38:46 (y/N):"
...welp. Guess this came at a good time. ;-)
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Tom Laermans Sent: Thursday, July 17, 2014 9:27 AM To: Observium Network Observation System Subject: Re: [Observium] Housekeeping script
It's PHP - PHP can do math. 30*86400 works ;) But Mike patched it so 30d should work, if I understand correctly :)
Tom
On 17/07/2014 18:21, Nikolay Shopik wrote:
Woohoo, finally. Thanks Tom. Am I suppose to enter integers in seconds
or "30*86400" as value do same trick?
On 17 июля 2014 г., at 15:49, Tom Laermans <
tom.laermans@powersource.cx> wrote:
Hi everyone,
Yesterday I committed a housekeeping script to clean up after
Observium.
If you have a nice number of devices combined with a long running time
of Observium, you're sure to have a gigantic database of event log, syslog and performance timing information. Some of you have resorted to periodically truncating tables - fear no more, housekeeping.php is here!
You have a choice of specifying either -a for all modules or s, e, r,
p and/or t for specific things.
Apart from that, there are options in the configuration to set before
most of these things actually do anything.
What it does: -s Cleans up syslog data, based on the age of the entries.
Configured by $config['housekeeping']['syslog']['age'] = 0; // Maximum age of syslog entries in seconds; 0 to disable i.e. 7*86400
for 7 days.
-e Cleans up eventlog data, based on the age of the entries.
Configured by $config['housekeeping']['eventlog']['age'] = 0; // Maximum age of eventlog entries in seconds; 0 to disable i.e.30*86400 for 30 days.
-r Cleans up RRD files, there are 2 options here:
Cleanup of RRD files with a modification time of a certain age. Ports
that are gone, sensors that are gone, erroneaously created RRD files due to bugs or hardware changes, etc.
Configured by $config['housekeeping']['rrd']['age'] = 0; // Maximum age of unused rrd files in seconds before automatically purging; 0 to
disable i.e. 90*86400 for 90 days.
Cleanup of invalid RRD files. Requires $config['file'] to be set
correctly to the path of the "file" magic utility.
Runs over all .rrd files to see if they are identified as RRDTool
Databases. They can sometimes get corrupted (0 bytes, half a file) when your disk/ramdrive is full. When they exist, they are not recreated even if they're invalid. Consequently RRDtool refuses to add data into them as they are not valid RRD database files. This scripts deletes any .rrd file not identified as RRDTool.
Configured by $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
Note: this defaults to true currently. Handle with care.
-p Cleans up deleted ports. You can do this manually by using the
purge button in the web interface, but one could probably safely assume if a port has not returned after 30 days it really is gone. Shorter could be useful in your environment, mine is set to 7 days.
Configured by $config['housekeeping']['deleted_ports']['age'] = 0; // Maximum age of deleted ports in seconds before automatically purging;
0 to disable i.e. 30*86400 for 30 days.
-t: Cleans up perf time information. Observium keeps performance times
per module, per device and per run. This quickly accumulates into a giant number of rows in the database.
Configured by $config['housekeeping']['timing']['age'] = 0; // Maximum age of timing (discovery and poll time) entries in seconds; 0
to disable i.e. 7*86400 for 7 days.
Housekeeping logs into housekeeping.log in the log_dir. (*)
I recommend putting this into cron, daily or weekly (yes, even though
the age is set in seconds, I don't recommend running it every minute ;), but only after you've run it manually once. It'll take quite a while the first time to clean up large databases.
The "check for invalid RRD files" is probably quite I/O heavy.
i'll be documenting this on the wiki soonish.
(*) log_dir was also added as a new configuration directive, it
defaults to ${installdir}/logs which will be created automatically for you if possible.
Tom _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Founded in 2007, IO is a worldwide leader in software defined data center technology, services and solutions that enable businesses and governments to intelligently control their information.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Founded in 2007, IO is a worldwide leader in software defined data center technology, services and solutions that enable businesses and governments to intelligently control their information.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Mike Stupalov http://observium.org/ _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
For now yes, will be included in next ce version
On 18 июля 2014 г., at 11:17, Ximena Cardinali ximenacardinali@gmail.com wrote:
Hello, just a question. The Housekeeping Script will only be available on the paid version?
On 17 July 2014 21:11, Mike Stupalov mike@observium.org wrote:
Fixed in r5661.
On Thu, Jul 17, 2014 at 10:37 PM, Pedersen, Sean spedersen@io.com wrote: When using -p, I'm receiving a "sh: 1: port_id: not found" error on every port the script identifies. Turning on debug mode shows the script attempting to look up a blank port_id.
SQL[SELECT `port_id` FROM `ports` WHERE `deleted` = 1 AND UNIX_TIMESTAMP(`ifLastChange`) < 1403027038;] Will be deleted 85 ports marked as 'DELETED' from before 2014-06-17 11:43:58 (y/N): y
...
SQL[SELECT * FROM `ports` AS P, `devices` AS D WHERE P.`port_id` = '' AND D.`device_id` = P.`device_id`]
SQL[DELETE FROM `bill_ports` WHERE `port_id` = '']
SQL[DELETE FROM `eigrp_ports` WHERE `port_id` = '']
SQL[DELETE FROM `ipv4_addresses` WHERE `port_id` = '']
SQL[DELETE FROM `ipv6_addresses` WHERE `port_id` = '']
SQL[DELETE FROM `ip_mac` WHERE `port_id` = '']
SQL[DELETE FROM `juniAtmVp` WHERE `port_id` = '']
SQL[DELETE FROM `mac_accounting` WHERE `port_id` = '']
SQL[DELETE FROM `ospf_nb` WHERE `port_id` = '']
SQL[DELETE FROM `ospf_ports` WHERE `port_id` = '']
SQL[DELETE FROM `ports_adsl` WHERE `port_id` = '']
SQL[DELETE FROM `ports_cbqos` WHERE `port_id` = '']
SQL[DELETE FROM `ports_vlans` WHERE `port_id` = '']
SQL[DELETE FROM `pseudowires` WHERE `port_id` = '']
SQL[DELETE FROM `vlans_fdb` WHERE `port_id` = '']
SQL[DELETE FROM `ports` WHERE `port_id` = '']
SQL[DELETE FROM `ports_stack` WHERE `port_id_high` = '' OR `port_id_low` = '']
SQL[DELETE FROM `links` WHERE `local_port_id` = '' OR `remote_port_id` = '']
SQL[DELETE FROM `entity_permissions` WHERE `entity_type` = 'port' AND `entity_id` = '']
SQL[SELECT * FROM `devices` WHERE `device_id` = ''] sh: 1: port_id: not found!
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Pedersen, Sean Sent: Thursday, July 17, 2014 9:42 AM To: Observium Network Observation System Subject: Re: [Observium] Housekeeping script
"Will be deleted 21974152 eventlog entries from before 2014-01-15 21:38:46 (y/N):"
...welp. Guess this came at a good time. ;-)
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Tom Laermans Sent: Thursday, July 17, 2014 9:27 AM To: Observium Network Observation System Subject: Re: [Observium] Housekeeping script
It's PHP - PHP can do math. 30*86400 works ;) But Mike patched it so 30d should work, if I understand correctly :)
Tom
On 17/07/2014 18:21, Nikolay Shopik wrote:
Woohoo, finally. Thanks Tom. Am I suppose to enter integers in seconds or "30*86400" as value do same trick?
On 17 июля 2014 г., at 15:49, Tom Laermans tom.laermans@powersource.cx wrote:
Hi everyone,
Yesterday I committed a housekeeping script to clean up after Observium.
If you have a nice number of devices combined with a long running time of Observium, you're sure to have a gigantic database of event log, syslog and performance timing information. Some of you have resorted to periodically truncating tables - fear no more, housekeeping.php is here!
You have a choice of specifying either -a for all modules or s, e, r, p and/or t for specific things. Apart from that, there are options in the configuration to set before most of these things actually do anything.
What it does: -s Cleans up syslog data, based on the age of the entries.
Configured by $config['housekeeping']['syslog']['age'] = 0; // Maximum age of syslog entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
-e Cleans up eventlog data, based on the age of the entries.
Configured by $config['housekeeping']['eventlog']['age'] = 0; // Maximum age of eventlog entries in seconds; 0 to disable i.e.30*86400 for 30 days.
-r Cleans up RRD files, there are 2 options here:
Cleanup of RRD files with a modification time of a certain age. Ports that are gone, sensors that are gone, erroneaously created RRD files due to bugs or hardware changes, etc.
Configured by $config['housekeeping']['rrd']['age'] = 0; // Maximum age of unused rrd files in seconds before automatically purging; 0 to disable i.e. 90*86400 for 90 days.
Cleanup of invalid RRD files. Requires $config['file'] to be set correctly to the path of the "file" magic utility. Runs over all .rrd files to see if they are identified as RRDTool Databases. They can sometimes get corrupted (0 bytes, half a file) when your disk/ramdrive is full. When they exist, they are not recreated even if they're invalid. Consequently RRDtool refuses to add data into them as they are not valid RRD database files. This scripts deletes any .rrd file not identified as RRDTool.
Configured by $config['housekeeping']['rrd']['invalid'] = TRUE; // Delete .rrd files that are not valid RRD files (eg created with a full disk)
Note: this defaults to true currently. Handle with care.
-p Cleans up deleted ports. You can do this manually by using the purge button in the web interface, but one could probably safely assume if a port has not returned after 30 days it really is gone. Shorter could be useful in your environment, mine is set to 7 days.
Configured by $config['housekeeping']['deleted_ports']['age'] = 0; // Maximum age of deleted ports in seconds before automatically purging; 0 to disable i.e. 30*86400 for 30 days.
-t: Cleans up perf time information. Observium keeps performance times per module, per device and per run. This quickly accumulates into a giant number of rows in the database.
Configured by $config['housekeeping']['timing']['age'] = 0; // Maximum age of timing (discovery and poll time) entries in seconds; 0 to disable i.e. 7*86400 for 7 days.
Housekeeping logs into housekeeping.log in the log_dir. (*)
I recommend putting this into cron, daily or weekly (yes, even though the age is set in seconds, I don't recommend running it every minute ;), but only after you've run it manually once. It'll take quite a while the first time to clean up large databases. The "check for invalid RRD files" is probably quite I/O heavy.
i'll be documenting this on the wiki soonish.
(*) log_dir was also added as a new configuration directive, it defaults to ${installdir}/logs which will be created automatically for you if possible.
Tom _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Founded in 2007, IO is a worldwide leader in software defined data center technology, services and solutions that enable businesses and governments to intelligently control their information.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Founded in 2007, IO is a worldwide leader in software defined data center technology, services and solutions that enable businesses and governments to intelligently control their information.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Mike Stupalov http://observium.org/ _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (6)
-
Mike Stupalov -
Moore, Cameron -
Nikolay Shopik -
Pedersen, Sean -
Tom Laermans -
Ximena Cardinali