Starting this weekend rrdcached appears to have stopped working correctly. Has anyone seen an issue with this? We've been using it for a long time and it seems to have broken totally early Sat morning (somewhere around 1 to 2 AM)
Running the poller with debug doesn't give me a status any more, and I have hundreds of rrdtool processes being spawned to the point it actually took over most of the disk IO on our NAS on Saturday.
/opt/observium# poller.php -h localhost -m system -dd | grep RRD RRDtool 1.7.0 Copyright by Tobias Oetiker tobi@oetiker.ch RRDCacheD 1.7.0 o RRDtool 1.7.0 (rrdcached 1.7.0: unix:/var/run/rrdcached.sock) RRD PIPE OPEN[TRUE] o RRDTool Usage (0s 0%) RRD PIPE CLOSE[TRUE]
The host setup is: Ubuntu 18.04 Observium Professional 19.5.9870 PHP Weathermap 0.97b rrdcached 1.7.0-1build1 rrdtool-1.7.0-1build1 PHP 7.2
The rrdcached command line is /usr/bin/rrdcached -B -R -F -L -w 1800 -z 1800 -f 3600 -t 10 -b /opt/observium/rrd -j /var/lib/rrdcached/journal/ -G www-data -U www-data -p /var/run/rrdcached.pid -s www-data -m 0660 -l unix:/var/run/rrdcached.sock
Really need to get this one solved if anyone can offer help.
________________________________
This e-mail and any documents accompanying it may contain legally privileged and/or confidential information belonging to Exegy, Inc. Such information may be protected from disclosure by law. The information is intended for use by only the addressee. If you are not the intended recipient, you are hereby notified that any disclosure or use of the information is strictly prohibited. If you have received this e-mail in error, please immediately contact the sender by e-mail or phone regarding instructions for return or destruction and do not use or disclose the content to others.
Do you actually need rrdcached?
It slows down graph creation, and is really only useful on single systems if you are I/O write starved.
adam. On 2019-05-06 16:43:28, Hamilton, Kent via observium observium@observium.org wrote: Starting this weekend rrdcached appears to have stopped working correctly. Has anyone seen an issue with this? We’ve been using it for a long time and it seems to have broken totally early Sat morning (somewhere around 1 to 2 AM) Running the poller with debug doesn’t give me a status any more, and I have hundreds of rrdtool processes being spawned to the point it actually took over most of the disk IO on our NAS on Saturday. /opt/observium# poller.php -h localhost -m system -dd | grep RRD RRDtool 1.7.0 Copyright by Tobias Oetiker tobi@oetiker.ch RRDCacheD 1.7.0 o RRDtool 1.7.0 (rrdcached 1.7.0: unix:/var/run/rrdcached.sock) RRD PIPE OPEN[TRUE] o RRDTool Usage (0s 0%) RRD PIPE CLOSE[TRUE] The host setup is: Ubuntu 18.04 Observium Professional 19.5.9870 PHP Weathermap 0.97b rrdcached 1.7.0-1build1 rrdtool-1.7.0-1build1 PHP 7.2 The rrdcached command line is /usr/bin/rrdcached -B -R -F -L -w 1800 -z 1800 -f 3600 -t 10 -b /opt/observium/rrd -j /var/lib/rrdcached/journal/ -G www-data -U www-data -p /var/run/rrdcached.pid -s www-data -m 0660 -l unix:/var/run/rrdcached.sock Really need to get this one solved if anyone can offer help.
This e-mail and any documents accompanying it may contain legally privileged and/or confidential information belonging to Exegy, Inc. Such information may be protected from disclosure by law. The information is intended for use by only the addressee. If you are not the intended recipient, you are hereby notified that any disclosure or use of the information is strictly prohibited. If you have received this e-mail in error, please immediately contact the sender by e-mail or phone regarding instructions for return or destruction and do not use or disclose the content to others.
I’ve got it disabled at the moment and so far our storage guy hasn’t screamed at me. We started using it some time ago before we upgraded our NAS boxes because our old ones were in fact write I/O starved badly, with the newer units it might not be so bad but… why did it suddenly stop working, and why did it spawn enough processes off to totally kill I/O access to our four month old TrueNAS boxes taking our SOC monitoring platform (Running on the same SSD volume), and a couple of other servers out. Customers don’t like it much when you lose monitoring of their systems and the networks to get to them.
From: observium observium-bounces@observium.org On Behalf Of Adam Armstrong via observium Sent: Monday, May 6, 2019 10:46 AM To: Hamilton, Kent via observium observium@observium.org Cc: Adam Armstrong adama@memetic.org Subject: Re: [Observium] RRDCached issue - HELP!
Do you actually need rrdcached?
It slows down graph creation, and is really only useful on single systems if you are I/O write starved.
adam.
On 2019-05-06 16:43:28, Hamilton, Kent via observium <observium@observium.orgmailto:observium@observium.org> wrote: Starting this weekend rrdcached appears to have stopped working correctly. Has anyone seen an issue with this? We’ve been using it for a long time and it seems to have broken totally early Sat morning (somewhere around 1 to 2 AM)
Running the poller with debug doesn’t give me a status any more, and I have hundreds of rrdtool processes being spawned to the point it actually took over most of the disk IO on our NAS on Saturday.
/opt/observium# poller.php -h localhost -m system -dd | grep RRD RRDtool 1.7.0 Copyright by Tobias Oetiker <tobi@oetiker.chmailto:tobi@oetiker.ch> RRDCacheD 1.7.0 o RRDtool 1.7.0 (rrdcached 1.7.0: unix:/var/run/rrdcached.sock) RRD PIPE OPEN[TRUE] o RRDTool Usage (0s 0%) RRD PIPE CLOSE[TRUE]
The host setup is: Ubuntu 18.04 Observium Professional 19.5.9870 PHP Weathermap 0.97b rrdcached 1.7.0-1build1 rrdtool-1.7.0-1build1 PHP 7.2
The rrdcached command line is /usr/bin/rrdcached -B -R -F -L -w 1800 -z 1800 -f 3600 -t 10 -b /opt/observium/rrd -j /var/lib/rrdcached/journal/ -G www-data -U www-data -p /var/run/rrdcached.pid -s www-data -m 0660 -l unix:/var/run/rrdcached.sock
Really need to get this one solved if anyone can offer help.
________________________________
This e-mail and any documents accompanying it may contain legally privileged and/or confidential information belonging to Exegy, Inc. Such information may be protected from disclosure by law. The information is intended for use by only the addressee. If you are not the intended recipient, you are hereby notified that any disclosure or use of the information is strictly prohibited. If you have received this e-mail in error, please immediately contact the sender by e-mail or phone regarding instructions for return or destruction and do not use or disclose the content to others.
________________________________
This e-mail and any documents accompanying it may contain legally privileged and/or confidential information belonging to Exegy, Inc. Such information may be protected from disclosure by law. The information is intended for use by only the addressee. If you are not the intended recipient, you are hereby notified that any disclosure or use of the information is strictly prohibited. If you have received this e-mail in error, please immediately contact the sender by e-mail or phone regarding instructions for return or destruction and do not use or disclose the content to others.
participants (2)
-
Adam Armstrong -
Hamilton, Kent