Reduce your storage IO with rrdcached
There was some discussion on IRC the other day about rrdcached and how it can be used with Observium to reduce your storage IO a lot when polling a huge amount of interfaces. However, not a lot of documentation exists yet on how to get rrdcached to work with observium. I did some trial and error and found a working method and wrote a quick and dirty guide. You can find it here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html Feedback is highly appreciated!
/Markus
I came across the permissions issue as well when I played around with it. You can set the permissions on the socket, but you need to put those arguments (-P <perm> -s <group>) before you specify the socket.
However, if you are integrating any other RRD source (e.g smokeping), then you will also need to make that rrdcached aware as well because observium will only look at rrdcached for its rrd’s if you enable rrdcached. I couldn’t find any integration support for rrdcached in smokeping, but I don’t think it should be hard to add since it all goes via the rrdtool perl module.
-- Milton Ngan Sent with Airmail
On October 23, 2014 at 2:27:28 PM, Markus Klock (markus@best-practice.semailto:markus@best-practice.se) wrote:
There was some discussion on IRC the other day about rrdcached and how it can be used with Observium to reduce your storage IO a lot when polling a huge amount of interfaces. However, not a lot of documentation exists yet on how to get rrdcached to work with observium. I did some trial and error and found a working method and wrote a quick and dirty guide. You can find it here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html Feedback is highly appreciated!
/Markus _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Thank you Milton, you were right, setting the -s flag before -l solves the whole problem. Updated the blogpost! /Markus
2014-10-23 23:39 GMT+02:00 Milton Ngan milton@valvesoftware.com:
I came across the permissions issue as well when I played around with it. You can set the permissions on the socket, but you need to put those arguments (-P <perm> -s <group>) before you specify the socket.
However, if you are integrating any other RRD source (e.g smokeping), then you will also need to make that rrdcached aware as well because observium will only look at rrdcached for its rrd’s if you enable rrdcached. I couldn’t find any integration support for rrdcached in smokeping, but I don’t think it should be hard to add since it all goes via the rrdtool perl module.
-- Milton Ngan Sent with Airmail
On October 23, 2014 at 2:27:28 PM, Markus Klock (markus@best-practice.se) wrote:
There was some discussion on IRC the other day about rrdcached and how it can be used with Observium to reduce your storage IO a lot when polling a huge amount of interfaces. However, not a lot of documentation exists yet on how to get rrdcached to work with observium. I did some trial and error and found a working method and wrote a quick and dirty guide. You can find it here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html Feedback is highly appreciated!
/Markus _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
FWIW, I just looked through this quickly and wanted to recommend changing item 3 to edit options in /etc/default/rrdcached. Testing this now.... thanks for writing that up!
Regards, Mark
On 10/24/2014 12:37 AM, Markus Klock wrote:
Thank you Milton, you were right, setting the -s flag before -l solves the whole problem. Updated the blogpost! /Markus
2014-10-23 23:39 GMT+02:00 Milton Ngan <milton@valvesoftware.com mailto:milton@valvesoftware.com>:
I came across the permissions issue as well when I played around with it. You can set the permissions on the socket, but you need to put those arguments (-P <perm> -s <group>) before you specify the socket. However, if you are integrating any other RRD source (e.g smokeping), then you will also need to make that rrdcached aware as well because observium will only look at rrdcached for its rrd’s if you enable rrdcached. I couldn’t find any integration support for rrdcached in smokeping, but I don’t think it should be hard to add since it all goes via the rrdtool perl module. -- Milton Ngan Sent with Airmail On October 23, 2014 at 2:27:28 PM, Markus Klock (markus@best-practice.se <mailto:markus@best-practice.se>) wrote:There was some discussion on IRC the other day about rrdcached and how it can be used with Observium to reduce your storage IO a lot when polling a huge amount of interfaces. However, not a lot of documentation exists yet on how to get rrdcached to work with observium. I did some trial and error and found a working method and wrote a quick and dirty guide. You can find it here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html <https://canit.willingminds.com/canit/urlproxy.php?q=aHR0cDovL2Jsb2cuYmVzdC1wcmFjdGljZS5zZS8yMDE0LzEwL3VzaW5nLXJyZGNhY2hlZC13aXRoLW9ic2Vydml1bS5odG1s> Feedback is highly appreciated!
Thank you Mark! That is a much better solution, did not find that configfile at first :) Updated the blogpost /Markus
2014-10-24 17:49 GMT+02:00 Mark D. Nagel mnagel@willingminds.com:
FWIW, I just looked through this quickly and wanted to recommend changing item 3 to edit options in /etc/default/rrdcached. Testing this now.... thanks for writing that up!
Regards, Mark
On 10/24/2014 12:37 AM, Markus Klock wrote:
Thank you Milton, you were right, setting the -s flag before -l solves the whole problem. Updated the blogpost! /Markus
2014-10-23 23:39 GMT+02:00 Milton Ngan milton@valvesoftware.com:
I came across the permissions issue as well when I played around with it. You can set the permissions on the socket, but you need to put those arguments (-P <perm> -s <group>) before you specify the socket.
However, if you are integrating any other RRD source (e.g smokeping), then you will also need to make that rrdcached aware as well because observium will only look at rrdcached for its rrd’s if you enable rrdcached. I couldn’t find any integration support for rrdcached in smokeping, but I don’t think it should be hard to add since it all goes via the rrdtool perl module.
-- Milton Ngan Sent with Airmail
On October 23, 2014 at 2:27:28 PM, Markus Klock (markus@best-practice.se) wrote:
There was some discussion on IRC the other day about rrdcached and how it can be used with Observium to reduce your storage IO a lot when polling a huge amount of interfaces. However, not a lot of documentation exists yet on how to get rrdcached to work with observium. I did some trial and error and found a working method and wrote a quick and dirty guide. You can find it here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html https://canit.willingminds.com/canit/urlproxy.php?q=aHR0cDovL2Jsb2cuYmVzdC1wcmFjdGljZS5zZS8yMDE0LzEwL3VzaW5nLXJyZGNhY2hlZC13aXRoLW9ic2Vydml1bS5odG1s Feedback is highly appreciated!
-- Mark D. Nagel, CCIE #3177 mnagel@willingminds.com mnagel@willingminds.com Principal Consultant, Willing Minds LLC (http://www.willingminds.com) cell: 949-279-5817, desk: 714-495-4001, fax: 714-646-8277
** For faster support response time, please ** email support@willingminds.com or call 714-495-4000
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Nice stuff. There is some interesting reading here on tuning dirty writes - https://code.google.com/p/epicnms/wiki/Scaling
From: Markus Klock <markus@best-practice.semailto:markus@best-practice.se> Reply-To: Observium <observium@observium.orgmailto:observium@observium.org> Date: Friday, 24 October 2014 7:57 am To: Observium <observium@observium.orgmailto:observium@observium.org> Subject: [Observium] Reduce your storage IO with rrdcached
There was some discussion on IRC the other day about rrdcached and how it can be used with Observium to reduce your storage IO a lot when polling a huge amount of interfaces. However, not a lot of documentation exists yet on how to get rrdcached to work with observium. I did some trial and error and found a working method and wrote a quick and dirty guide. You can find it here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html Feedback is highly appreciated!
/Markus
Nice, it's working.
But I see it's not working when using the weathermap (it can't read the last rrd values anymore so all links are shown as « 0 »).
Have you any idea of workaround for this ? I tried to reduce a lot the timers but I completely lost the performance advantages (weathermap is generated every 5 minutes in my setup)
De : observium [mailto:observium-bounces@observium.org] De la part de Peter Childs Envoyé : vendredi 24 octobre 2014 05:44 À : Observium Network Observation System Objet : Re: [Observium] Reduce your storage IO with rrdcached
Nice stuff. There is some interesting reading here on tuning dirty writes - https://code.google.com/p/epicnms/wiki/Scaling
From: Markus Klock <markus@best-practice.semailto:markus@best-practice.se> Reply-To: Observium <observium@observium.orgmailto:observium@observium.org> Date: Friday, 24 October 2014 7:57 am To: Observium <observium@observium.orgmailto:observium@observium.org> Subject: [Observium] Reduce your storage IO with rrdcached
There was some discussion on IRC the other day about rrdcached and how it can be used with Observium to reduce your storage IO a lot when polling a huge amount of interfaces. However, not a lot of documentation exists yet on how to get rrdcached to work with observium. I did some trial and error and found a working method and wrote a quick and dirty guide. You can find it here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html Feedback is highly appreciated!
/Markus
Do you mean 'weathermap' as in the third party plugin ? If so you would need to ensure weathermap hits rrdcached to do a flush when it does a fetch.
From: Mathieu POUSSIN <Mathieu.POUSSIN@oxalide.commailto:Mathieu.POUSSIN@oxalide.com> Reply-To: Observium <observium@observium.orgmailto:observium@observium.org> Date: Friday, 24 October 2014 11:59 pm To: Observium <observium@observium.orgmailto:observium@observium.org> Subject: Re: [Observium] Reduce your storage IO with rrdcached
Nice, it’s working.
But I see it’s not working when using the weathermap (it can’t read the last rrd values anymore so all links are shown as « 0 »).
Have you any idea of workaround for this ? I tried to reduce a lot the timers but I completely lost the performance advantages (weathermap is generated every 5 minutes in my setup)
De : observium [mailto:observium-bounces@observium.org] De la part de Peter Childs Envoyé : vendredi 24 octobre 2014 05:44 À : Observium Network Observation System Objet : Re: [Observium] Reduce your storage IO with rrdcached
Nice stuff. There is some interesting reading here on tuning dirty writes - https://code.google.com/p/epicnms/wiki/Scaling
From: Markus Klock <markus@best-practice.semailto:markus@best-practice.se> Reply-To: Observium <observium@observium.orgmailto:observium@observium.org> Date: Friday, 24 October 2014 7:57 am To: Observium <observium@observium.orgmailto:observium@observium.org> Subject: [Observium] Reduce your storage IO with rrdcached
There was some discussion on IRC the other day about rrdcached and how it can be used with Observium to reduce your storage IO a lot when polling a huge amount of interfaces. However, not a lot of documentation exists yet on how to get rrdcached to work with observium. I did some trial and error and found a working method and wrote a quick and dirty guide. You can find it here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html Feedback is highly appreciated!
/Markus
participants (5)
-
Mark D. Nagel -
Markus Klock -
Mathieu POUSSIN -
Milton Ngan -
Peter Childs