rrdcached worked for me! (Or "How to solve gaps in your graphs")
I was seeing gaps in my Observium graphs, so I knew that something was wrong with my system.
The output from the Observium "poller-wrapper.py" cron job has statistics on how long it takes to complete each polling cycle. These are scheduled every 5 minutes (300 seconds). Here is what I found:
"INFO: poller-wrapper polled 312 devices in 1142 seconds with 20 workers WARNING: the process took more than 5 minutes to finish, you need faster hardware or more threads INFO: in sequential style polling the elapsed time would have been: 21663 seconds WARNING: Consider setting a minimum of 77 threads. (This does not constitute professional advice!)"
I tried increasing threads on the poller-wrapper.py cron job, but this did not speed things up. The "top" command showed me that this system was always waiting for disk I/O, which is indicated by a high value for "wa". I decided to give rrdcached a try, after reading this blog:
http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
With rrdcached enabled, I am finally completing my polls with 5-minutes!
INFO: poller-wrapper polled 312 devices in 51 seconds with 20 workers
Yes, it went from 1142 seconds to 51 seconds, over 22 times faster! I know people have said the web interface would be slower, but from my perspective it is loading much faster now that rrdcached is saving some disk IOPS for Apache. (Disclaimer: I have compiled and installed rrdtool 1.4.9 to try to maximize speed on the web-interface)
I hope this help people who are bottle-necked on IOPS from their disks, like I was.
Cheers,
Tristan
*Tristan Rhodes* Network Engineer Weber State University 801.626.8549
This is the kind of instance where rrdcached is likely to help, massively overloaded I/O.
adam. On 30/01/2015 21:30:05, Tristan Rhodes tristanrhodes@weber.edu wrote: I was seeing gaps in my Observium graphs, so I knew that something was wrong with my system.
The output from the Observium "poller-wrapper.py" cron job has statistics on how long it takes to complete each polling cycle. These are scheduled every 5 minutes (300 seconds). Here is what I found:
"INFO: poller-wrapper polled 312 devices in 1142 seconds with 20 workers WARNING: the process took more than 5 minutes to finish, you need faster hardware or more threads INFO: in sequential style polling the elapsed time would have been: 21663 seconds WARNING: Consider setting a minimum of 77 threads. (This does not constitute professional advice!)"
I tried increasing threads on the poller-wrapper.py cron job, but this did not speed things up. The "top" command showed me that this system was always waiting for disk I/O, which is indicated by a high value for "wa". I decided to give rrdcached a try, after reading this blog:
http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html]
With rrdcached enabled, I am finally completing my polls with 5-minutes!
INFO: poller-wrapper polled 312 devices in 51 seconds with 20 workers
Yes, it went from 1142 seconds to 51 seconds, over 22 times faster! I know people have said the web interface would be slower, but from my perspective it is loading much faster now that rrdcached is saving some disk IOPS for Apache. (Disclaimer: I have compiled and installed rrdtool 1.4.9 to try to maximize speed on the web-interface)
I hope this help people who are bottle-necked on IOPS from their disks, like I was.
Cheers,
Tristan
Tristan Rhodes Network Engineer Weber State University
801.626.8549 [tel:801.626.8549]
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Which version of rrdtool?
-- Eduardo Schoedler
2015-01-30 19:29 GMT-02:00 Tristan Rhodes tristanrhodes@weber.edu:
I was seeing gaps in my Observium graphs, so I knew that something was wrong with my system.
The output from the Observium "poller-wrapper.py" cron job has statistics on how long it takes to complete each polling cycle. These are scheduled every 5 minutes (300 seconds). Here is what I found:
"INFO: poller-wrapper polled 312 devices in 1142 seconds with 20 workers WARNING: the process took more than 5 minutes to finish, you need faster hardware or more threads INFO: in sequential style polling the elapsed time would have been: 21663 seconds WARNING: Consider setting a minimum of 77 threads. (This does not constitute professional advice!)"
I tried increasing threads on the poller-wrapper.py cron job, but this did not speed things up. The "top" command showed me that this system was always waiting for disk I/O, which is indicated by a high value for "wa". I decided to give rrdcached a try, after reading this blog:
http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
With rrdcached enabled, I am finally completing my polls with 5-minutes!
INFO: poller-wrapper polled 312 devices in 51 seconds with 20 workers
Yes, it went from 1142 seconds to 51 seconds, over 22 times faster! I know people have said the web interface would be slower, but from my perspective it is loading much faster now that rrdcached is saving some disk IOPS for Apache. (Disclaimer: I have compiled and installed rrdtool 1.4.9 to try to maximize speed on the web-interface)
I hope this help people who are bottle-necked on IOPS from their disks, like I was.
Cheers,
Tristan
*Tristan Rhodes* Network Engineer Weber State University 801.626.8549
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (3)
-
Adam Armstrong -
Eduardo Schoedler -
Tristan Rhodes