Hi,
Ok, so I believe the fix you mention is :
RRDtool 1.4.8 - 2012-05-23 ========================== Highlights ---------- * rrd_graph now uses a map to lookup variable names causing graphs with many items to be drawn magnitudes faster as the linear search of the variable tables is gone now.
I am using 1.4.7 and will see if I can upgrade easily to 1.4.8 to test if it speeds things up. Best regards, Louis
January 12 2015 4:10 PM, "Сережка Хомяков" wrote:
Hi!
In the March 2013 at the openNMS user conference, Tobias told he found a non-optimal code in the RRDtool, which lead to 10 times slowdown of graphing when using devices with a lot of interfaces. This seems to be relevant for initial requestor - he has about 100 ports per device. Tobias proposed a fix-up for this in the future releases. As of March 2013 RRDtool was version 1.4.7 (and is still this version for Ubuntu APT repos), so this should be fixed in 1.4.8 or 1.4.9 Now even RRDtool 1.5.0 is available, if anyone could get it working it might speed-up graphing.
HTH
-- Best, Sergey
понедельник, 12 января 2015г., 17:09 +0300 от "Louis Bailleul" :
Hi,
I have the same behaviour with Chrome 39, Firefox 34 and even IE 8.
I tried changing Firefox network.http.max-persistent-connections-per-server; from the default 6 to 32 , 256 and even 4096.
This doesn't seems to provide the expected behaviour as it only creates more httpd forks (188 counted with the 4096 setting) but still only one of them seems to be calling rrdtool.
There is no speed up in the graph rendering and even worse, with 256 some graphs failed with "No Auth" and with 4096 the page stop loading before the end leaving missing graphs.
By the way I am using apache 2.2-15 with MPM prefork if that matters.
Best regards, Louis
January 12 2015 1:33 PM, "Tom Laermans" < tom.laermans@powersource.cx (mailto:tom.laermans@powersource.cx) > wrote:
Hi,
For what Observium's concerned:
- Every image is an tag in HTML, as such it is a separate request to the webserver
- Images are created by graph.php, which calls rrdtool once (twice if there's an error, to give you the "error" graph)
- Your browser contacts the webserver for all assets it thinks it needs to load.
What may be at play here are your browser settings, which limit the number of simultaneous connections to a single webserver, to not overload it with a barrage of requests, using http keepalives to request one image after the other.
Try fiddling with browser-side settings and see if you can get it to hammer your Observium server harder.
Tom
On 01/12/2015 12:19 PM, Louis Bailleul wrote:
Alright, I'll give a go to the ramdisk when I have the chance.
Something that I noticed when digging around was that rrdtool seems to be unable to be called more than once by httpd :
httpd-+-httpd |-httpd |-httpd |-httpd---rrdtool |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd `-httpd
At any point in time only one of the httpd process has a forked rrdtool process.
Is it the expected behaviour ?
There is about 300 graphs to render during the output above, so I expected to have all httpd process trying to render graphs.
Best regards, Louis
January 12 2015 10:41 AM, "Markus Klock" < markus@best-practice.se (mailto:markus@best-practice.se) > wrote:
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive. An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html (http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html) SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster. /Markus 2015-01-12 10:52 GMT+01:00 Louis Bailleul < louis.bailleul@phangos.fr (mailto:louis.bailleul@phangos.fr) > :
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create.
What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" < adama@memetic.org (mailto:adama@memetic.org) > wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs. adam. ------ Original Message ------ From: "Louis Bailleul" < louis.bailleul@phangos.fr (mailto:louis.bailleul@phangos.fr) > To: "Observium Network Observation System" < observium@observium.org (mailto:observium@observium.org) > Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" < mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx (mailto:3Ctom.laermans@powersource.cx)%3E > wrote: > >Hi, > >You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O). > >Tom > >On 01/09/2015 04:08 PM, Pedersen, Sean wrote: >>It might help if you included a couple of references/examples of the pages that are slow. >> >>To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports. >> >>I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds. >> >>Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background. >> >>From: Louis Bailleul < louis.bailleul@phangos.fr (mailto:louis.bailleul@phangos.fr) > >>Reply-To: " observium@observium.org (mailto:observium@observium.org) " < observium@observium.org (mailto:observium@observium.org) > >>Date: Friday, January 9, 2015 at 7:46 AM >>To: " observium@observium.org (mailto:observium@observium.org) " < observium@observium.org (mailto:observium@observium.org) > >>Subject: [Observium] Speed up graph rendering >> >>Hi, >> >>I am currently using Observium on 149 switches with a total of 10173 ports. >>I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs. >> >>Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. >>It's virtual disks are on a local raid5 composed of 10K SAS disks. >> >>Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. >>The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN. >> >>Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ? >> >>Are they special requirements/tunnings for the rrd storage or the mysql database ? >> >> >>Best regards, >>Louis >> >> >> >> >> >>Founded in 2007, IO provides the data center as a service to businesses and governments around the world. >> >>The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. >> >>
_______________________________________________ observium mailing list
>>observium@observium.org (mailto:observium@observium.org) >>http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium) >
observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
_______________________________________________ observium mailing list
observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
_______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)