Hi,

Sorry for the delay but I couldn't test it before.

I finally managed to replace rrdtool 1.4.7 by rrdtool 1.4.9.

And I am glad I did!

There is indeed something broken with 1.4.7 because the speed up is quite amazing.

The web interface is now blazing fast, even for my switches with 700+ ports.
Pages which used to render in between 8 and 12min now are displayed in 34 seconds.

Thanks a lot for pointing this out and keep up the good work on this amazing tool.

Best regards,
Louis

January 13 2015 6:59 PM, "Adam Armstrong" <adama@memetic.org> wrote:
> If rrd write performance (poller runtime) is an issue, you might benefit
> from rrdcached (or SSD/ramdisk storage).
>
> If rrd read performance is an issue, you will be better served by ssd or
> ramdisk storage. rrdcached /might/ give some performance improvements
> from reducing write i/o load, but the problem with rrdcached is that in
> order to read an rrd you need to signal to rrdcached to flush its cache
> so you can read the rrd, adding an extra step, which may not be trivial
> in terms of performance on our graph-heavy pages.
>
> I still say that the best performance comes from ramdisk, with ssd an
> option too.
>
> I've run all of my Observium systems from ramdisk for years, with
> automated backup/restore scripts. If I reboot a server, I only lose data
> since the last backup (backing up rrd data with fast compression is
> pretty quick, so you can run it fairly frequently).
>
> My dislike of rrdcached comes from it never really helping in any of the
> instances i had performance problems. Sure, it makes polling slightly
> quicker, but it doesn't really do anything for the loading of graphs,
> which is where the human<>computer irritation comes from.
>
> adam.
>
> On 2015-01-13 00:36, Tristan Rhodes wrote:
>
>> This is all good info. I haven't used rrdcached before but I think I
>> will give it a shot.
>>
>> Regarding writes versus reads, aren't there many, many times the
>> amount of writes compared to reads?
>>
>> For instance, the RRDs for all components on all devices are written
>> every 5 mins, correct?
>>
>> Reads only occur when someone is browsing the website.
>>
>> Am I correct about this?
>>
>> Cheers,
>>
>> Tristan
>>
>> TRISTAN RHODES
>> Network Engineer
>> Weber State University
>> 801.626.8549
>>
>> On Mon, Jan 12, 2015 at 3:41 PM, Adam Armstrong <adama@memetic.org>
>> wrote:
>>
>> Not that rrdcached only speeds up RRD writing, which is why we don't
>> really use it ourselves.
>>
>> It actually slows down reading (generating graphs), too!
>>
>> adam.
>>
>> On 2015-01-12 10:42, Markus Klock wrote:
>>
>> The ramdisk itself is not very reliable no, the trick is to make a
>> backup of the ramdisk to a reglar harddrive or SSD once in a while so
>> that in a case of power outage or unexpected reboot you could just
>> copy the data back to the ramdisk from the harddrive.
>>
>> An option to ramdisk would be to use rrdcached. In large enviroments
>> rrdcached could cut I/O in half or more. A quick guide can be found
>> here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
>> [1]
>> [2]
>>
>> SSD disks + rrdcached could handle a pretty big enviroment. But using
>> a ramdisk is still very very much faster.
>>
>> /Markus
>>
>> 2015-01-12 10:52 GMT+01:00 Louis Bailleul <louis.bailleul@phangos.fr>:
>>
>> Hi Adam ,
>>
>> Is ramdisk storage reliable enough ?
>> Not that I have no faith in the reliability of our setup, but storing
>> large amount of data in ram isn't a little risky ?
>>
>> Currently our rrd dir is
>> du -sh rrd/
>> 21G rrd/
>>
>> That fit in a 24-32Go ram VM which is something that we can create.
>>
>> What do you think of the mysql tuning that have been suggested by
>> Eduardo and Alastair ?
>>
>> Is there any other option than ramdisk storage ?
>>
>> Best regards,
>> Louis
>>
>> January 9 2015 5:01 PM, "Adam Armstrong" <adama@memetic.org> wrote:
>>
>> Depending upon your deployment size, I usually recommend ramdisk
>> storage for RRDs.
>>
>> adam.
>>
>> ------ Original Message ------
>> From: "Louis Bailleul" <louis.bailleul@phangos.fr>
>> To: "Observium Network Observation System" <observium@observium.org>
>> Sent: 1/9/2015 10:51:40 AM
>> Subject: Re: [Observium] Speed up graph rendering
>>
>> Hi,
>>
>> Thank you for the pointers.
>>
>> To be more specific the page generation for an arista switch with 760
>> ports take 0.567 seconds.
>> But the loading of all graphs (a lot of sensors) take 8m 53s according
>> to firebug.
>>
>> Generating the ports view page takes 1.421s but loading all the graphs
>> takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says
>> the log limit was reached).
>>
>> On a force10 switch with 48 ports page generates in 0.207seconds and
>> the page loads in 2.73s.
>> The ports view takes 0.311 seconds to generate and the page finishes
>> loading graphs after 1m8s (205 HTTP requests in total).
>>
>> So from what I understand the mysql speed might not be in cause but
>> maybe the RRD disk I/O.
>> What can I do to speed it up ?
>> Moving the disk on faster storage will be enough ?
>>
>> Could it be that apache need tuning to handle that amount of call to
>> graph.php ?
>>
>> For the record the disk system is an IBM V3700 SAS attached to the ESXi
>> hosts.
>>
>> Best regards,
>> Louis
>>
>> January 9 2015 3:11 PM, "Tom Laermans"
>> <mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx%3E>
>> wrote:
>>
>> Hi,
>>
>> You must note however, that the load time for the page as displayed by
>> Observium is the generation time of the HTML (which is influenced by
>> MySQL speed and CPU, mostly), which includes image tags to load the
>> graphs - but not the loading of the graphs themselves (which is heavily
>> influenced by both CPU and RRD Disk I/O).
>>
>> Tom
>>
>> On 01/09/2015 04:08 PM, Pedersen, Sean wrote:
>> It might help if you included a couple of references/examples of the
>> pages that are slow.
>>
>> To give you something to compare against, we’re running Observium on an
>> ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but
>> we’re at a pretty consistent 40-50% utilization at all times, using
>> about 2GB of RAM. This is with ~200 devices and 16K ports.
>>
>> I looked at a Cisco switch stack that has 115 active ports. Graphs >
>> System loaded in .45 seconds, which is only 16 different graphs between
>> four categories. Likewise, the Ports page loaded at .64 seconds with
>> the miniature preview graphs. Clicking on a specific port, it loaded 24
>> graphs in .49 seconds.
>>
>> Now…I have some custom port lists, one of which is 36 graphs, which
>> loaded in .53 seconds. However, I have another group that has…a metric
>> shit-tonne of ports and graphs on it – hundreds of them. The page
>> loaded in 1.07 seconds, but it took several seconds more for all of the
>> individual graphs to populate, which I expect considering the sheer
>> number of them present in that page. I don’t know if that’s simply the
>> browser catching up, or if it’s still generating the actual graphs in
>> the background.
>>
>> From: Louis Bailleul <louis.bailleul@phangos.fr>
>> Reply-To: "observium@observium.org" <observium@observium.org>
>> Date: Friday, January 9, 2015 at 7:46 AM
>> To: "observium@observium.org" <observium@observium.org>
>> Subject: [Observium] Speed up graph rendering
>>
>> Hi,
>>
>> I am currently using Observium on 149 switches with a total of 10173
>> ports.
>> I can't say that anything is wrong with it, but the web interface seems
>> a bit slow especially when you want to display pages with a lot of
>> graphs.
>>
>> Observium is currently running in a VM with 4 xeon E5-2650 cores and
>> 3Gb of ram.
>> It's virtual disks are on a local raid5 composed of 10K SAS disks.
>>
>> Observium report that it's VM use on average 25% of its CPUs with
>> spikes at 40% when the web interface is heavily used.
>> The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
>>
>> Anyone could advise on what kind of hardware or software tweaking could
>> help speed up the thing ?
>>
>> Are they special requirements/tunnings for the rrd storage or the mysql
>> database ?
>>
>> Best regards,
>> Louis
>>
>> FOUNDED IN 2007, IO PROVIDES THE DATA CENTER AS A SERVICE TO BUSINESSES
>> AND GOVERNMENTS AROUND THE WORLD.
>>
>> The communication contained in this e-mail is confidential and is
>> intended only for the named recipient(s) and may contain information
>> that is privileged, proprietary, attorney work product or exempt from
>> disclosure under applicable law. If you have received this message in
>> error, or are not the named recipient(s), please note that any form of
>> distribution, copying or use of this communication or the information
>> in it is strictly prohibited and may be unlawful. Please immediately
>> notify the sender of the error, and delete this communication including
>> any attached files from your system. Thank you for your cooperation.
>>
>> _______________________________________________
>> observium mailing list
>> observium@observium.org
>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [1]
>>
>> _______________________________________________
>> observium mailing list
>> observium@observium.org
>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [1]
>>
>> Links:
>> ------
>> [1] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2]
>> [2]
>> http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
>> [1]
>>
>> _______________________________________________
>> observium mailing list
>> observium@observium.org
>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2]
>>
>> _______________________________________________
>> observium mailing list
>> observium@observium.org
>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2]
>>
>> Links:
>> ------
>> [1]
>> http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
>> [2] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>>
>> _______________________________________________
>> observium mailing list
>> observium@observium.org
>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>
> _______________________________________________
> observium mailing list
> observium@observium.org
> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium