Re: [Observium] Speed up graph rendering

30 Jan 2015


      Perhaps someone would like to make 1.4.9 or 1.5.0-rc packages for ubuntu 
and/or centos/rhel?
:>
adam.
On 2015-01-29 16:01, Simon Smith wrote:
...
Hi,
I used that page to compile the tool myself
and I can run the 1.4.9 and 1.4.7 side by side,
I just installed the bin into /opt/rrdtool-1.4.9/bin/rrdtool instead
of the default folder
and changed the config.php to that folder
Simon
...
On 29 Jan 2015, at 3:54 pm, Tristan Rhodes tristanrhodes@weber.edu
wrote:
I assume you compiled rrdtool from source?
http://oss.oetiker.ch/rrdtool/doc/rrdbuild.en.html [1]
Are these the correct steps to do this?

Stop observium cronjobs
Uninstall rrdtool package (apt-get or yum)
Compile new version of rrdtool following instructions above
Move or link "rrdtool" to /usr/bin/rrdtool, or change

"config.php" to where you compiled it:
$config['rrdtool'] = "/usr/bin/rrdtool";
I guess I'll try this and see how bad I break my system... :)
Cheers,
Tristan
TRISTAN RHODES
Network Engineer
Weber State University
801.626.8549
On Thu, Jan 29, 2015 at 4:21 AM, Сережка Хомяков
xomka686@mail.ru wrote:
Hi!
That's great! I'm on the way of testing RRDtool 1.4.9 too. Now, even
RRDtool 1.5.0 is available.
--
Best, Sergey
Четверг, 29 января 2015, 9:50 UTC от "Louis Bailleul"
louis.bailleul@phangos.fr:
Hi,
Sorry for the delay but I couldn't test it before.
I finally managed to replace rrdtool 1.4.7 by rrdtool 1.4.9.
And I am glad I did!
There is indeed something broken with 1.4.7 because the speed up is
quite amazing.
The web interface is now blazing fast, even for my switches with
700+ ports.
Pages which used to render in between 8 and 12min now are displayed
in 34 seconds.
Thanks a lot for pointing this out and keep up the good work on
this amazing tool.
Best regards,
Louis
January 13 2015 6:59 PM, "Adam Armstrong" <adama@memetic.org [2]>
wrote:
...
If rrd write performance (poller runtime) is an issue, you might
benefit
...
from rrdcached (or SSD/ramdisk storage).
If rrd read performance is an issue, you will be better served by
ssd or
...
ramdisk storage. rrdcached /might/ give some performance
improvements
...
from reducing write i/o load, but the problem with rrdcached is
that in
...
order to read an rrd you need to signal to rrdcached to flush its
cache
...
so you can read the rrd, adding an extra step, which may not be
trivial
...
in terms of performance on our graph-heavy pages.
I still say that the best performance comes from ramdisk, with
ssd an
...
option too.
I've run all of my Observium systems from ramdisk for years, with
...
automated backup/restore scripts. If I reboot a server, I only
lose data
...
since the last backup (backing up rrd data with fast compression
is
...
pretty quick, so you can run it fairly frequently).
My dislike of rrdcached comes from it never really helping in any
of the
...
instances i had performance problems. Sure, it makes polling
slightly
...
quicker, but it doesn't really do anything for the loading of
graphs,
...
which is where the human<>computer irritation comes from.
adam.
On 2015-01-13 00:36, Tristan Rhodes wrote:
...
This is all good info. I haven't used rrdcached before but I
think I
...
...
will give it a shot.
Regarding writes versus reads, aren't there many, many times the
amount of writes compared to reads?
For instance, the RRDs for all components on all devices are
written
...
...
every 5 mins, correct?
Reads only occur when someone is browsing the website.
Am I correct about this?
Cheers,
Tristan
TRISTAN RHODES
Network Engineer
Weber State University
801.626.8549 [3]
On Mon, Jan 12, 2015 at 3:41 PM, Adam Armstrong
<adama@memetic.org [2]>
...
...
wrote:
Not that rrdcached only speeds up RRD writing, which is why we
don't
...
...
really use it ourselves.
It actually slows down reading (generating graphs), too!
adam.
On 2015-01-12 10:42, Markus Klock wrote:
The ramdisk itself is not very reliable no, the trick is to make
a
...
...
backup of the ramdisk to a reglar harddrive or SSD once in a
while so
...
...
that in a case of power outage or unexpected reboot you could
just
...
...
copy the data back to the ramdisk from the harddrive.
An option to ramdisk would be to use rrdcached. In large
enviroments
...
...
rrdcached could cut I/O in half or more. A quick guide can be
found
...
...
here:
http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
...
[4]
...
...
[1]
[2]
SSD disks + rrdcached could handle a pretty big enviroment. But
using
...
...
a ramdisk is still very very much faster.
/Markus
2015-01-12 10:52 GMT+01:00 Louis Bailleul
<louis.bailleul@phangos.fr [5]>:
...
...
Hi Adam ,
Is ramdisk storage reliable enough ?
Not that I have no faith in the reliability of our setup, but
storing
...
...
large amount of data in ram isn't a little risky ?
Currently our rrd dir is
du -sh rrd/
21G rrd/
That fit in a 24-32Go ram VM which is something that we can
create.
...
...
What do you think of the mysql tuning that have been suggested
by
...
...
Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards,
Louis
January 9 2015 5:01 PM, "Adam Armstrong" <adama@memetic.org [2]>
wrote:
...
...
Depending upon your deployment size, I usually recommend ramdisk
...
...
storage for RRDs.
adam.
------ Original Message ------
From: "Louis Bailleul" <louis.bailleul@phangos.fr [5]>
To: "Observium Network Observation System"
<observium@observium.org [6]>
...
...
Sent: 1/9/2015 10:51:40 AM
Subject: Re: [Observium] Speed up graph rendering
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch
with 760
...
...
ports take 0.567 seconds.
But the loading of all graphs (a lot of sensors) take 8m 53s
according
...
...
to firebug.
Generating the ports view page takes 1.421s but loading all the
graphs
...
...
takes 12m 33s and 493 HTTP request (maybe not accurate as
firebug says
...
...
the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds
and
...
...
the page loads in 2.73s.
The ports view takes 0.311 seconds to generate and the page
finishes
...
...
loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause
but
...
...
maybe the RRD disk I/O.
What can I do to speed it up ?
Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of
call to
...
...
graph.php ?
For the record the disk system is an IBM V3700 SAS attached to
the ESXi
...
...
hosts.
Best regards,
Louis
January 9 2015 3:11 PM, "Tom Laermans"
mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx%3E
...
...
wrote:
Hi,
You must note however, that the load time for the page as
displayed by
...
...
Observium is the generation time of the HTML (which is
influenced by
...
...
MySQL speed and CPU, mostly), which includes image tags to load
the
...
...
graphs - but not the loading of the graphs themselves (which is
heavily
...
...
influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote:
It might help if you included a couple of references/examples of
the
...
...
pages that are slow.
To give you something to compare against, we’re running
Observium on an
...
...
ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage
– but
...
...
we’re at a pretty consistent 40-50% utilization at all times,
using
...
...
about 2GB of RAM. This is with ~200 devices and 16K ports.
I looked at a Cisco switch stack that has 115 active ports.
Graphs >
...
...
System loaded in .45 seconds, which is only 16 different graphs
between
...
...
four categories. Likewise, the Ports page loaded at .64 seconds
with
...
...
the miniature preview graphs. Clicking on a specific port, it
loaded 24
...
...
graphs in .49 seconds.
Now…I have some custom port lists, one of which is 36 graphs,
which
...
...
loaded in .53 seconds. However, I have another group that
has…a metric
...
...
shit-tonne of ports and graphs on it – hundreds of them. The
page
...
...
loaded in 1.07 seconds, but it took several seconds more for all
of the
...
...
individual graphs to populate, which I expect considering the
sheer
...
...
number of them present in that page. I don’t know if that’s
simply the
...
...
browser catching up, or if it’s still generating the actual
graphs in
...
...
the background.
From: Louis Bailleul <louis.bailleul@phangos.fr [5]>
Reply-To: "observium@observium.org [6]" <observium@observium.org
[6]>
...
...
Date: Friday, January 9, 2015 at 7:46 AM
To: "observium@observium.org [6]" <observium@observium.org [6]>
Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of
10173
...
...
ports.
I can't say that anything is wrong with it, but the web
interface seems
...
...
a bit slow especially when you want to display pages with a lot
of
...
...
graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores
and
...
...
3Gb of ram.
It's virtual disks are on a local raid5 composed of 10K SAS
disks.
...
...
Observium report that it's VM use on average 25% of its CPUs
with
...
...
spikes at 40% when the web interface is heavily used.
The disk I/O doesn't go past 150 ops OUT and rare spikes at 50
IN.
...
...
Anyone could advise on what kind of hardware or software
tweaking could
...
...
help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or
the mysql
...
...
database ?
Best regards,
Louis
FOUNDED IN 2007, IO PROVIDES THE DATA CENTER AS A SERVICE TO
BUSINESSES
...
...
AND GOVERNMENTS AROUND THE WORLD.
The communication contained in this e-mail is confidential and
is
...
...
intended only for the named recipient(s) and may contain
information
...
...
that is privileged, proprietary, attorney work product or exempt
from
...
...
disclosure under applicable law. If you have received this
message in
...
...
error, or are not the named recipient(s), please note that any
form of
...
...
distribution, copying or use of this communication or the
information
...
...
in it is strictly prohibited and may be unlawful. Please
immediately
...
...
notify the sender of the error, and delete this communication
including
...
...
any attached files from your system. Thank you for your
cooperation.
...
...

observium mailing list
observium@observium.org [6]
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
[7] [2] [1]
...
...

observium mailing list
observium@observium.org [6]
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
[7] [2] [1]
...
...
Links:
[1]
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [7]
[2]
...
...
[2]
http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
...
[4]
...
...
[1]

observium mailing list
observium@observium.org [6]
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
[7] [2]
...
...

observium mailing list
observium@observium.org [6]
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
[7] [2]
...
...
Links:
[1]
http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
...
[4]
...
...
[2]
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [7]
...
...

observium mailing list
observium@observium.org [6]
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
[7]
...

observium mailing list
observium@observium.org [6]
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [7]

observium mailing list
observium@observium.org [6]
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [7]

observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [7]

observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Links:
[1] http://oss.oetiker.ch/rrdtool/doc/rrdbuild.en.html
[2] http:/compose?To=adama@memetic.org
[3] tel:801.626.8549
[4] 
http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
[5] http:/compose?To=louis.bailleul@phangos.fr
[6] http:/compose?To=observium@observium.org
[7] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium