No, it's not that big. I would have thought the current hardware could handle it, but it doesn't.

I think part of it is that we end up with a lot of deleted interfaces because 50% of the nodes are Linux boxes running ADSL connections on crap copper lines, which means that every time the ADSL line drops, we end up with 2 deleted interfaces (one for ppp0 and one for tun0). If you have any thoughts on a preferred solution for interface combining, i'm all ears, and if i have time i would be willing to work on a patch for that.

We'll see how the tweak of increasing time between write-back of dirty buffers goes. It certainly doesn't seem to be hitting the disk hard at the moment.

FWIW, the index on bill_data(bill_id) was a furphy. It didn't make any difference; the lull in the slow query log must have just been between billing runs.

Paul

On 08/09/2013 11:55 AM, Adam Armstrong wrote:

Go with the fastest processor cores you can find to make the web interface feel faster, especially with a lot of ports.

Though, your install doesn't sound all that big...

Adam.

Paul Gear observium@gear.dyndns.org wrote:

Hi Peter,

I'm reasonably confident that the poor performance isn't MySQL-related, but i could be wrong. I have slow query logging turned on, and the only place it seems to be an issue is in the billing code. I managed to improve this by adding indexes to bill_data(timestamp) and bill_data(bill_id) (strangely enough the index already on bill_data(bill_id) as the primary key doesn't seem to be enough - not sure why; but the slow query log stopped logging anything as soon as i added those indexes).

I like their idea in that link about reducing the write-back time of dirty buffers - that makes use of the RAM without any extra management overhead. I'll do a bit of playing around with that on my current system.

Any other hints gratefully accepted.

Regards, Paul

On 08/09/2013 11:08 AM, Peter Childs wrote:

...

I haven't done any testing with it, but this article seems to imply some improvements from kernel adjustments to ensure rrd read/writes are in 'memory' until flushed.

http://code.google.com/p/epicnms/wiki/Scaling

It would be interesting to know which part of your existing system was your bottleneck ? Perhaps moving mySQL to its own node might move some of the load?

From: Paul Gear <observium@gear.dyndns.org mailto:observium@gear.dyndns.org> Reply-To: Observium <observium@observium.org mailto:observium@observium.org> Date: Friday, 9 August 2013 10:27 AM To: Observium <observium@observium.org mailto:observium@observium.org> Subject: [Observium] Performance optimisation (incl. RAM disk vs. SSD)

Hi all,

We've got some money in the budget to upgrade our struggling monitoring server, and i'm trying to optimise Observium. Here are our Observium installation stats:

Total Up Down Ignored Disabled *Devices http://observium.buq.org.au/devices/* 153 http://observium.buq.org.au/devices/ 138 up http://observium.buq.org.au/devices/status=1/ 2 down http://observium.buq.org.au/devices/status=0/ 1 ignored http://observium.buq.org.au/devices/ignore=1/ 12 disabled http://observium.buq.org.au/devices/disabled=1/ *Ports http://observium.buq.org.au/ports/* 8896 http://observium.buq.org.au/ports/ 887 up http://observium.buq.org.au/ports/state=up/ 118 down http://observium.buq.org.au/ports/state=down/ 927 ignored http://observium.buq.org.au/ports/ignore=1/ 6837 shutdown http://observium.buq.org.au/ports/state=admindown/

We're currently running 4 cores, 12 GB RAM, 6 x 15K RPM 3.5" drives in RAID 10. We have about 20 GB of RRDs. Poller-wrapper.py runs with 32 workers and regularly goes over the 500-second mark. The web interface is quite sluggish despite php-xcache being installed.

I'm wondering if anyone has done testing as to which is the better approach for RRD storage performance: RAM disk or SSD. RAM seems likely to offer better IOPS, but managing the RAM disk is obviously more overhead, and i'm concerned that syncing the RAM disk elsewhere will have too much of a performance hit while it happens (especially during reboot cycles). RAM is $25 per GB, whereas SSD is $3 per GB (or $25 per GB if you use "write intensive" SSDs - presumably these are rated for a larger number of lifetime writes?).

My guess at a config for the new box: 12 cores, 64 GB RAM, 4 x 10K RPM 2.5" drives in RAID 10 for OS, 2 x SSD for /opt/observium/rrd. Any other recommendations about which hardware to put our money towards? Do we need to consider other issues like which type of file system to use for /opt/observium/rrd?

Thanks in advance, Paul

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium