Hi Adam, 

2013/2/13 Adam Armstrong <adama@memetic.org>

ZFS is quite possibly the least optimal filesystem for running RRDs from.
You want a simple filesystem without much overhead on operations, like EXT
with journalling and atime turned off.

noatime is the first item on my ToDo. Was expecting to leave it working for a while before touching anything (just to have a baseline)
 

Also, it's possible that RRD is poorly optimised on solaris and uses slow
system calls?

Hmm, how to tell?
 

You say 41 devices in an hour, but your paste suggests 468 devices in 73
minutes.

Well, that line didn't get to the mail. I was just taking one poller process into account. I thought that an execution time of about an hour is not nice while you intent to collect every 5 minutes... :)
 
What is your I/O subsystem? disk? ssd? how many?

Right now the BL460c machine is connected with 2 x 4GB interfaces to our SAN. The LUN provided is presented through a Hitachi USP-VM (hitting it's cache) but using a AMS2500 disk group (about 45  disks if I'm not wrong, shared with other workloads). Also I have available 2 SSD local disks that I'm considering using as ZIL and/or L2ARC (if/when they're needed)
 
A spindled disk
will not behave well with more than 4-8 concurrent processes, an SSD might
behave better.

You should also look at the new poller wrapper script, which might help
you squeeze a bit more out of it, once you've sized it correctly

I'll search  the archives about that...
 

One must ask why, if you have a dedicated machine, are you not just
running it on Ubuntu, like we recommend?

Hmm, well, I'm really a happy SLES user, and as long as I can run an app on it, I'll choose it every time :D. The migration to Solaris was based on the posibility to use ZFS caching to SSD disks on the same Zpool, also, I was looking forward to try its new release.. I would just like to have references from other large installations, maybe I'm hitting a PHP limitation, maybe I need more parallel processes, maybe there's a freaking DNS problem, not sure right now, but I'm sure I shoudn't need more hardware :D


 

adam.


Regards,
CI.-

 
On Wed, 13 Feb 2013 18:37:21 -0300, Ciro Iriarte <cyruspy@gmail.com>
wrote:
> Hi!, I've been running Observium on SLES11 for almost a year. This was
on a
> 8 vCPU + 2GB RAM VMware guest with a 30GB RRD footprint.
>
> After adding about 320 new hosts (all of them, network gear) the RRD
> database went to 130GB and the little machine got peaks of 3k IOPS. The
I/O
> wait was rather hi and the machine would suddently stop responding
> (Hypervisor stats showed 0% CPU usage).
>
> So, I've migrated the installation to a physical machine running Solaris
11
> (I know, it's not supported) and without any tunning (mysql/zfs are
> candidates) I'm seeing polling of 41 devices in about an hour. Is this
> usual/expected?
>
> --
>
> ./poller.php 0/24 February 13, 2013, 18:27 - 27 devices polled in 4048.
> secs
> ./poller.php 7/24 February 13, 2013, 18:27 - 27 devices polled in 4075.
> secs
> ./poller.php 0/24 February 13, 2013, 18:28 - 27 devices polled in 3803.
> secs
> ./poller.php 22/24 February 13, 2013, 18:28 - 27 devices polled in 4404.
> secs
> ./poller.php 3/24 February 13, 2013, 18:28 - 28 devices polled in 4417.
> secs
> --
>
> IOPS climbed to 5k, comments?
>
> Regards,
_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium



--
Ciro Iriarte
http://cyruspy.wordpress.com
--