Hi Markus,

Thank you for your answer.

I played with the config and read carrefully the performance tuning guide on the website and see my conclusion :

If i poll all 844 devices (routers and switches), then i need 28min by default... After tunning to 48 threads, it takes 17min ... I disable polling on every switches (less critical than routers for me), it takes less than 60 sec for 350 routers !!!

I assume that switches need to write data in more rrd files ...

So, i'm now tuning the config and i will ask my boss to provide me 96GB of RAM for implement an RAM drive. I will keep you informed about the result.

Tarik. 11 avril 2018 07:02 "Markus Klock" <markus@best-practice.se (mailto:%22Markus%20Klock%22%20markus@best-practice.se)> a écrit: Yeah you shuld really use SSD-storage when polling 800+ devices.Other than that, how many poller-wrapper threads have you configured? You can probably bump Them up to 48 with that many CPU cores in the server. /Markus Den tis 10 apr. 2018 22:23 <todevnull@free.fr (mailto:todevnull@free.fr)> skrev: with pictures in attached files ... 10 avril 2018 22:18 todevnull@free.fr (mailto:todevnull@free.fr) a écrit:

Hello all,

After a long period of test, i'm putting in production this great software in my company. The server is quite good generaly in term of performance except for the disks ... According to the fact that a polling task shouldn't exceed 5 min, i'm in some trouble ... :-/ A polling session (not the first one), needs 25min to be completed :_(

Except the fact that i should optimize my rrd storage, is there is some particular settings for such installation ?

Thanks in advance for your help.

Tarik. 2x intel xeon X5650 (total of 24 threads), 24GB of RAM, RAID 5 logical drive with 3x15k disks _______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)

Can you please attached a screenshot of the polling-performance page? If you monitor the Observium server(localhost) with Observium then can you please attach a screenshot from the graph: localhost>Graphs>Processor>Extended Processor Utilisation? If the main bottleneck is disk IO you should see a big performance increase if you setup and configure rrdcached on the server.

/Markus

2018-04-11 13:17 GMT+02:00 Klimek, Denis DKlimek@stadtwerke-norderstedt.de :

You can try rrdcache aswell and of course increasing the threads J

Mit freundlichem Gruß

Stadtwerke Norderstedt

*Denis Klimek*

Professional Network Engineer

IP-Systemtechnik

Tel: 040 / 521 04 – 1049

Mobil: 0151 / 652 219 06

dklimek@stadtwerke-norderstedt.de

www.stadtwerke-norderstedt.de

*Von:* observium [mailto:observium-bounces@observium.org] *Im Auftrag von *todevnull@free.fr *Gesendet:* Mittwoch, 11. April 2018 13:14 *An:* Observium *Betreff:* Re: [Observium] Performance issue

Hi Markus,

Thank you for your answer.

I played with the config and read carrefully the performance tuning guide on the website and see my conclusion :

If i poll all 844 devices (routers and switches), then i need 28min by default... After tunning to 48 threads, it takes 17min ... I disable polling on every switches (less critical than routers for me), it takes less than 60 sec for 350 routers !!!

I assume that switches need to write data in more rrd files ...

So, i'm now tuning the config and i will ask my boss to provide me 96GB of RAM for implement an RAM drive. I will keep you informed about the result.

Tarik.

11 avril 2018 07:02 "Markus Klock" <markus@best-practice.se <%22Markus%20Klock%22%20%3cmarkus@best-practice.se%3e>> a écrit:

Yeah you shuld really use SSD-storage when polling 800+ devices.

Other than that, how many poller-wrapper threads have you configured? You can probably bump Them up to 48 with that many CPU cores in the server.

/Markus

Den tis 10 apr. 2018 22:23 todevnull@free.fr skrev:

with pictures in attached files ...

10 avril 2018 22:18 todevnull@free.fr a écrit:

Hello all,

After a long period of test, i'm putting in production this great software in my company. The server is quite good generaly in term of performance except for the disks ... According to the fact that a polling task shouldn't exceed 5 min, i'm in some trouble ... :-/ A polling session (not the first one), needs 25min to be completed :_(

Except the fact that i should optimize my rrd storage, is there is some particular settings for such installation ?

Thanks in advance for your help.

Tarik.

2x intel xeon X5650 (total of 24 threads), 24GB of RAM, RAID 5 logical drive with 3x15k disks

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

Here is a guide for Ubuntu: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html

/Markus

Den ons 11 apr. 2018 16:39Simon Mousey Smith simonsmith5521@gmail.com skrev:

Hi

Is the a setup guide for rrdcached with observium?

I had an old problem with disk io speed which slowed down the polling, so would be nice to see if this fixes my issue

Regards

Simon - Hestor

On 11 Apr 2018, at 12:23, Markus Klock markus@best-practice.se wrote:

Can you please attached a screenshot of the polling-performance page? If you monitor the Observium server(localhost) with Observium then can you please attach a screenshot from the graph: localhost>Graphs>Processor>Extended Processor Utilisation? If the main bottleneck is disk IO you should see a big performance increase if you setup and configure rrdcached on the server.

/Markus

2018-04-11 13:17 GMT+02:00 Klimek, Denis < DKlimek@stadtwerke-norderstedt.de>:

...

You can try rrdcache aswell and of course increasing the threads J

Mit freundlichem Gruß

Stadtwerke Norderstedt

*Denis Klimek*

Professional Network Engineer

IP-Systemtechnik

Tel: 040 / 521 04 – 1049

Mobil: 0151 / 652 219 06

dklimek@stadtwerke-norderstedt.de

www.stadtwerke-norderstedt.de

*Von:* observium [mailto:observium-bounces@observium.org] *Im Auftrag von *todevnull@free.fr *Gesendet:* Mittwoch, 11. April 2018 13:14 *An:* Observium *Betreff:* Re: [Observium] Performance issue

Hi Markus,

Thank you for your answer.

I played with the config and read carrefully the performance tuning guide on the website and see my conclusion :

If i poll all 844 devices (routers and switches), then i need 28min by default... After tunning to 48 threads, it takes 17min ... I disable polling on every switches (less critical than routers for me), it takes less than 60 sec for 350 routers !!!

I assume that switches need to write data in more rrd files ...

So, i'm now tuning the config and i will ask my boss to provide me 96GB of RAM for implement an RAM drive. I will keep you informed about the result.

Tarik.

11 avril 2018 07:02 "Markus Klock" <markus@best-practice.se <%22Markus%20Klock%22%20%3cmarkus@best-practice.se%3e>> a écrit:

Yeah you shuld really use SSD-storage when polling 800+ devices.

Other than that, how many poller-wrapper threads have you configured? You can probably bump Them up to 48 with that many CPU cores in the server.

/Markus

Den tis 10 apr. 2018 22:23 todevnull@free.fr skrev:

with pictures in attached files ...

10 avril 2018 22:18 todevnull@free.fr a écrit:

Hello all,

After a long period of test, i'm putting in production this great software in my company. The server is quite good generaly in term of performance except for the disks ... According to the fact that a polling task shouldn't exceed 5 min, i'm in some trouble ... :-/ A polling session (not the first one), needs 25min to be completed :_(

Except the fact that i should optimize my rrd storage, is there is some particular settings for such installation ?

Thanks in advance for your help.

Tarik.

2x intel xeon X5650 (total of 24 threads), 24GB of RAM, RAID 5 logical drive with 3x15k disks

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

Observium scales primarily on port count rather than device count, especially when it comes to I/O requirements. Our port RRDs are pretty big, so they have the biggest individual contribution to I/O.

96GB is a lot of RAM, you might be better off with an SSD, since dumping all of that data out of RAM disk frequently for backup will be painful (too slow to disk, and pretty punishing to an SSD). :)

adam.

------ Original Message ------ From: todevnull@free.fr To: "Observium" observium@observium.org Sent: 2018-04-11 12:14:01 Subject: Re: [Observium] Performance issue

Hi Markus,

Thank you for your answer.

I played with the config and read carrefully the performance tuning guide on the website and see my conclusion :

If i poll all 844 devices (routers and switches), then i need 28min by default... After tunning to 48 threads, it takes 17min ... I disable polling on every switches (less critical than routers for me), it takes less than 60 sec for 350 routers !!!

I assume that switches need to write data in more rrd files ...

So, i'm now tuning the config and i will ask my boss to provide me 96GB of RAM for implement an RAM drive. I will keep you informed about the result.

Tarik.

11 avril 2018 07:02 "Markus Klock" <markus@best-practice.se <mailto:%22Markus%20Klock%22%20markus@best-practice.se>> a écrit:

...

Yeah you shuld really use SSD-storage when polling 800+ devices. Other than that, how many poller-wrapper threads have you configured? You can probably bump Them up to 48 with that many CPU cores in the server. /Markus Den tis 10 apr. 2018 22:23 todevnull@free.fr skrev:

...

with pictures in attached files ...

10 avril 2018 22:18 todevnull@free.fr a écrit:

...

Hello all,

After a long period of test, i'm putting in production this great software in my company. The server is quite good generaly in term of performance except for the disks ... According to the fact that a polling task shouldn't exceed 5 min, i'm in some trouble ... :-/ A polling session (not the first one), needs 25min to be completed :_(

Except the fact that i should optimize my rrd storage, is there is some particular settings for such installation ?

Thanks in advance for your help.

Tarik.

2x intel xeon X5650 (total of 24 threads), 24GB of RAM, RAID 5 logical drive with 3x15k disks

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

SSDs are fairly resilient these days.

You'd buy a lot of SSD for 96GB of RAM at current prices. A RAID1 of two 512GB enterprise SSDs would be a good solution.

Using tmpfs was really a solution for not being able to scale magnetic disks (and for unreliability of early SSDs). I think it's of less use these days, and a bit of an unnecessary complexity if you can achieve the I/O throughout conventionally.

I personally don't see rrdcached as really worth the effort in most situations. If you can afford the I/O capacity, don't complicate things with rrdcached.

Adam.

⁣Sent from BlueMail ​

On 11 Apr 2018, 20:49, at 20:49, todevnull@free.fr wrote:

The thing is that an SSD won't appreciate such update every 5min :-/ (the lifetime would be impacted) One sync per week will be acceptable for me. (RAM to disk). In the worse case i will lost one week of stats ... but there is 2 power supply in the server which is located in a DC. The risk sounds mitigated...

First i would like to test the benefit of rrdcached. Do we know if there is any 'counterpart'/cons coming with this tool ?

Thanks in advance.

Tarik. 11 avril 2018 20:58 "Adam Armstrong" <adama@memetic.org (mailto:%22Adam%20Armstrong%22%20adama@memetic.org)> a écrit: Observium scales primarily on port count rather than device count, especially when it comes to I/O requirements. Our port RRDs are pretty big, so they have the biggest individual contribution to I/O. 96GB is a lot of RAM, you might be better off with an SSD, since dumping all of that data out of RAM disk frequently for backup will be painful (too slow to disk, and pretty punishing to an SSD). :) adam. ------ Original Message ------ From: todevnull@free.fr (mailto:todevnull@free.fr) To: "Observium" <observium@observium.org (mailto:observium@observium.org)> Sent: 2018-04-11 12:14:01 Subject: Re: [Observium] Performance issue Hi Markus,

Thank you for your answer.

I played with the config and read carrefully the performance tuning guide on the website and see my conclusion :

If i poll all 844 devices (routers and switches), then i need 28min by default... After tunning to 48 threads, it takes 17min ... I disable polling on every switches (less critical than routers for me), it takes less than 60 sec for 350 routers !!!

I assume that switches need to write data in more rrd files ...

So, i'm now tuning the config and i will ask my boss to provide me 96GB of RAM for implement an RAM drive. I will keep you informed about the result.

Tarik. 11 avril 2018 07:02 "Markus Klock" <markus@best-practice.se (mailto:%22Markus%20Klock%22%20markus@best-practice.se)> a écrit:

Yeah you shuld really use SSD-storage when polling 800+ devices.Other than that, how many poller-wrapper threads have you configured? You can probably bump Them up to 48 with that many CPU cores in the server. /Markus Den tis 10 apr. 2018 22:23 <todevnull@free.fr (mailto:todevnull@free.fr)> skrev: with pictures in attached files ... 10 avril 2018 22:18 todevnull@free.fr (mailto:todevnull@free.fr) a écrit:

Hello all,

After a long period of test, i'm putting in production this great software in my company. The server is quite good generaly in term of performance except for the disks ... According to the fact that a polling task shouldn't exceed 5 min, i'm in some trouble ... :-/ A polling session (not the first one), needs 25min to be completed :_(

Except the fact that i should optimize my rrd storage, is there is some particular settings for such installation ?

Thanks in advance for your help.

Tarik. 2x intel xeon X5650 (total of 24 threads), 24GB of RAM, RAID 5 logical drive with 3x15k disks _______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)

---

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

Tarik, I do not understand the issue. In the screenshot you sent us your Observium-install is doing just fine, everything is polled in about 90s witch is great. And you server shows no sign of bad IO-performance. /Markus

2018-04-11 21:59 GMT+02:00 todevnull@free.fr:

Ok. Thank you for your advice Adam.

As i'm not yet able to place any order for hardware upgrade , i will play with rrdcached 'just to see' ... :)

Tarik.

11 avril 2018 21:54 "Adam Armstrong" <adama@memetic.org <%22Adam%20Armstrong%22%20%3Cadama@memetic.org%3E>> a écrit:

SSDs are fairly resilient these days. You'd buy a lot of SSD for 96GB of RAM at current prices. A RAID1 of two 512GB enterprise SSDs would be a good solution. Using tmpfs was really a solution for not being able to scale magnetic disks (and for unreliability of early SSDs). I think it's of less use these days, and a bit of an unnecessary complexity if you can achieve the I/O throughout conventionally. I personally don't see rrdcached as really worth the effort in most situations. If you can afford the I/O capacity, don't complicate things with rrdcached. Adam. Sent from BlueMail http://www.bluemail.me/r?b=12687 On 11 Apr 2018, at 20:49, todevnull@free.fr wrote:

The thing is that an SSD won't appreciate such update every 5min :-/ (the lifetime would be impacted) One sync per week will be acceptable for me. (RAM to disk). In the worse case i will lost one week of stats ... but there is 2 power supply in the server which is located in a DC. The risk sounds mitigated...

First i would like to test the benefit of rrdcached. Do we know if there is any 'counterpart'/cons coming with this tool ?

Thanks in advance.

Tarik.

11 avril 2018 20:58 "Adam Armstrong" < ">adama@memetic.org <%22Adam%20Armstrong%22%20>> a écrit:

Observium scales primarily on port count rather than device count, especially when it comes to I/O requirements. Our port RRDs are pretty big, so they have the biggest individual contribution to I/O. 96GB is a lot of RAM, you might be better off with an SSD, since dumping all of that data out of RAM disk frequently for backup will be painful (too slow to disk, and pretty punishing to an SSD). :) adam. ------ Original Message ------ From: todevnull@free.fr To: "Observium" < observium@observium.org> Sent: 2018-04-11 12:14:01 Subject: Re: [Observium] Performance issue

Hi Markus,

Thank you for your answer.

I played with the config and read carrefully the performance tuning guide on the website and see my conclusion :

If i poll all 844 devices (routers and switches), then i need 28min by default... After tunning to 48 threads, it takes 17min ... I disable polling on every switches (less critical than routers for me), it takes less than 60 sec for 350 routers !!!

I assume that switches need to write data in more rrd files ...

So, i'm now tuning the config and i will ask my boss to provide me 96GB of RAM for implement an RAM drive. I will keep you informed about the result.

Tarik.

11 avril 2018 07:02 "Markus Klock" < ">markus@best-practice.se <%22Markus%20Klock%22%20>> a écrit:

Yeah you shuld really use SSD-storage when polling 800+ devices. Other than that, how many poller-wrapper threads have you configured? You can probably bump Them up to 48 with that many CPU cores in the server. /Markus Den tis 10 apr. 2018 22:23 < todevnull@free.fr> skrev:

with pictures in attached files ...

10 avril 2018 22:18 todevnull@free.fr a écrit:

Hello all,

After a long period of test, i'm putting in production this great software in my company. The server is quite good generaly in term of performance except for the disks ... According to the fact that a polling task shouldn't exceed 5 min, i'm in some trouble ... :-/ A polling session (not the first one), needs 25min to be completed :_(

Except the fact that i should optimize my rrd storage, is there is some particular settings for such installation ?

Thanks in advance for your help.

Tarik.

2x intel xeon X5650 (total of 24 threads), 24GB of RAM, RAID 5 logical drive with 3x15k disks

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

Hi guys!

I have a "yet another proposal" on how to store the RRDs with Observium. My last installation uses a tmpfs-based RAMdisk with ZFS on top of it for RRDs. Mostly it is based on the RAMdisk config example at http://docs.observium.org/persistent_ramdisk/, but the part for ZFS I've added myself. The general idea behind running ZFS on top of the tmpfs is to have compression of RRDs, and my installation has shown about 7x compression ratio when using ZFS with lz4 compression, so for me 8GB tmpfs drive is sufficient for ~3500 ports. I'm attaching a modified /etc/init.d/ramdisk script (looking forward for someone to make it SystemD script) for this setup to work, as well as some instructions.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On April 11, 2018 11:15 PM, todevnull@free.fr wrote:

yes i know that all sounds good now, but if i just enable polling on the disabled devices (showed in the screenshot), arround +60%, the polling take 17min... During this time, is observing the behavior of the server with nmon ... and i can observe all CPUs in wait state and the disk writing again and again :-(

I will try to provide you a screenshot of that.

Tarik.

11 avril 2018 22:10 "Markus Klock" <[markus@best-practice.se](mailto:%22Markus%20Klock%22%20markus@best-practice.se)> a écrit:

...

Tarik, I do not understand the issue. In the screenshot you sent us your Observium-install is doing just fine, everything is polled in about 90s witch is great. And you server shows no sign of bad IO-performance. /Markus 2018-04-11 21:59 GMT+02:00 todevnull@free.fr:

...

Ok. Thank you for your advice Adam.

As i'm not yet able to place any order for hardware upgrade , i will play with rrdcached 'just to see' ... :)

Tarik.

11 avril 2018 21:54 "Adam Armstrong" <[adama@memetic.org](mailto:%22Adam%20Armstrong%22%20%3Cadama@memetic.org%3E)> a écrit:

...

SSDs are fairly resilient these days. You'd buy a lot of SSD for 96GB of RAM at current prices. A RAID1 of two 512GB enterprise SSDs would be a good solution. Using tmpfs was really a solution for not being able to scale magnetic disks (and for unreliability of early SSDs). I think it's of less use these days, and a bit of an unnecessary complexity if you can achieve the I/O throughout conventionally. I personally don't see rrdcached as really worth the effort in most situations. If you can afford the I/O capacity, don't complicate things with rrdcached. Adam. Sent from [BlueMail](http://www.bluemail.me/r?b=12687) On 11 Apr 2018, at 20:49, todevnull@free.fr wrote:

...

The thing is that an SSD won't appreciate such update every 5min :-/ (the lifetime would be impacted) One sync per week will be acceptable for me. (RAM to disk). In the worse case i will lost one week of stats ... but there is 2 power supply in the server which is located in a DC. The risk sounds mitigated...

First i would like to test the benefit of rrdcached. Do we know if there is any 'counterpart'/cons coming with this tool ?

Thanks in advance.

Tarik.

11 avril 2018 20:58 "Adam Armstrong" < [">adama@memetic.org](mailto:%22Adam%20Armstrong%22%20)> a écrit:

...

Observium scales primarily on port count rather than device count, especially when it comes to I/O requirements. Our port RRDs are pretty big, so they have the biggest individual contribution to I/O.

96GB is a lot of RAM, you might be better off with an SSD, since dumping all of that data out of RAM disk frequently for backup will be painful (too slow to disk, and pretty punishing to an SSD). :)

adam.

------ Original Message ------ From: todevnull@free.fr To: "Observium" < observium@observium.org> Sent: 2018-04-11 12:14:01 Subject: Re: [Observium] Performance issue

> Hi Markus, > > Thank you for your answer. > > I played with the config and read carrefully the performance tuning guide on the website and see my conclusion : > > If i poll all 844 devices (routers and switches), then i need 28min by default... > After tunning to 48 threads, it takes 17min ... > I disable polling on every switches (less critical than routers for me), it takes less than 60 sec for 350 routers !!! > > I assume that switches need to write data in more rrd files ... > > So, i'm now tuning the config and i will ask my boss to provide me 96GB of RAM for implement an RAM drive. > I will keep you informed about the result. > > Tarik. > > 11 avril 2018 07:02 "Markus Klock" < [">markus@best-practice.se](mailto:%22Markus%20Klock%22%20)> a écrit: > >> Yeah you shuld really use SSD-storage when polling 800+ devices. >> Other than that, how many poller-wrapper threads have you configured? You can probably bump Them up to 48 with that many CPU cores in the server. >> /Markus >> Den tis 10 apr. 2018 22:23 < todevnull@free.fr> skrev: >> >>> with pictures in attached files ... >>> >>> 10 avril 2018 22:18 todevnull@free.fr a écrit: >>> >>>> Hello all, >>>> >>>> After a long period of test, i'm putting in production this great software in my company. >>>> The server is quite good generaly in term of performance except for the disks ... >>>> According to the fact that a polling task shouldn't exceed 5 min, i'm in some trouble ... :-/ >>>> A polling session (not the first one), needs 25min to be completed :_( >>>> >>>> Except the fact that i should optimize my rrd storage, is there is some particular settings for such installation ? >>>> >>>> Thanks in advance for your help. >>>> >>>> Tarik. >>>> >>>> 2x intel xeon X5650 (total of 24 threads), 24GB of RAM, RAID 5 logical drive with 3x15k disks >>> >>> _______________________________________________ >>> observium mailing list >>> observium@observium.org >>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

---

observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium