What to do when all optimizations fails ...
Hello,
Although I double check every tuning guide from the website, I'm in trouble with my Observium installation. First of all, the size of the network is huge... From the Statistics on about page :
- 1023 Devices
- 57K Ports
- 51K Sensors
- 128 GB RRD
- 20GB DB
I run the latest version of Observium with PHP 7.3 on a Debian 10 physical server. It is a dual Intel Xeon 6248 2.5Ghz (20C/40T) equipped with 3 RAID groups :
- RAID1 for the OS
- RAID1 for the RRD files and Observium application
- RAID5 for the database All the disks are SSD.
The number of poller is set to 48 and the whole polling time has an average of 264s and a max of @334s (I just increase the poller to 60 today...).
I do housekeeping every week to cut off the event log and other stuff older than 30 days. I also use fast userspace caching with php-apcu For the port discovery, some of ports are ignored like vlans, control plan, stack interface, voice-port, etc...
The installation runs like a big diesel machine ... not so fast but very robust.
My problem is that it takes more than 30 seconds to get the detail of simple 48 ports switch ... and much more when I try to load a detailed backbone with many more ports.
Do you have any suggestion to improve the GUI responsiveness ?
Thank you for your time Kind regards Vincent
Hi Graf,
Can you turn on sql profiling (it’ll be in the options, probably under ui or debugging) and show the slowest queries?
The bottom right of the screen, there should be a database icon that’ll show this.
Beyond this MySQL/web benefits from being on a system with faster cores. MySQL and web generation is single threaded, so finding a system with the fastest individual cores for the web ui and database (and RRDs) can be a big improvement.
2.5ghz is snail speed :)
Adam.
Sent from my iPhone
On 31 Mar 2022, at 07:22, Graf Vincent via observium observium@observium.org wrote:
Hello,
Although I double check every tuning guide from the website, I’m in trouble with my Observium installation. First of all, the size of the network is huge… From the Statistics on about page :
1023 Devices57K Ports51K Sensors128 GB RRD20GB DBI run the latest version of Observium with PHP 7.3 on a Debian 10 physical server. It is a dual Intel Xeon 6248 2.5Ghz (20C/40T) equipped with 3 RAID groups :
RAID1 for the OSRAID1 for the RRD files and Observium applicationRAID5 for the databaseAll the disks are SSD.
The number of poller is set to 48 and the whole polling time has an average of 264s and a max of @334s (I just increase the poller to 60 today…).
I do housekeeping every week to cut off the event log and other stuff older than 30 days. I also use fast userspace caching with php-apcu For the port discovery, some of ports are ignored like vlans, control plan, stack interface, voice-port, etc…
The installation runs like a big diesel machine … not so fast but very robust.
My problem is that it takes more than 30 seconds to get the detail of simple 48 ports switch … and much more when I try to load a detailed backbone with many more ports.
Do you have any suggestion to improve the GUI responsiveness ?
Thank you for your time Kind regards Vincent
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Adam,
Thank you for your response.
I turn on the sql profiling and saw that the longest query is “SELECT EXISTS (SELECT 1 FROM `accesspoints` WHERE `device_id` = 'xxxxxx')”
I check tree different type of devices with this results: SELECT EXISTS (SELECT 1 FROM `accesspoints` WHERE `device_id` = '79') 19.62462497
SELECT EXISTS (SELECT 1 FROM `accesspoints` WHERE `device_id` = '170') 14.97394896
SELECT EXISTS (SELECT 1 FROM `accesspoints` WHERE `device_id` = '156') 21.80661798
Kind regards Vincent
De : observium observium-bounces@observium.org De la part de Adam Armstrong via observium Envoyé : jeudi 31 mars 2022 12:07 À : Observium observium@observium.org Cc : Adam Armstrong adama@observium.org Objet : Re: [Observium] What to do when all optimizations fails ...
Hi Graf,
Can you turn on sql profiling (it’ll be in the options, probably under ui or debugging) and show the slowest queries?
The bottom right of the screen, there should be a database icon that’ll show this.
Beyond this MySQL/web benefits from being on a system with faster cores. MySQL and web generation is single threaded, so finding a system with the fastest individual cores for the web ui and database (and RRDs) can be a big improvement.
2.5ghz is snail speed :)
Adam. Sent from my iPhone
On 31 Mar 2022, at 07:22, Graf Vincent via observium <observium@observium.orgmailto:observium@observium.org> wrote: Hello,
Although I double check every tuning guide from the website, I’m in trouble with my Observium installation. First of all, the size of the network is huge… From the Statistics on about page :
- 1023 Devices
- 57K Ports
- 51K Sensors
- 128 GB RRD
- 20GB DB
I run the latest version of Observium with PHP 7.3 on a Debian 10 physical server. It is a dual Intel Xeon 6248 2.5Ghz (20C/40T) equipped with 3 RAID groups :
- RAID1 for the OS
- RAID1 for the RRD files and Observium application
- RAID5 for the database All the disks are SSD.
The number of poller is set to 48 and the whole polling time has an average of 264s and a max of @334s (I just increase the poller to 60 today…).
I do housekeeping every week to cut off the event log and other stuff older than 30 days. I also use fast userspace caching with php-apcu For the port discovery, some of ports are ignored like vlans, control plan, stack interface, voice-port, etc…
The installation runs like a big diesel machine … not so fast but very robust.
My problem is that it takes more than 30 seconds to get the detail of simple 48 ports switch … and much more when I try to load a detailed backbone with many more ports.
Do you have any suggestion to improve the GUI responsiveness ?
Thank you for your time Kind regards Vincent
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Vincent,
This was a bug that was fixed a couple of weeks back. If you update and run ./discover.php -u, it should resolve :)
Thanks,
Adam.
From: observium observium-bounces@observium.org On Behalf Of Graf Vincent via observium Sent: 31 March 2022 15:44 To: Observium observium@observium.org Cc: Graf Vincent Vincent.Graf@chuv.ch Subject: Re: [Observium] What to do when all optimizations fails ...
Hi Adam,
Thank you for your response.
I turn on the sql profiling and saw that the longest query is “SELECT EXISTS (SELECT 1 FROM `accesspoints` WHERE `device_id` = 'xxxxxx')”
I check tree different type of devices with this results:
SELECT EXISTS (SELECT 1 FROM `accesspoints` WHERE `device_id` = '79')
19.62462497
SELECT EXISTS (SELECT 1 FROM `accesspoints` WHERE `device_id` = '170')
14.97394896
SELECT EXISTS (SELECT 1 FROM `accesspoints` WHERE `device_id` = '156')
21.80661798
Kind regards
Vincent
De : observium <observium-bounces@observium.org mailto:observium-bounces@observium.org > De la part de Adam Armstrong via observium Envoyé : jeudi 31 mars 2022 12:07 À : Observium <observium@observium.org mailto:observium@observium.org > Cc : Adam Armstrong <adama@observium.org mailto:adama@observium.org > Objet : Re: [Observium] What to do when all optimizations fails ...
Hi Graf,
Can you turn on sql profiling (it’ll be in the options, probably under ui or debugging) and show the slowest queries?
The bottom right of the screen, there should be a database icon that’ll show this.
Beyond this MySQL/web benefits from being on a system with faster cores. MySQL and web generation is single threaded, so finding a system with the fastest individual cores for the web ui and database (and RRDs) can be a big improvement.
2.5ghz is snail speed :)
Adam.
Sent from my iPhone
On 31 Mar 2022, at 07:22, Graf Vincent via observium <observium@observium.org mailto:observium@observium.org > wrote:
Hello,
Although I double check every tuning guide from the website, I’m in trouble with my Observium installation.
First of all, the size of the network is huge… From the Statistics on about page :
* 1023 Devices * 57K Ports * 51K Sensors * 128 GB RRD * 20GB DB
I run the latest version of Observium with PHP 7.3 on a Debian 10 physical server. It is a dual Intel Xeon 6248 2.5Ghz (20C/40T) equipped with 3 RAID groups :
* RAID1 for the OS * RAID1 for the RRD files and Observium application * RAID5 for the database
All the disks are SSD.
The number of poller is set to 48 and the whole polling time has an average of 264s and a max of @334s (I just increase the poller to 60 today…).
I do housekeeping every week to cut off the event log and other stuff older than 30 days. I also use fast userspace caching with php-apcu
For the port discovery, some of ports are ignored like vlans, control plan, stack interface, voice-port, etc…
The installation runs like a big diesel machine … not so fast but very robust.
My problem is that it takes more than 30 seconds to get the detail of simple 48 ports switch … and much more when I try to load a detailed backbone with many more ports.
Do you have any suggestion to improve the GUI responsiveness ?
Thank you for your time
Kind regards
Vincent
_______________________________________________ observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Adam,
Nice ☺ it works very well now !
Kind regards Vincent
De : observium observium-bounces@observium.org De la part de Adam Armstrong via observium Envoyé : jeudi 31 mars 2022 16:50 À : 'Observium' observium@observium.org Cc : Adam Armstrong adama@observium.org Objet : Re: [Observium] What to do when all optimizations fails ...
Hi Vincent,
This was a bug that was fixed a couple of weeks back. If you update and run ./discover.php -u, it should resolve ☺
Thanks, Adam.
From: observium <observium-bounces@observium.orgmailto:observium-bounces@observium.org> On Behalf Of Graf Vincent via observium Sent: 31 March 2022 15:44 To: Observium <observium@observium.orgmailto:observium@observium.org> Cc: Graf Vincent <Vincent.Graf@chuv.chmailto:Vincent.Graf@chuv.ch> Subject: Re: [Observium] What to do when all optimizations fails ...
Hi Adam,
Thank you for your response.
I turn on the sql profiling and saw that the longest query is “SELECT EXISTS (SELECT 1 FROM `accesspoints` WHERE `device_id` = 'xxxxxx')”
I check tree different type of devices with this results: SELECT EXISTS (SELECT 1 FROM `accesspoints` WHERE `device_id` = '79') 19.62462497
SELECT EXISTS (SELECT 1 FROM `accesspoints` WHERE `device_id` = '170') 14.97394896
SELECT EXISTS (SELECT 1 FROM `accesspoints` WHERE `device_id` = '156') 21.80661798
Kind regards Vincent
De : observium <observium-bounces@observium.orgmailto:observium-bounces@observium.org> De la part de Adam Armstrong via observium Envoyé : jeudi 31 mars 2022 12:07 À : Observium <observium@observium.orgmailto:observium@observium.org> Cc : Adam Armstrong <adama@observium.orgmailto:adama@observium.org> Objet : Re: [Observium] What to do when all optimizations fails ...
Hi Graf,
Can you turn on sql profiling (it’ll be in the options, probably under ui or debugging) and show the slowest queries?
The bottom right of the screen, there should be a database icon that’ll show this.
Beyond this MySQL/web benefits from being on a system with faster cores. MySQL and web generation is single threaded, so finding a system with the fastest individual cores for the web ui and database (and RRDs) can be a big improvement.
2.5ghz is snail speed :)
Adam. Sent from my iPhone
On 31 Mar 2022, at 07:22, Graf Vincent via observium <observium@observium.orgmailto:observium@observium.org> wrote: Hello,
Although I double check every tuning guide from the website, I’m in trouble with my Observium installation. First of all, the size of the network is huge… From the Statistics on about page : - 1023 Devices - 57K Ports - 51K Sensors - 128 GB RRD - 20GB DB
I run the latest version of Observium with PHP 7.3 on a Debian 10 physical server. It is a dual Intel Xeon 6248 2.5Ghz (20C/40T) equipped with 3 RAID groups : - RAID1 for the OS - RAID1 for the RRD files and Observium application - RAID5 for the database All the disks are SSD.
The number of poller is set to 48 and the whole polling time has an average of 264s and a max of @334s (I just increase the poller to 60 today…).
I do housekeeping every week to cut off the event log and other stuff older than 30 days. I also use fast userspace caching with php-apcu For the port discovery, some of ports are ignored like vlans, control plan, stack interface, voice-port, etc…
The installation runs like a big diesel machine … not so fast but very robust.
My problem is that it takes more than 30 seconds to get the detail of simple 48 ports switch … and much more when I try to load a detailed backbone with many more ports.
Do you have any suggestion to improve the GUI responsiveness ?
Thank you for your time Kind regards Vincent
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (2)
-
Adam Armstrong -
Graf Vincent