Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ? Best regards, Louis
Hi Louis,
We had a similar issue where observium was falling over due to the amount of hosts we are monitoring and so it was alarming on everything all the time as the polls werent dealt with quick enough.
Our DBA moved the mySQL and RRD files to different volumes to spread the load on the drives. He also sorted an indexing issue on the mySQL side.
How many spindles do you have servicing the array? And what caching policy are you using on the RAID controller?
Alastair
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Louis Bailleul Sent: 09 January 2015 14:46 To: observium@observium.org Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
What indexes did he create?
adam.
------ Original Message ------ From: "Alastair Chamorro" Alastair.Chamorro@voip-unlimited.net To: "Observium Network Observation System" observium@observium.org Sent: 1/9/2015 9:00:09 AM Subject: Re: [Observium] Speed up graph rendering
Hi Louis,
We had a similar issue where observium was falling over due to the amount of hosts we are monitoring and so it was alarming on everything all the time as the polls werent dealt with quick enough.
Our DBA moved the mySQL and RRD files to different volumes to spread the load on the drives. He also sorted an indexing issue on the mySQL side.
How many spindles do you have servicing the array? And what caching policy are you using on the RAID controller?
Alastair
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Louis Bailleul Sent: 09 January 2015 14:46 To:observium@observium.org Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
alter table devices_attribs add key device_id(device_id); alter table alert_table add key dev_id(device_id);
I believe these are the two new indexes he added.
Kind Regards
Alastair From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: 09 January 2015 17:00 To: Observium Network Observation System Subject: Re: [Observium] Speed up graph rendering
What indexes did he create?
adam.
------ Original Message ------ From: "Alastair Chamorro" <Alastair.Chamorro@voip-unlimited.netmailto:Alastair.Chamorro@voip-unlimited.net> To: "Observium Network Observation System" <observium@observium.orgmailto:observium@observium.org> Sent: 1/9/2015 9:00:09 AM Subject: Re: [Observium] Speed up graph rendering
Hi Louis,
We had a similar issue where observium was falling over due to the amount of hosts we are monitoring and so it was alarming on everything all the time as the polls werent dealt with quick enough.
Our DBA moved the mySQL and RRD files to different volumes to spread the load on the drives. He also sorted an indexing issue on the mySQL side.
How many spindles do you have servicing the array? And what caching policy are you using on the RAID controller?
Alastair
From: observium [mailto:observium-bounces@observium.orgmailto:observium-bounces@observium.org] On Behalf Of Louis Bailleul Sent: 09 January 2015 14:46 To: observium@observium.orgmailto:observium@observium.org Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
I did:
ALTER TABLE ADD KEY `idx0_devices_attribs` (`device_id`,`attrib_type`(50)) USING BTREE;
2015-01-09 15:03 GMT-02:00 Alastair Chamorro < Alastair.Chamorro@voip-unlimited.net>:
alter table devices_attribs add key device_id(device_id); alter table alert_table add key dev_id(device_id);
I believe these are the two new indexes he added.
Kind Regards
Alastair
*From:* observium [mailto:observium-bounces@observium.org] *On Behalf Of *Adam Armstrong *Sent:* 09 January 2015 17:00 *To:* Observium Network Observation System *Subject:* Re: [Observium] Speed up graph rendering
What indexes did he create?
adam.
------ Original Message ------
From: "Alastair Chamorro" <*Alastair.Chamorro@voip-unlimited.net* Alastair.Chamorro@voip-unlimited.net>
To: "Observium Network Observation System" <*observium@observium.org* observium@observium.org>
Sent: 1/9/2015 9:00:09 AM
Subject: Re: [Observium] Speed up graph rendering
Hi Louis,
We had a similar issue where observium was falling over due to the amount of hosts we are monitoring and so it was alarming on everything all the time as the polls werent dealt with quick enough.
Our DBA moved the mySQL and RRD files to different volumes to spread the load on the drives. He also sorted an indexing issue on the mySQL side.
How many spindles do you have servicing the array? And what caching policy are you using on the RAID controller?
Alastair
*From:* observium [mailto:*observium-bounces@observium.org* observium-bounces@observium.org] *On Behalf Of *Louis Bailleul *Sent:* 09 January 2015 14:46 *To:* *observium@observium.org* observium@observium.org *Subject:* [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Sorry, my previous sql is incorrect.
ALTER TABLE devices_attribs ADD KEY `idx0_devices_attribs` (`device_id`,`attrib_type`(50)) USING BTREE;
More indexes: ALTER TABLE applications ADD KEY `idx0_applications` (`app_type`(32)) USING BTREE; ALTER TABLE bgpPeers ADD KEY `idx0_bgpPeers` (`device_id`,`bgpPeerAdminStatus`(15),`bgpPeerState`(15)) USING BTREE; ALTER TABLE ip_mac ADD KEY `idx0_ip_mac` (`ip_address`(15)) USING BTREE; ALTER TABLE ipv4_addresses ADD KEY `idx0_ipv4_addresses` (`port_id`,`ipv4_network_id`(10)) USING BTREE; ALTER TABLE ipv4_addresses ADD KEY KEY `idx1_ipv4_addresses` (`ipv4_address`(15)) USING BTREE; ALTER TABLE ipv6_addresses ADD KEY `idx0_ipv6_addresses` (`ipv6_address`(64)) USING BTREE; ALTER TABLE ospf_instances ADD KEY `idx0_ospf_instances` (`ospfAdminStat`) USING BTREE; ALTER TABLE ports ADD KEY `idx0_ports` (`ifOperStatus`(4),`ifAdminStatus`(5),`ignore`,`deleted`,`ifLastChange`) USING BTREE; ALTER TABLE ports ADD KEY `idx1_ports` (`device_id`,`port_id`) USING BTREE; ALTER TABLE ports ADD KEY `idx2_ports` (`device_id`,`deleted`,`port_descr_type`(100),`ifAlias`(100)) USING BTREE; ALTER TABLE ports-state ADD KEY `idx0_ports-state` (`port_id`,`ifInErrors_delta`,`ifOutErrors_delta`) USING HASH; ALTER TABLE pseudowires ADD KEY `idx0_pseudowires` (`port_id`) USING BTREE; ALTER TABLE syslog ADD KEY `idx0_syslog` (`timestamp`,`device_id`) USING BTREE; ALTER TABLE syslog ADD KEY `idx1_syslog` (`device_id`,`program`) USING BTREE; ALTER TABLE syslog ADD KEY `idx2_syslog` (`device_id`,`seq`) USING BTREE; ALTER TABLE users_ckeys ADD KEY `idx0_users_ckeys` (`user_uniq`,`user_ckey`) USING BTREE;
For memory tables, HASH is better.
All indexes was created monitoring slow queries. I run this queries with EXPLAIN to view execution plan.
Hope that helps.
2015-01-09 16:26 GMT-02:00 Eduardo Schoedler listas@esds.com.br:
I did:
ALTER TABLE ADD KEY `idx0_devices_attribs` (`device_id`,`attrib_type`(50)) USING BTREE;
2015-01-09 15:03 GMT-02:00 Alastair Chamorro < Alastair.Chamorro@voip-unlimited.net>:
alter table devices_attribs add key device_id(device_id); alter table alert_table add key dev_id(device_id);
I believe these are the two new indexes he added.
Kind Regards
Alastair
*From:* observium [mailto:observium-bounces@observium.org] *On Behalf Of *Adam Armstrong *Sent:* 09 January 2015 17:00 *To:* Observium Network Observation System *Subject:* Re: [Observium] Speed up graph rendering
What indexes did he create?
adam.
------ Original Message ------
From: "Alastair Chamorro" <*Alastair.Chamorro@voip-unlimited.net* Alastair.Chamorro@voip-unlimited.net>
To: "Observium Network Observation System" <*observium@observium.org* observium@observium.org>
Sent: 1/9/2015 9:00:09 AM
Subject: Re: [Observium] Speed up graph rendering
Hi Louis,
We had a similar issue where observium was falling over due to the amount of hosts we are monitoring and so it was alarming on everything all the time as the polls werent dealt with quick enough.
Our DBA moved the mySQL and RRD files to different volumes to spread the load on the drives. He also sorted an indexing issue on the mySQL side.
How many spindles do you have servicing the array? And what caching policy are you using on the RAID controller?
Alastair
*From:* observium [mailto:*observium-bounces@observium.org* observium-bounces@observium.org] *On Behalf Of *Louis Bailleul *Sent:* 09 January 2015 14:46 *To:* *observium@observium.org* observium@observium.org *Subject:* [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Eduardo Schoedler
Can we, if there are no issues with those indexes push them into the Observium repository to make them “official” ? If no, is there a risk of breaking futures update if we push those indexes ? (about the sql files ran when an update is run)
De : observium [mailto:observium-bounces@observium.org] De la part de Eduardo Schoedler Envoyé : vendredi 9 janvier 2015 19:43 À : Observium Network Observation System Objet : Re: [Observium] Speed up graph rendering
Sorry, my previous sql is incorrect.
ALTER TABLE devices_attribs ADD KEY `idx0_devices_attribs` (`device_id`,`attrib_type`(50)) USING BTREE;
More indexes: ALTER TABLE applications ADD KEY `idx0_applications` (`app_type`(32)) USING BTREE; ALTER TABLE bgpPeers ADD KEY `idx0_bgpPeers` (`device_id`,`bgpPeerAdminStatus`(15),`bgpPeerState`(15)) USING BTREE; ALTER TABLE ip_mac ADD KEY `idx0_ip_mac` (`ip_address`(15)) USING BTREE; ALTER TABLE ipv4_addresses ADD KEY `idx0_ipv4_addresses` (`port_id`,`ipv4_network_id`(10)) USING BTREE; ALTER TABLE ipv4_addresses ADD KEY KEY `idx1_ipv4_addresses` (`ipv4_address`(15)) USING BTREE; ALTER TABLE ipv6_addresses ADD KEY `idx0_ipv6_addresses` (`ipv6_address`(64)) USING BTREE; ALTER TABLE ospf_instances ADD KEY `idx0_ospf_instances` (`ospfAdminStat`) USING BTREE; ALTER TABLE ports ADD KEY `idx0_ports` (`ifOperStatus`(4),`ifAdminStatus`(5),`ignore`,`deleted`,`ifLastChange`) USING BTREE; ALTER TABLE ports ADD KEY `idx1_ports` (`device_id`,`port_id`) USING BTREE; ALTER TABLE ports ADD KEY `idx2_ports` (`device_id`,`deleted`,`port_descr_type`(100),`ifAlias`(100)) USING BTREE; ALTER TABLE ports-state ADD KEY `idx0_ports-state` (`port_id`,`ifInErrors_delta`,`ifOutErrors_delta`) USING HASH; ALTER TABLE pseudowires ADD KEY `idx0_pseudowires` (`port_id`) USING BTREE; ALTER TABLE syslog ADD KEY `idx0_syslog` (`timestamp`,`device_id`) USING BTREE; ALTER TABLE syslog ADD KEY `idx1_syslog` (`device_id`,`program`) USING BTREE; ALTER TABLE syslog ADD KEY `idx2_syslog` (`device_id`,`seq`) USING BTREE; ALTER TABLE users_ckeys ADD KEY `idx0_users_ckeys` (`user_uniq`,`user_ckey`) USING BTREE;
For memory tables, HASH is better.
All indexes was created monitoring slow queries. I run this queries with EXPLAIN to view execution plan.
Hope that helps.
2015-01-09 16:26 GMT-02:00 Eduardo Schoedler <listas@esds.com.brmailto:listas@esds.com.br>: I did: ALTER TABLE ADD KEY `idx0_devices_attribs` (`device_id`,`attrib_type`(50)) USING BTREE;
2015-01-09 15:03 GMT-02:00 Alastair Chamorro <Alastair.Chamorro@voip-unlimited.netmailto:Alastair.Chamorro@voip-unlimited.net>: alter table devices_attribs add key device_id(device_id); alter table alert_table add key dev_id(device_id);
I believe these are the two new indexes he added.
Kind Regards
Alastair From: observium [mailto:observium-bounces@observium.orgmailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: 09 January 2015 17:00 To: Observium Network Observation System Subject: Re: [Observium] Speed up graph rendering
What indexes did he create?
adam.
------ Original Message ------ From: "Alastair Chamorro" <Alastair.Chamorro@voip-unlimited.netmailto:Alastair.Chamorro@voip-unlimited.net> To: "Observium Network Observation System" <observium@observium.orgmailto:observium@observium.org> Sent: 1/9/2015 9:00:09 AM Subject: Re: [Observium] Speed up graph rendering
Hi Louis,
We had a similar issue where observium was falling over due to the amount of hosts we are monitoring and so it was alarming on everything all the time as the polls werent dealt with quick enough.
Our DBA moved the mySQL and RRD files to different volumes to spread the load on the drives. He also sorted an indexing issue on the mySQL side.
How many spindles do you have servicing the array? And what caching policy are you using on the RAID controller?
Alastair
From: observium [mailto:observium-bounces@observium.orgmailto:observium-bounces@observium.org] On Behalf Of Louis Bailleul Sent: 09 January 2015 14:46 To: observium@observium.orgmailto:observium@observium.org Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Eduardo Schoedler
-- Eduardo Schoedler
I'll make them official shortly :)
adam.
On 2015-01-12 12:15, Mathieu POUSSIN wrote:
Can we, if there are no issues with those indexes push them into the Observium repository to make them “official” ?
If no, is there a risk of breaking futures update if we push those indexes ? (about the sql files ran when an update is run)
DE : observium [mailto:observium-bounces@observium.org] DE LA PART DE Eduardo Schoedler ENVOYÉ : vendredi 9 janvier 2015 19:43 À : Observium Network Observation System OBJET : Re: [Observium] Speed up graph rendering
Sorry, my previous sql is incorrect.
ALTER TABLE devices_attribs ADD KEY `idx0_devices_attribs` (`device_id`,`attrib_type`(50)) USING BTREE;
More indexes:
ALTER TABLE applications ADD KEY `idx0_applications` (`app_type`(32)) USING BTREE;
ALTER TABLE bgpPeers ADD KEY `idx0_bgpPeers` (`device_id`,`bgpPeerAdminStatus`(15),`bgpPeerState`(15)) USING BTREE;
ALTER TABLE ip_mac ADD KEY `idx0_ip_mac` (`ip_address`(15)) USING BTREE;
ALTER TABLE ipv4_addresses ADD KEY `idx0_ipv4_addresses` (`port_id`,`ipv4_network_id`(10)) USING BTREE;
ALTER TABLE ipv4_addresses ADD KEY KEY `idx1_ipv4_addresses` (`ipv4_address`(15)) USING BTREE;
ALTER TABLE ipv6_addresses ADD KEY `idx0_ipv6_addresses` (`ipv6_address`(64)) USING BTREE;
ALTER TABLE ospf_instances ADD KEY `idx0_ospf_instances` (`ospfAdminStat`) USING BTREE;
ALTER TABLE ports ADD KEY `idx0_ports` (`ifOperStatus`(4),`ifAdminStatus`(5),`ignore`,`deleted`,`ifLastChange`) USING BTREE;
ALTER TABLE ports ADD KEY `idx1_ports` (`device_id`,`port_id`) USING BTREE;
ALTER TABLE ports ADD KEY `idx2_ports` (`device_id`,`deleted`,`port_descr_type`(100),`ifAlias`(100)) USING BTREE;
ALTER TABLE ports-state ADD KEY `idx0_ports-state` (`port_id`,`ifInErrors_delta`,`ifOutErrors_delta`) USING HASH;
ALTER TABLE pseudowires ADD KEY `idx0_pseudowires` (`port_id`) USING BTREE;
ALTER TABLE syslog ADD KEY `idx0_syslog` (`timestamp`,`device_id`) USING BTREE;
ALTER TABLE syslog ADD KEY `idx1_syslog` (`device_id`,`program`) USING BTREE;
ALTER TABLE syslog ADD KEY `idx2_syslog` (`device_id`,`seq`) USING BTREE;
ALTER TABLE users_ckeys ADD KEY `idx0_users_ckeys` (`user_uniq`,`user_ckey`) USING BTREE;
For memory tables, HASH is better.
All indexes was created monitoring slow queries.
I run this queries with EXPLAIN to view execution plan.
Hope that helps.
2015-01-09 16:26 GMT-02:00 Eduardo Schoedler listas@esds.com.br:
I did:
ALTER TABLE ADD KEY `idx0_devices_attribs` (`device_id`,`attrib_type`(50)) USING BTREE;
2015-01-09 15:03 GMT-02:00 Alastair Chamorro Alastair.Chamorro@voip-unlimited.net:
alter table devices_attribs add key device_id(device_id); alter table alert_table add key dev_id(device_id);
I believe these are the two new indexes he added.
Kind Regards
Alastair
FROM: observium [mailto:observium-bounces@observium.org] ON BEHALF OF Adam Armstrong SENT: 09 January 2015 17:00 TO: Observium Network Observation System
SUBJECT: Re: [Observium] Speed up graph rendering
What indexes did he create?
adam.
------ Original Message ------
From: "Alastair Chamorro" Alastair.Chamorro@voip-unlimited.net
To: "Observium Network Observation System" observium@observium.org
Sent: 1/9/2015 9:00:09 AM
Subject: Re: [Observium] Speed up graph rendering
Hi Louis,
We had a similar issue where observium was falling over due to the amount of hosts we are monitoring and so it was alarming on everything all the time as the polls werent dealt with quick enough.
Our DBA moved the mySQL and RRD files to different volumes to spread the load on the drives. He also sorted an indexing issue on the mySQL side.
How many spindles do you have servicing the array? And what caching policy are you using on the RAID controller?
Alastair
FROM: observium [mailto:observium-bounces@observium.org] ON BEHALF OF Louis Bailleul SENT: 09 January 2015 14:46 TO: observium@observium.org SUBJECT: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [1]
--
Eduardo Schoedler
--
Eduardo Schoedler
Links:
[1] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Some indexes already added in r6167. All other not really required, because not speedup any queries.
Mysql slow log just say this queries not use index, but queries already "fast" (< 0.00001).
If someone have really slow queries in log, please send examples from slow-log (with execute times).
On Tue, Jan 13, 2015 at 9:51 PM, Adam Armstrong adama@memetic.org wrote:
I'll make them official shortly :)
adam.
On 2015-01-12 12:15, Mathieu POUSSIN wrote:
Can we, if there are no issues with those indexes push them into the Observium repository to make them “official” ?
If no, is there a risk of breaking futures update if we push those indexes ? (about the sql files ran when an update is run)
DE : observium [mailto:observium-bounces@observium.org] DE LA PART DE Eduardo Schoedler ENVOYÉ : vendredi 9 janvier 2015 19:43 À : Observium Network Observation System OBJET : Re: [Observium] Speed up graph rendering
Sorry, my previous sql is incorrect.
ALTER TABLE devices_attribs ADD KEY `idx0_devices_attribs` (`device_id`,`attrib_type`(50)) USING BTREE;
More indexes:
ALTER TABLE applications ADD KEY `idx0_applications` (`app_type`(32)) USING BTREE;
ALTER TABLE bgpPeers ADD KEY `idx0_bgpPeers` (`device_id`,`bgpPeerAdminStatus`(15),`bgpPeerState`(15)) USING BTREE;
ALTER TABLE ip_mac ADD KEY `idx0_ip_mac` (`ip_address`(15)) USING BTREE;
ALTER TABLE ipv4_addresses ADD KEY `idx0_ipv4_addresses` (`port_id`,`ipv4_network_id`(10)) USING BTREE;
ALTER TABLE ipv4_addresses ADD KEY KEY `idx1_ipv4_addresses` (`ipv4_address`(15)) USING BTREE;
ALTER TABLE ipv6_addresses ADD KEY `idx0_ipv6_addresses` (`ipv6_address`(64)) USING BTREE;
ALTER TABLE ospf_instances ADD KEY `idx0_ospf_instances` (`ospfAdminStat`) USING BTREE;
ALTER TABLE ports ADD KEY `idx0_ports` (`ifOperStatus`(4),`ifAdminStatus`(5),`ignore`,`deleted`,`ifLastChange`) USING BTREE;
ALTER TABLE ports ADD KEY `idx1_ports` (`device_id`,`port_id`) USING BTREE;
ALTER TABLE ports ADD KEY `idx2_ports` (`device_id`,`deleted`,`port_descr_type`(100),`ifAlias`(100)) USING BTREE;
ALTER TABLE ports-state ADD KEY `idx0_ports-state` (`port_id`,`ifInErrors_delta`,`ifOutErrors_delta`) USING HASH;
ALTER TABLE pseudowires ADD KEY `idx0_pseudowires` (`port_id`) USING BTREE;
ALTER TABLE syslog ADD KEY `idx0_syslog` (`timestamp`,`device_id`) USING BTREE;
ALTER TABLE syslog ADD KEY `idx1_syslog` (`device_id`,`program`) USING BTREE;
ALTER TABLE syslog ADD KEY `idx2_syslog` (`device_id`,`seq`) USING BTREE;
ALTER TABLE users_ckeys ADD KEY `idx0_users_ckeys` (`user_uniq`,`user_ckey`) USING BTREE;
For memory tables, HASH is better.
All indexes was created monitoring slow queries.
I run this queries with EXPLAIN to view execution plan.
Hope that helps.
2015-01-09 16:26 GMT-02:00 Eduardo Schoedler listas@esds.com.br:
I did:
ALTER TABLE ADD KEY `idx0_devices_attribs` (`device_id`,`attrib_type`(50)) USING BTREE;
2015-01-09 15:03 GMT-02:00 Alastair Chamorro <Alastair.Chamorro@voip- unlimited.net>:
alter table devices_attribs add key device_id(device_id); alter table alert_table add key dev_id(device_id);
I believe these are the two new indexes he added.
Kind Regards
Alastair
FROM: observium [mailto:observium-bounces@observium.org] ON BEHALF OF Adam Armstrong SENT: 09 January 2015 17:00 TO: Observium Network Observation System
SUBJECT: Re: [Observium] Speed up graph rendering
What indexes did he create?
adam.
------ Original Message ------
From: "Alastair Chamorro" Alastair.Chamorro@voip-unlimited.net
To: "Observium Network Observation System" observium@observium.org
Sent: 1/9/2015 9:00:09 AM
Subject: Re: [Observium] Speed up graph rendering
Hi Louis,
We had a similar issue where observium was falling over due to the amount of hosts we are monitoring and so it was alarming on everything all the time as the polls werent dealt with quick enough.
Our DBA moved the mySQL and RRD files to different volumes to spread the load on the drives. He also sorted an indexing issue on the mySQL side.
How many spindles do you have servicing the array? And what caching policy are you using on the RAID controller?
Alastair
FROM: observium [mailto:observium-bounces@observium.org] ON BEHALF OF Louis Bailleul SENT: 09 January 2015 14:46 TO: observium@observium.org SUBJECT: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [1]
--
Eduardo Schoedler
--
Eduardo Schoedler
Links:
[1] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
It might help if you included a couple of references/examples of the pages that are slow.
To give you something to compare against, we're running Observium on an ESX VM - 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage - but we're at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports.
I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds.
Now...I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has...a metric shit-tonne of ports and graphs on it - hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don't know if that's simply the browser catching up, or if it's still generating the actual graphs in the background.
From: Louis Bailleul <louis.bailleul@phangos.frmailto:louis.bailleul@phangos.fr> Reply-To: "observium@observium.orgmailto:observium@observium.org" <observium@observium.orgmailto:observium@observium.org> Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.orgmailto:observium@observium.org" <observium@observium.orgmailto:observium@observium.org> Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
Founded in 2007, IO provides the data center as a service to businesses and governments around the world.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote:
It might help if you included a couple of references/examples of the pages that are slow.
To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports.
I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds.
Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background.
From: Louis Bailleul <louis.bailleul@phangos.fr mailto:louis.bailleul@phangos.fr> Reply-To: "observium@observium.org mailto:observium@observium.org" <observium@observium.org mailto:observium@observium.org> Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org mailto:observium@observium.org" <observium@observium.org mailto:observium@observium.org> Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
*Founded in 2007, IO provides the data center as a service to businesses and governments around the world.*
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total). So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts. Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of the pages that are slow. To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports. I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds. Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background. From: Louis Bailleul Reply-To: "observium@observium.org (mailto:observium@observium.org)" Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org (mailto:observium@observium.org)" Subject: [Observium] Speed up graph rendering Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ? Best regards, Louis
Founded in 2007, IO provides the data center as a service to businesses and governments around the world.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
_______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
Hi Lewis,
The DBA separated the RRD and MySQL to different volumes as they compete for disk resources and so caused issues with the amount of hosts we were polling.
This would have a much greater effect on performance than simply moving to faster storage.
Hope this helps Alastair From: observium [mailto:observium-bounces@observium.org] On Behalf Of Louis Bailleul Sent: 09 January 2015 16:52 To: Observium Network Observation System Subject: Re: [Observium] Speed up graph rendering
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" <tom.laermans@powersource.cxmailto:%22Tom%20Laermans%22%20%3ctom.laermans@powersource.cx%3e> wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote:
It might help if you included a couple of references/examples of the pages that are slow.
To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports.
I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds.
Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background.
From: Louis Bailleul <louis.bailleul@phangos.frmailto:louis.bailleul@phangos.fr> Reply-To: "observium@observium.orgmailto:observium@observium.org" <observium@observium.orgmailto:observium@observium.org> Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.orgmailto:observium@observium.org" <observium@observium.orgmailto:observium@observium.org> Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
Founded in 2007, IO provides the data center as a service to businesses and governments around the world.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
_______________________________________________
observium mailing list
observium@observium.orgmailto:observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
The information contained in this email and any attachments is confidential and may be subject to copyright or other intellectual property protection. If you are not the intended recipient, you are not authorized to use or disclose this information, and we request that you notify us by reply mail or telephone and delete the original message from your mail system.
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs.
adam.
------ Original Message ------ From: "Louis Bailleul" louis.bailleul@phangos.fr To: "Observium Network Observation System" observium@observium.org Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx%3E wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote:
It might help if you included a couple of references/examples of the pages that are slow.
To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports.
I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds.
Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background.
From: Louis Bailleul louis.bailleul@phangos.fr Reply-To: "observium@observium.org" observium@observium.org Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org" observium@observium.org Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
Founded in 2007, IO provides the data center as a service to businesses and governments around the world.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. _______________________________________________ observium mailing list observium@observium.orghttp://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create. What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ? Best regards, Louis January 9 2015 5:01 PM, "Adam Armstrong" wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs. adam. ------ Original Message ------ From: "Louis Bailleul" To: "Observium Network Observation System" Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total). So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts. Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of the pages that are slow. To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports. I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds. Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background. From: Louis Bailleul Reply-To: "observium@observium.org (mailto:observium@observium.org)" Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org (mailto:observium@observium.org)" Subject: [Observium] Speed up graph rendering Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ? Best regards, Louis
Founded in 2007, IO provides the data center as a service to businesses and governments around the world.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
_______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive.
An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster.
/Markus
2015-01-12 10:52 GMT+01:00 Louis Bailleul louis.bailleul@phangos.fr:
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create.
What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" <adama@memetic.org <%22Adam%20Armstrong%22%20%3Cadama@memetic.org%3E>> wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs.
adam.
------ Original Message ------ From: "Louis Bailleul" louis.bailleul@phangos.fr To: "Observium Network Observation System" observium@observium.org Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" < mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx%3E <%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx%3E>> wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote:
It might help if you included a couple of references/examples of the pages that are slow.
To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports.
I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds.
Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background.
From: Louis Bailleul louis.bailleul@phangos.fr Reply-To: "observium@observium.org" observium@observium.org Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org" observium@observium.org Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
*Founded in 2007, IO provides the data center as a service to businesses and governments around the world.*
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
observium mailing listobservium@observium.orghttp://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Alright, I'll give a go to the ramdisk when I have the chance.
Something that I noticed when digging around was that rrdtool seems to be unable to be called more than once by httpd :
httpd-+-httpd |-httpd |-httpd |-httpd---rrdtool |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd `-httpd
At any point in time only one of the httpd process has a forked rrdtool process.
Is it the expected behaviour ?
There is about 300 graphs to render during the output above, so I expected to have all httpd process trying to render graphs. Best regards, Louis January 12 2015 10:41 AM, "Markus Klock" wrote:
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive. An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html (http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html) SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster. /Markus 2015-01-12 10:52 GMT+01:00 Louis Bailleul :
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create. What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ? Best regards, Louis January 9 2015 5:01 PM, "Adam Armstrong" wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs. adam. ------ Original Message ------ From: "Louis Bailleul" To: "Observium Network Observation System" Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total). So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts. Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of the pages that are slow. To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports. I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds. Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background. From: Louis Bailleul Reply-To: "observium@observium.org (mailto:observium@observium.org)" Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org (mailto:observium@observium.org)" Subject: [Observium] Speed up graph rendering Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ? Best regards, Louis
Founded in 2007, IO provides the data center as a service to businesses and governments around the world.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
_______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
_______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
Hi,
For what Observium's concerned:
- Every image is an <img> tag in HTML, as such it is a separate request to the webserver - Images are created by graph.php, which calls rrdtool once (twice if there's an error, to give you the "error" graph) - Your browser contacts the webserver for all assets it thinks it needs to load.
What may be at play here are your browser settings, which limit the number of simultaneous connections to a single webserver, to not overload it with a barrage of requests, using http keepalives to request one image after the other.
Try fiddling with browser-side settings and see if you can get it to hammer your Observium server harder.
Tom
On 01/12/2015 12:19 PM, Louis Bailleul wrote:
Alright, I'll give a go to the ramdisk when I have the chance.
Something that I noticed when digging around was that rrdtool seems to be unable to be called more than once by httpd :
httpd-+-httpd |-httpd |-httpd |-httpd---rrdtool |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd `-httpd
At any point in time only one of the httpd process has a forked rrdtool process.
Is it the expected behaviour ?
There is about 300 graphs to render during the output above, so I expected to have all httpd process trying to render graphs.
Best regards, Louis
January 12 2015 10:41 AM, "Markus Klock" <markus@best-practice.se mailto:%22Markus%20Klock%22%20%3Cmarkus@best-practice.se%3E> wrote:
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive. An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster. /Markus 2015-01-12 10:52 GMT+01:00 Louis Bailleul <louis.bailleul@phangos.fr <mailto:louis.bailleul@phangos.fr>>: Hi Adam , Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ? Currently our rrd dir is du -sh rrd/ 21G rrd/ That fit in a 24-32Go ram VM which is something that we can create. What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ? Is there any other option than ramdisk storage ? Best regards, Louis January 9 2015 5:01 PM, "Adam Armstrong" <adama@memetic.org <mailto:%22Adam%20Armstrong%22%20%3Cadama@memetic.org%3E>> wrote: Depending upon your deployment size, I usually recommend ramdisk storage for RRDs. adam. ------ Original Message ------ From: "Louis Bailleul" <louis.bailleul@phangos.fr <mailto:louis.bailleul@phangos.fr>> To: "Observium Network Observation System" <observium@observium.org <mailto:observium@observium.org>> Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph renderingHi, Thank you for the pointers. To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug. Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached). On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total). So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ? Could it be that apache need tuning to handle that amount of call to graph.php ? For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts. Best regards, Louis January 9 2015 3:11 PM, "Tom Laermans" <mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx%3E> wrote: Hi, You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O). Tom On 01/09/2015 04:08 PM, Pedersen, Sean wrote:It might help if you included a couple of references/examples of the pages that are slow. To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports. I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds. Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background. From: Louis Bailleul <louis.bailleul@phangos.fr <mailto:louis.bailleul@phangos.fr>> Reply-To: "observium@observium.org <mailto:observium@observium.org>" <observium@observium.org <mailto:observium@observium.org>> Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org <mailto:observium@observium.org>" <observium@observium.org <mailto:observium@observium.org>> Subject: [Observium] Speed up graph rendering Hi, I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs. Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks. Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN. Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ? Are they special requirements/tunnings for the rrd storage or the mysql database ? Best regards, Louis *Founded in 2007, IO provides the data center as a service to businesses and governments around the world.* The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. _______________________________________________ observium mailing list observium@observium.org <mailto:observium@observium.org> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium_______________________________________________ observium mailing list observium@observium.org <mailto:observium@observium.org> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi,
I have the same behaviour with Chrome 39, Firefox 34 and even IE 8.
I tried changing Firefox network.http.max-persistent-connections-per-server; from the default 6 to 32 , 256 and even 4096.
This doesn't seems to provide the expected behaviour as it only creates more httpd forks (188 counted with the 4096 setting) but still only one of them seems to be calling rrdtool.
There is no speed up in the graph rendering and even worse, with 256 some graphs failed with "No Auth" and with 4096 the page stop loading before the end leaving missing graphs. By the way I am using apache 2.2-15 with MPM prefork if that matters. Best regards, Louis January 12 2015 1:33 PM, "Tom Laermans" wrote:
Hi,
For what Observium's concerned:
- Every image is an tag in HTML, as such it is a separate request to the webserver - Images are created by graph.php, which calls rrdtool once (twice if there's an error, to give you the "error" graph) - Your browser contacts the webserver for all assets it thinks it needs to load.
What may be at play here are your browser settings, which limit the number of simultaneous connections to a single webserver, to not overload it with a barrage of requests, using http keepalives to request one image after the other.
Try fiddling with browser-side settings and see if you can get it to hammer your Observium server harder.
Tom
On 01/12/2015 12:19 PM, Louis Bailleul wrote: Alright, I'll give a go to the ramdisk when I have the chance.
Something that I noticed when digging around was that rrdtool seems to be unable to be called more than once by httpd :
httpd-+-httpd |-httpd |-httpd |-httpd---rrdtool |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd `-httpd
At any point in time only one of the httpd process has a forked rrdtool process.
Is it the expected behaviour ?
There is about 300 graphs to render during the output above, so I expected to have all httpd process trying to render graphs. Best regards, Louis January 12 2015 10:41 AM, "Markus Klock" wrote:
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive. An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html (http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html) SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster. /Markus 2015-01-12 10:52 GMT+01:00 Louis Bailleul :
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create. What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ? Best regards, Louis January 9 2015 5:01 PM, "Adam Armstrong" wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs. adam. ------ Original Message ------ From: "Louis Bailleul" To: "Observium Network Observation System" Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total). So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts. Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of the pages that are slow. To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports. I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds. Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background. From: Louis Bailleul Reply-To: "observium@observium.org (mailto:observium@observium.org)" Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org (mailto:observium@observium.org)" Subject: [Observium] Speed up graph rendering Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ? Best regards, Louis
Founded in 2007, IO provides the data center as a service to businesses and governments around the world.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
_______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
_______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
_______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
Hi! In the March 2013 at the openNMS user conference, Tobias told he found a non-optimal code in the RRDtool, which lead to 10 times slowdown of graphing when using devices with a lot of interfaces. This seems to be relevant for initial requestor - he has about 100 ports per device. Tobias proposed a fix-up for this in the future releases. As of March 2013 RRDtool was version 1.4.7 (and is still this version for Ubuntu APT repos), so this should be fixed in 1.4.8 or 1.4.9 Now even RRDtool 1.5.0 is available, if anyone could get it working it might speed-up graphing. HTH -- Best, Sergey понедельник, 12 января 2015г., 17:09 +0300 от "Louis Bailleul" louis.bailleul@phangos.fr: Hi, I have the same behaviour with Chrome 39, Firefox 34 and even IE 8. I tried changing Firefox network.http.max-persistent-connections-per-server; from the default 6 to 32 , 256 and even 4096. This doesn't seems to provide the expected behaviour as it only creates more httpd forks (188 counted with the 4096 setting) but still only one of them seems to be calling rrdtool. There is no speed up in the graph rendering and even worse, with 256 some graphs failed with "No Auth" and with 4096 the page stop loading before the end leaving missing graphs. By the way I am using apache 2.2-15 with MPM prefork if that matters. Best regards, Louis January 12 2015 1:33 PM, "Tom Laermans" < tom.laermans@powersource.cx > wrote:
Hi,
For what Observium's concerned:
- Every image is an <img> tag in HTML, as such it is a separate request to the webserver
- Images are created by graph.php, which calls rrdtool once (twice if there's an error, to give you the "error" graph)
- Your browser contacts the webserver for all assets it thinks it needs to load.
What may be at play here are your browser settings, which limit the number of simultaneous connections to a single webserver, to not overload it with a barrage of requests, using http keepalives to request one image after the other.
Try fiddling with browser-side settings and see if you can get it to hammer your Observium server harder.
Tom
On 01/12/2015 12:19 PM, Louis Bailleul wrote:
Alright, I'll give a go to the ramdisk when I have the chance.
Something that I noticed when digging around was that rrdtool seems to be unable to be called more than once by httpd :
httpd-+-httpd |-httpd |-httpd |-httpd---rrdtool |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd `-httpd
At any point in time only one of the httpd process has a forked rrdtool process.
Is it the expected behaviour ?
There is about 300 graphs to render during the output above, so I expected to have all httpd process trying to render graphs.
Best regards, Louis
January 12 2015 10:41 AM, "Markus Klock" < markus@best-practice.se > wrote:
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive. An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster. /Markus 2015-01-12 10:52 GMT+01:00 Louis Bailleul < louis.bailleul@phangos.fr > :
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create.
What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" < adama@memetic.org > wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs. adam. ------ Original Message ------ From: "Louis Bailleul" < louis.bailleul@phangos.fr > To: "Observium Network Observation System" < observium@observium.org > Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" < mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx%3E > wrote: > >Hi, > >You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O). > >Tom > >On 01/09/2015 04:08 PM, Pedersen, Sean wrote: >>It might help if you included a couple of references/examples of the pages that are slow. >> >>To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports. >> >>I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds. >> >>Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background. >> >>From: Louis Bailleul < louis.bailleul@phangos.fr > >>Reply-To: " observium@observium.org " < observium@observium.org > >>Date: Friday, January 9, 2015 at 7:46 AM >>To: " observium@observium.org " < observium@observium.org > >>Subject: [Observium] Speed up graph rendering >> >>Hi, >> >>I am currently using Observium on 149 switches with a total of 10173 ports. >>I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs. >> >>Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. >>It's virtual disks are on a local raid5 composed of 10K SAS disks. >> >>Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. >>The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN. >> >>Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ? >> >>Are they special requirements/tunnings for the rrd storage or the mysql database ? >> >> >>Best regards, >>Louis >> >> >> >> >> >>Founded in 2007, IO provides the data center as a service to businesses and governments around the world. >> >>The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. >> >>
_______________________________________________ observium mailing list
>>observium@observium.org >>http://postman.memetic.org/cgi-bin/mailman/listinfo/observium >
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list
observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi,
Ok, so I believe the fix you mention is :
RRDtool 1.4.8 - 2012-05-23 ========================== Highlights ---------- * rrd_graph now uses a map to lookup variable names causing graphs with many items to be drawn magnitudes faster as the linear search of the variable tables is gone now.
I am using 1.4.7 and will see if I can upgrade easily to 1.4.8 to test if it speeds things up. Best regards, Louis
January 12 2015 4:10 PM, "Сережка Хомяков" wrote:
Hi!
In the March 2013 at the openNMS user conference, Tobias told he found a non-optimal code in the RRDtool, which lead to 10 times slowdown of graphing when using devices with a lot of interfaces. This seems to be relevant for initial requestor - he has about 100 ports per device. Tobias proposed a fix-up for this in the future releases. As of March 2013 RRDtool was version 1.4.7 (and is still this version for Ubuntu APT repos), so this should be fixed in 1.4.8 or 1.4.9 Now even RRDtool 1.5.0 is available, if anyone could get it working it might speed-up graphing.
HTH
-- Best, Sergey
понедельник, 12 января 2015г., 17:09 +0300 от "Louis Bailleul" :
Hi,
I have the same behaviour with Chrome 39, Firefox 34 and even IE 8.
I tried changing Firefox network.http.max-persistent-connections-per-server; from the default 6 to 32 , 256 and even 4096.
This doesn't seems to provide the expected behaviour as it only creates more httpd forks (188 counted with the 4096 setting) but still only one of them seems to be calling rrdtool.
There is no speed up in the graph rendering and even worse, with 256 some graphs failed with "No Auth" and with 4096 the page stop loading before the end leaving missing graphs.
By the way I am using apache 2.2-15 with MPM prefork if that matters.
Best regards, Louis
January 12 2015 1:33 PM, "Tom Laermans" < tom.laermans@powersource.cx (mailto:tom.laermans@powersource.cx) > wrote:
Hi,
For what Observium's concerned:
- Every image is an tag in HTML, as such it is a separate request to the webserver
- Images are created by graph.php, which calls rrdtool once (twice if there's an error, to give you the "error" graph)
- Your browser contacts the webserver for all assets it thinks it needs to load.
What may be at play here are your browser settings, which limit the number of simultaneous connections to a single webserver, to not overload it with a barrage of requests, using http keepalives to request one image after the other.
Try fiddling with browser-side settings and see if you can get it to hammer your Observium server harder.
Tom
On 01/12/2015 12:19 PM, Louis Bailleul wrote:
Alright, I'll give a go to the ramdisk when I have the chance.
Something that I noticed when digging around was that rrdtool seems to be unable to be called more than once by httpd :
httpd-+-httpd |-httpd |-httpd |-httpd---rrdtool |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd `-httpd
At any point in time only one of the httpd process has a forked rrdtool process.
Is it the expected behaviour ?
There is about 300 graphs to render during the output above, so I expected to have all httpd process trying to render graphs.
Best regards, Louis
January 12 2015 10:41 AM, "Markus Klock" < markus@best-practice.se (mailto:markus@best-practice.se) > wrote:
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive. An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html (http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html) SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster. /Markus 2015-01-12 10:52 GMT+01:00 Louis Bailleul < louis.bailleul@phangos.fr (mailto:louis.bailleul@phangos.fr) > :
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create.
What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" < adama@memetic.org (mailto:adama@memetic.org) > wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs. adam. ------ Original Message ------ From: "Louis Bailleul" < louis.bailleul@phangos.fr (mailto:louis.bailleul@phangos.fr) > To: "Observium Network Observation System" < observium@observium.org (mailto:observium@observium.org) > Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" < mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx (mailto:3Ctom.laermans@powersource.cx)%3E > wrote: > >Hi, > >You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O). > >Tom > >On 01/09/2015 04:08 PM, Pedersen, Sean wrote: >>It might help if you included a couple of references/examples of the pages that are slow. >> >>To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports. >> >>I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds. >> >>Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background. >> >>From: Louis Bailleul < louis.bailleul@phangos.fr (mailto:louis.bailleul@phangos.fr) > >>Reply-To: " observium@observium.org (mailto:observium@observium.org) " < observium@observium.org (mailto:observium@observium.org) > >>Date: Friday, January 9, 2015 at 7:46 AM >>To: " observium@observium.org (mailto:observium@observium.org) " < observium@observium.org (mailto:observium@observium.org) > >>Subject: [Observium] Speed up graph rendering >> >>Hi, >> >>I am currently using Observium on 149 switches with a total of 10173 ports. >>I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs. >> >>Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. >>It's virtual disks are on a local raid5 composed of 10K SAS disks. >> >>Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. >>The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN. >> >>Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ? >> >>Are they special requirements/tunnings for the rrd storage or the mysql database ? >> >> >>Best regards, >>Louis >> >> >> >> >> >>Founded in 2007, IO provides the data center as a service to businesses and governments around the world. >> >>The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. >> >>
_______________________________________________ observium mailing list
>>observium@observium.org (mailto:observium@observium.org) >>http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium) >
observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
_______________________________________________ observium mailing list
observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
_______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
Use rrdcached.
Em segunda-feira, 12 de janeiro de 2015, Louis Bailleul < louis.bailleul@phangos.fr> escreveu:
Hi,
Ok, so I believe the fix you mention is :
RRDtool 1.4.8 - 2012-05-23
Highlights
- rrd_graph now uses a map to lookup variable names causing graphs with many items to be drawn magnitudes faster as the linear search of the variable tables is gone now.
I am using 1.4.7 and will see if I can upgrade easily to 1.4.8 to test if it speeds things up.
Best regards, Louis
January 12 2015 4:10 PM, "Сережка Хомяков" <xomka686@mail.ru javascript:_e(%7B%7D,'cvml','%5Cx22%5Cu0421%5Cu0435%5Cu0440%5Cu0435%5Cu0436%5Cu043a%5Cu0430+%5Cu0425%5Cu043e%5Cu043c%5Cu044f%5Cu043a%5Cu043e%5Cu0432%5Cx22+%5Cx3cxomka686@mail.ru%5Cx3e');> wrote:
Hi!
In the March 2013 at the openNMS user conference, Tobias told he found a non-optimal code in the RRDtool, which lead to 10 times slowdown of graphing when using devices with a lot of interfaces. This seems to be relevant for initial requestor - he has about 100 ports per device. Tobias proposed a fix-up for this in the future releases. As of March 2013 RRDtool was version 1.4.7 (and is still this version for Ubuntu APT repos), so this should be fixed in 1.4.8 or 1.4.9 Now even RRDtool 1.5.0 is available, if anyone could get it working it might speed-up graphing.
HTH
-- Best, Sergey
понедельник, 12 января 2015г., 17:09 +0300 от "Louis Bailleul" < louis.bailleul@phangos.fr javascript:_e(%7B%7D,'cvml','louis.bailleul@phangos.fr');>:
Hi,
I have the same behaviour with Chrome 39, Firefox 34 and even IE 8.
I tried changing Firefox network.http.max-persistent-connections-per-server; from the default 6 to 32 , 256 and even 4096.
This doesn't seems to provide the expected behaviour as it only creates more httpd forks (188 counted with the 4096 setting) but still only one of them seems to be calling rrdtool.
There is no speed up in the graph rendering and even worse, with 256 some graphs failed with "No Auth" and with 4096 the page stop loading before the end leaving missing graphs.
By the way I am using apache 2.2-15 with MPM prefork if that matters.
Best regards, Louis
January 12 2015 1:33 PM, "Tom Laermans" < tom.laermans@powersource.cx javascript:_e(%7B%7D,'cvml','tom.laermans@powersource.cx'); > wrote:
Hi,
For what Observium's concerned:
- Every image is an <img> tag in HTML, as such it is a separate request
to the webserver
- Images are created by graph.php, which calls rrdtool once (twice if
there's an error, to give you the "error" graph)
- Your browser contacts the webserver for all assets it thinks it needs
to load.
What may be at play here are your browser settings, which limit the
number of simultaneous connections to a single webserver, to not overload it with a barrage of requests, using http keepalives to request one image after the other.
Try fiddling with browser-side settings and see if you can get it to
hammer your Observium server harder.
Tom
On 01/12/2015 12:19 PM, Louis Bailleul wrote:
Alright, I'll give a go to the ramdisk when I have the chance.
Something that I noticed when digging around was that rrdtool seems to
be unable to be called more than once by httpd :
httpd-+-httpd |-httpd |-httpd |-httpd---rrdtool |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd |-httpd `-httpd
At any point in time only one of the httpd process has a forked rrdtool
process.
Is it the expected behaviour ?
There is about 300 graphs to render during the output above, so I
expected to have all httpd process trying to render graphs.
Best regards, Louis
January 12 2015 10:41 AM, "Markus Klock" < markus@best-practice.se
javascript:_e(%7B%7D,'cvml','markus@best-practice.se'); > wrote:
The ramdisk itself is not very reliable no, the trick is to make a
backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive.
An option to ramdisk would be to use rrdcached. In large enviroments
rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
SSD disks + rrdcached could handle a pretty big enviroment. But using a
ramdisk is still very very much faster.
/Markus
2015-01-12 10:52 GMT+01:00 Louis Bailleul < louis.bailleul@phangos.fr
javascript:_e(%7B%7D,'cvml','louis.bailleul@phangos.fr'); > :
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing
large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create.
What do you think of the mysql tuning that have been suggested by
Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" < adama@memetic.org
javascript:_e(%7B%7D,'cvml','adama@memetic.org'); > wrote:
Depending upon your deployment size, I usually recommend ramdisk
storage for RRDs.
adam.
------ Original Message ------ From: "Louis Bailleul" < louis.bailleul@phangos.fr
javascript:_e(%7B%7D,'cvml','louis.bailleul@phangos.fr'); >
To: "Observium Network Observation System" < observium@observium.org
javascript:_e(%7B%7D,'cvml','observium@observium.org'); >
Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering
>Hi, > >Thank you for the pointers. > >To be more specific the page generation for an arista switch with
760 ports take 0.567 seconds.
>But the loading of all graphs (a lot of sensors) take 8m 53s
according to firebug.
> >Generating the ports view page takes 1.421s but loading all the
graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
> >On a force10 switch with 48 ports page generates in 0.207seconds and
the page loads in 2.73s.
>The ports view takes 0.311 seconds to generate and the page finishes
loading graphs after 1m8s (205 HTTP requests in total).
> > >So from what I understand the mysql speed might not be in cause but
maybe the RRD disk I/O.
>What can I do to speed it up ? >Moving the disk on faster storage will be enough ? > >Could it be that apache need tuning to handle that amount of call to
graph.php ?
> >For the record the disk system is an IBM V3700 SAS attached to the
ESXi hosts.
> > >Best regards, >Louis > >January 9 2015 3:11 PM, "Tom Laermans" < mailto:
%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx javascript:_e(%7B%7D,'cvml','3Ctom.laermans@powersource.cx');%3E > wrote:
>> >>Hi, >> >>You must note however, that the load time for the page as displayed
by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs
- but not the loading of the graphs themselves (which is heavily influenced
by both CPU and RRD Disk I/O).
>> >>Tom >> >>On 01/09/2015 04:08 PM, Pedersen, Sean wrote: >>>It might help if you included a couple of references/examples of
the pages that are slow.
>>> >>>To give you something to compare against, we’re running Observium
on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports.
>>> >>>I looked at a Cisco switch stack that has 115 active ports. Graphs
System loaded in .45 seconds, which is only 16 different graphs between
four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds.
>>> >>>Now…I have some custom port lists, one of which is 36 graphs,
which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background.
>>> >>>From: Louis Bailleul < louis.bailleul@phangos.fr
javascript:_e(%7B%7D,'cvml','louis.bailleul@phangos.fr'); >
>>>Reply-To: " observium@observium.org
javascript:_e(%7B%7D,'cvml','observium@observium.org'); " < observium@observium.org javascript:_e(%7B%7D,'cvml','observium@observium.org'); >
>>>Date: Friday, January 9, 2015 at 7:46 AM >>>To: " observium@observium.org
javascript:_e(%7B%7D,'cvml','observium@observium.org'); " < observium@observium.org javascript:_e(%7B%7D,'cvml','observium@observium.org'); >
>>>Subject: [Observium] Speed up graph rendering >>> >>>Hi, >>> >>>I am currently using Observium on 149 switches with a total of
10173 ports.
>>>I can't say that anything is wrong with it, but the web interface
seems a bit slow especially when you want to display pages with a lot of graphs.
>>> >>>Observium is currently running in a VM with 4 xeon E5-2650 cores
and 3Gb of ram.
>>>It's virtual disks are on a local raid5 composed of 10K SAS disks. >>> >>>Observium report that it's VM use on average 25% of its CPUs with
spikes at 40% when the web interface is heavily used.
>>>The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN. >>> >>>Anyone could advise on what kind of hardware or software tweaking
could help speed up the thing ?
>>> >>>Are they special requirements/tunnings for the rrd storage or the
mysql database ?
>>> >>> >>>Best regards, >>>Louis >>> >>> >>> >>> >>> >>>Founded in 2007, IO provides the data center as a service to
businesses and governments around the world.
>>> >>>The communication contained in this e-mail is confidential and is
intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
>>> >>>
observium mailing list
>>>observium@observium.org
javascript:_e(%7B%7D,'cvml','observium@observium.org');
>>>http://postman.memetic.org/cgi-bin/mailman/listinfo/observium >>
observium mailing list observium@observium.org
javascript:_e(%7B%7D,'cvml','observium@observium.org');
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list
observium@observium.org
javascript:_e(%7B%7D,'cvml','observium@observium.org');
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org javascript:_e(%7B%7D,'cvml','observium@observium.org'); http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Not that rrdcached only speeds up RRD writing, which is why we don't really use it ourselves.
It actually slows down reading (generating graphs), too!
adam.
On 2015-01-12 10:42, Markus Klock wrote:
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive.
An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [2] SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster.
/Markus
2015-01-12 10:52 GMT+01:00 Louis Bailleul louis.bailleul@phangos.fr:
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create.
What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" adama@memetic.org wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs. adam. ------ Original Message ------ From: "Louis Bailleul" louis.bailleul@phangos.fr To: "Observium Network Observation System" observium@observium.org Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx%3E wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of the pages that are slow. To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports. I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds. Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background. From: Louis Bailleul louis.bailleul@phangos.fr Reply-To: "observium@observium.org" observium@observium.org Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org" observium@observium.org Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
FOUNDED IN 2007, IO PROVIDES THE DATA CENTER AS A SERVICE TO BUSINESSES AND GOVERNMENTS AROUND THE WORLD.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [1]
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [1]
Links:
[1] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
This is all good info. I haven't used rrdcached before but I think I will give it a shot.
Regarding writes versus reads, aren't there many, many times the amount of writes compared to reads?
For instance, the RRDs for all components on all devices are written every 5 mins, correct?
Reads only occur when someone is browsing the website.
Am I correct about this?
Cheers,
Tristan
*Tristan Rhodes* Network Engineer Weber State University 801.626.8549
On Mon, Jan 12, 2015 at 3:41 PM, Adam Armstrong adama@memetic.org wrote:
Not that rrdcached only speeds up RRD writing, which is why we don't really use it ourselves.
It actually slows down reading (generating graphs), too!
adam.
On 2015-01-12 10:42, Markus Klock wrote:
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive.
An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using- rrdcached-with-observium.html [2]
SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster.
/Markus
2015-01-12 10:52 GMT+01:00 Louis Bailleul louis.bailleul@phangos.fr:
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create.
What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" adama@memetic.org wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs.
adam.
------ Original Message ------ From: "Louis Bailleul" louis.bailleul@phangos.fr To: "Observium Network Observation System" observium@observium.org Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" mailto:%22Tom%20Laermans%22% 20%3Ctom.laermans@powersource.cx%3E wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of the pages that are slow.
To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports.
I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds.
Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background.
From: Louis Bailleul louis.bailleul@phangos.fr Reply-To: "observium@observium.org" observium@observium.org Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org" observium@observium.org Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
FOUNDED IN 2007, IO PROVIDES THE DATA CENTER AS A SERVICE TO BUSINESSES AND GOVERNMENTS AROUND THE WORLD.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [1]
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [1]
Links:
[1] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] http://blog.best-practice.se/2014/10/using-rrdcached-with- observium.html
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi,
I agree with what Adam says about rrdcached :
I gave it a go yesterday and it does improve the overall I/O usage of Observium but it slowed even more the loading of the pages : A pages that used to load in 12m :
1625 requests 7.1 MB 17m 45s
But we clearly see a drop in the I/O usage of Observium. Best regards, Louis January 13 2015 12:35 AM, "Tristan Rhodes" wrote:
This is all good info. I haven't used rrdcached before but I think I will give it a shot. Regarding writes versus reads, aren't there many, many times the amount of writes compared to reads? For instance, the RRDs for all components on all devices are written every 5 mins, correct? Reads only occur when someone is browsing the website. Am I correct about this? Cheers, Tristan Tristan Rhodes Network Engineer Weber State University 801.626.8549
On Mon, Jan 12, 2015 at 3:41 PM, Adam Armstrong wrote:
Not that rrdcached only speeds up RRD writing, which is why we don't really use it ourselves.
It actually slows down reading (generating graphs), too!
adam. On 2015-01-12 10:42, Markus Klock wrote: The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive.
An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html (http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html) [2] SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster.
/Markus
2015-01-12 10:52 GMT+01:00 Louis Bailleul :
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create.
What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs. adam. ------ Original Message ------ From: "Louis Bailleul" To: "Observium Network Observation System" Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of the pages that are slow. To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports. I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds. Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background. From: Louis Bailleul Reply-To: "observium@observium.org (mailto:observium@observium.org)" Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org (mailto:observium@observium.org)" Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
FOUNDED IN 2007, IO PROVIDES THE DATA CENTER AS A SERVICE TO BUSINESSES AND GOVERNMENTS AROUND THE WORLD.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
_______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium) [1]
_______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium) [1] Links: ------ [1] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium) [2] http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html (http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html)
_______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium) _______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
Tristan,
This is correct. Do note this is a thread about speeding up the rendering, not slowing it down, though ;-)
Tom
On 01/13/2015 01:36 AM, Tristan Rhodes wrote:
This is all good info. I haven't used rrdcached before but I think I will give it a shot.
Regarding writes versus reads, aren't there many, many times the amount of writes compared to reads?
For instance, the RRDs for all components on all devices are written every 5 mins, correct?
Reads only occur when someone is browsing the website.
Am I correct about this?
Cheers,
Tristan
*Tristan Rhodes* Network Engineer Weber State University 801.626.8549
On Mon, Jan 12, 2015 at 3:41 PM, Adam Armstrong <adama@memetic.org mailto:adama@memetic.org> wrote:
Not that rrdcached only speeds up RRD writing, which is why we don't really use it ourselves. It actually slows down reading (generating graphs), too! adam. On 2015-01-12 10:42, Markus Klock wrote: The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive. An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [2] SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster. /Markus 2015-01-12 10:52 GMT+01:00 Louis Bailleul <louis.bailleul@phangos.fr <mailto:louis.bailleul@phangos.fr>>: Hi Adam , Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ? Currently our rrd dir is du -sh rrd/ 21G rrd/ That fit in a 24-32Go ram VM which is something that we can create. What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ? Is there any other option than ramdisk storage ? Best regards, Louis January 9 2015 5:01 PM, "Adam Armstrong" <adama@memetic.org <mailto:adama@memetic.org>> wrote: Depending upon your deployment size, I usually recommend ramdisk storage for RRDs. adam. ------ Original Message ------ From: "Louis Bailleul" <louis.bailleul@phangos.fr <mailto:louis.bailleul@phangos.fr>> To: "Observium Network Observation System" <observium@observium.org <mailto:observium@observium.org>> Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering Hi, Thank you for the pointers. To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug. Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached). On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total). So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ? Could it be that apache need tuning to handle that amount of call to graph.php ? For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts. Best regards, Louis January 9 2015 3:11 PM, "Tom Laermans" <mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx <mailto:22Tom%2520Laermans%2522%2520%253Ctom.laermans@powersource.cx>%3E> wrote: Hi, You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O). Tom On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of the pages that are slow. To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports. I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds. Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background. From: Louis Bailleul <louis.bailleul@phangos.fr <mailto:louis.bailleul@phangos.fr>> Reply-To: "observium@observium.org <mailto:observium@observium.org>" <observium@observium.org <mailto:observium@observium.org>> Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org <mailto:observium@observium.org>" <observium@observium.org <mailto:observium@observium.org>> Subject: [Observium] Speed up graph rendering Hi, I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs. Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks. Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN. Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ? Are they special requirements/tunnings for the rrd storage or the mysql database ? Best regards, Louis FOUNDED IN 2007, IO PROVIDES THE DATA CENTER AS A SERVICE TO BUSINESSES AND GOVERNMENTS AROUND THE WORLD. The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. _______________________________________________ observium mailing list observium@observium.org <mailto:observium@observium.org> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [1] _______________________________________________ observium mailing list observium@observium.org <mailto:observium@observium.org> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [1] Links: ------ [1] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html _______________________________________________ observium mailing list observium@observium.org <mailto:observium@observium.org> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org <mailto:observium@observium.org> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
2015-01-13 11:33 GMT-02:00 Tom Laermans tom.laermans@powersource.cx:
Tristan,
This is correct. Do note this is a thread about speeding up the rendering, not slowing it down, though ;-)
Indeed.
Tom, what about the indexes I sent?
-- Eduardo Schoedler
On 13/01/2015 15:15, Eduardo Schoedler wrote:
2015-01-13 11:33 GMT-02:00 Tom Laermans <tom.laermans@powersource.cx mailto:tom.laermans@powersource.cx>:
Tristan, This is correct. Do note this is a thread about speeding up the rendering, not slowing it down, though ;-)Indeed.
Tom, what about the indexes I sent?
There have been commits in that direction recently, I don't know if all of yours were applied or if they were others - I don't have any spare time currently to look at them unfortunately - I'm sure Adam or Mike will pick them up soon if they haven't already.
Tom
If rrd write performance (poller runtime) is an issue, you might benefit from rrdcached (or SSD/ramdisk storage).
If rrd read performance is an issue, you will be better served by ssd or ramdisk storage. rrdcached /might/ give some performance improvements from reducing write i/o load, but the problem with rrdcached is that in order to read an rrd you need to signal to rrdcached to flush its cache so you can read the rrd, adding an extra step, which may not be trivial in terms of performance on our graph-heavy pages.
I still say that the best performance comes from ramdisk, with ssd an option too.
I've run all of my Observium systems from ramdisk for years, with automated backup/restore scripts. If I reboot a server, I only lose data since the last backup (backing up rrd data with fast compression is pretty quick, so you can run it fairly frequently).
My dislike of rrdcached comes from it never really helping in any of the instances i had performance problems. Sure, it makes polling slightly quicker, but it doesn't really do anything for the loading of graphs, which is where the human<>computer irritation comes from.
adam.
On 2015-01-13 00:36, Tristan Rhodes wrote:
This is all good info. I haven't used rrdcached before but I think I will give it a shot.
Regarding writes versus reads, aren't there many, many times the amount of writes compared to reads?
For instance, the RRDs for all components on all devices are written every 5 mins, correct?
Reads only occur when someone is browsing the website.
Am I correct about this?
Cheers,
Tristan
TRISTAN RHODES Network Engineer Weber State University 801.626.8549
On Mon, Jan 12, 2015 at 3:41 PM, Adam Armstrong adama@memetic.org wrote:
Not that rrdcached only speeds up RRD writing, which is why we don't really use it ourselves.
It actually slows down reading (generating graphs), too!
adam.
On 2015-01-12 10:42, Markus Klock wrote:
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive.
An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [1] [2]
SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster.
/Markus
2015-01-12 10:52 GMT+01:00 Louis Bailleul louis.bailleul@phangos.fr:
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create.
What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" adama@memetic.org wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs. adam. ------ Original Message ------ From: "Louis Bailleul" louis.bailleul@phangos.fr To: "Observium Network Observation System" observium@observium.org Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx%3E wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of the pages that are slow. To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports. I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds. Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background. From: Louis Bailleul louis.bailleul@phangos.fr Reply-To: "observium@observium.org" observium@observium.org Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org" observium@observium.org Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
FOUNDED IN 2007, IO PROVIDES THE DATA CENTER AS A SERVICE TO BUSINESSES AND GOVERNMENTS AROUND THE WORLD.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [1]
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [1]
Links:
[1] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [2] http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [1]
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2]
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2]
Links:
[1] http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [2] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi,
Sorry for the delay but I couldn't test it before.
I finally managed to replace rrdtool 1.4.7 by rrdtool 1.4.9.
And I am glad I did!
There is indeed something broken with 1.4.7 because the speed up is quite amazing.
The web interface is now blazing fast, even for my switches with 700+ ports. Pages which used to render in between 8 and 12min now are displayed in 34 seconds.
Thanks a lot for pointing this out and keep up the good work on this amazing tool.
Best regards, Louis
January 13 2015 6:59 PM, "Adam Armstrong" adama@memetic.org wrote:
If rrd write performance (poller runtime) is an issue, you might benefit from rrdcached (or SSD/ramdisk storage).
If rrd read performance is an issue, you will be better served by ssd or ramdisk storage. rrdcached /might/ give some performance improvements from reducing write i/o load, but the problem with rrdcached is that in order to read an rrd you need to signal to rrdcached to flush its cache so you can read the rrd, adding an extra step, which may not be trivial in terms of performance on our graph-heavy pages.
I still say that the best performance comes from ramdisk, with ssd an option too.
I've run all of my Observium systems from ramdisk for years, with automated backup/restore scripts. If I reboot a server, I only lose data since the last backup (backing up rrd data with fast compression is pretty quick, so you can run it fairly frequently).
My dislike of rrdcached comes from it never really helping in any of the instances i had performance problems. Sure, it makes polling slightly quicker, but it doesn't really do anything for the loading of graphs, which is where the human<>computer irritation comes from.
adam.
On 2015-01-13 00:36, Tristan Rhodes wrote:
This is all good info. I haven't used rrdcached before but I think I will give it a shot.
Regarding writes versus reads, aren't there many, many times the amount of writes compared to reads?
For instance, the RRDs for all components on all devices are written every 5 mins, correct?
Reads only occur when someone is browsing the website.
Am I correct about this?
Cheers,
Tristan
TRISTAN RHODES Network Engineer Weber State University 801.626.8549
On Mon, Jan 12, 2015 at 3:41 PM, Adam Armstrong adama@memetic.org wrote:
Not that rrdcached only speeds up RRD writing, which is why we don't really use it ourselves.
It actually slows down reading (generating graphs), too!
adam.
On 2015-01-12 10:42, Markus Klock wrote:
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive.
An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [1] [2]
SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster.
/Markus
2015-01-12 10:52 GMT+01:00 Louis Bailleul louis.bailleul@phangos.fr:
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create.
What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" adama@memetic.org wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs.
adam.
------ Original Message ------ From: "Louis Bailleul" louis.bailleul@phangos.fr To: "Observium Network Observation System" observium@observium.org Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx%3E wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of the pages that are slow.
To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports.
I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds.
Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background.
From: Louis Bailleul louis.bailleul@phangos.fr Reply-To: "observium@observium.org" observium@observium.org Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org" observium@observium.org Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
FOUNDED IN 2007, IO PROVIDES THE DATA CENTER AS A SERVICE TO BUSINESSES AND GOVERNMENTS AROUND THE WORLD.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [1]
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [1]
Links:
[1] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [2] http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [1]
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2]
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2]
Links:
[1] http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [2] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi!
That's great! I'm on the way of testing RRDtool 1.4.9 too. Now, even RRDtool 1.5.0 is available. -- Best, Sergey
Четверг, 29 января 2015, 9:50 UTC от "Louis Bailleul" louis.bailleul@phangos.fr:
Hi,
Sorry for the delay but I couldn't test it before.
I finally managed to replace rrdtool 1.4.7 by rrdtool 1.4.9.
And I am glad I did!
There is indeed something broken with 1.4.7 because the speed up is quite amazing.
The web interface is now blazing fast, even for my switches with 700+ ports. Pages which used to render in between 8 and 12min now are displayed in 34 seconds.
Thanks a lot for pointing this out and keep up the good work on this amazing tool.
Best regards, Louis
January 13 2015 6:59 PM, "Adam Armstrong" < adama@memetic.org > wrote:
If rrd write performance (poller runtime) is an issue, you might benefit from rrdcached (or SSD/ramdisk storage).
If rrd read performance is an issue, you will be better served by ssd or ramdisk storage. rrdcached /might/ give some performance improvements from reducing write i/o load, but the problem with rrdcached is that in order to read an rrd you need to signal to rrdcached to flush its cache so you can read the rrd, adding an extra step, which may not be trivial in terms of performance on our graph-heavy pages.
I still say that the best performance comes from ramdisk, with ssd an option too.
I've run all of my Observium systems from ramdisk for years, with automated backup/restore scripts. If I reboot a server, I only lose data since the last backup (backing up rrd data with fast compression is pretty quick, so you can run it fairly frequently).
My dislike of rrdcached comes from it never really helping in any of the instances i had performance problems. Sure, it makes polling slightly quicker, but it doesn't really do anything for the loading of graphs, which is where the human<>computer irritation comes from.
adam.
On 2015-01-13 00:36, Tristan Rhodes wrote:
This is all good info. I haven't used rrdcached before but I think I will give it a shot.
Regarding writes versus reads, aren't there many, many times the amount of writes compared to reads?
For instance, the RRDs for all components on all devices are written every 5 mins, correct?
Reads only occur when someone is browsing the website.
Am I correct about this?
Cheers,
Tristan
TRISTAN RHODES Network Engineer Weber State University 801.626.8549
On Mon, Jan 12, 2015 at 3:41 PM, Adam Armstrong < adama@memetic.org > wrote:
Not that rrdcached only speeds up RRD writing, which is why we don't really use it ourselves.
It actually slows down reading (generating graphs), too!
adam.
On 2015-01-12 10:42, Markus Klock wrote:
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive.
An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [1] [2]
SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster.
/Markus
2015-01-12 10:52 GMT+01:00 Louis Bailleul < louis.bailleul@phangos.fr >:
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create.
What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" < adama@memetic.org > wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs.
adam.
------ Original Message ------ From: "Louis Bailleul" < louis.bailleul@phangos.fr > To: "Observium Network Observation System" < observium@observium.org > Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx%3E wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of the pages that are slow.
To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports.
I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds.
Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background.
From: Louis Bailleul < louis.bailleul@phangos.fr > Reply-To: " observium@observium.org " < observium@observium.org > Date: Friday, January 9, 2015 at 7:46 AM To: " observium@observium.org " < observium@observium.org > Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
FOUNDED IN 2007, IO PROVIDES THE DATA CENTER AS A SERVICE TO BUSINESSES AND GOVERNMENTS AROUND THE WORLD.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [1]
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [1]
Links:
[1] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [2] http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [1]
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2]
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2]
Links:
[1] http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [2] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
I assume you compiled rrdtool from source?
http://oss.oetiker.ch/rrdtool/doc/rrdbuild.en.html
Are these the correct steps to do this?
1) Stop observium cronjobs 2) Uninstall rrdtool package (apt-get or yum) 3) Compile new version of rrdtool following instructions above 4) Move or link "rrdtool" to /usr/bin/rrdtool, or change "config.php" to where you compiled it: $config['rrdtool'] = "/usr/bin/rrdtool";
I guess I'll try this and see how bad I break my system... :)
Cheers,
Tristan
*Tristan Rhodes* Network Engineer Weber State University 801.626.8549
On Thu, Jan 29, 2015 at 4:21 AM, Сережка Хомяков xomka686@mail.ru wrote:
Hi!
That's great! I'm on the way of testing RRDtool 1.4.9 too. Now, even RRDtool 1.5.0 is available.
-- Best, Sergey
Четверг, 29 января 2015, 9:50 UTC от "Louis Bailleul" < louis.bailleul@phangos.fr>:
Hi,
Sorry for the delay but I couldn't test it before.
I finally managed to replace rrdtool 1.4.7 by rrdtool 1.4.9.
And I am glad I did!
There is indeed something broken with 1.4.7 because the speed up is quite amazing.
The web interface is now blazing fast, even for my switches with 700+ ports. Pages which used to render in between 8 and 12min now are displayed in 34 seconds.
Thanks a lot for pointing this out and keep up the good work on this amazing tool.
Best regards, Louis
January 13 2015 6:59 PM, "Adam Armstrong" <adama@memetic.org http:///compose?To=adama@memetic.org> wrote:
If rrd write performance (poller runtime) is an issue, you might benefit from rrdcached (or SSD/ramdisk storage).
If rrd read performance is an issue, you will be better served by ssd or ramdisk storage. rrdcached /might/ give some performance improvements from reducing write i/o load, but the problem with rrdcached is that in order to read an rrd you need to signal to rrdcached to flush its cache so you can read the rrd, adding an extra step, which may not be trivial in terms of performance on our graph-heavy pages.
I still say that the best performance comes from ramdisk, with ssd an option too.
I've run all of my Observium systems from ramdisk for years, with automated backup/restore scripts. If I reboot a server, I only lose data since the last backup (backing up rrd data with fast compression is pretty quick, so you can run it fairly frequently).
My dislike of rrdcached comes from it never really helping in any of the instances i had performance problems. Sure, it makes polling slightly quicker, but it doesn't really do anything for the loading of graphs, which is where the human<>computer irritation comes from.
adam.
On 2015-01-13 00:36, Tristan Rhodes wrote:
This is all good info. I haven't used rrdcached before but I think I will give it a shot.
Regarding writes versus reads, aren't there many, many times the amount of writes compared to reads?
For instance, the RRDs for all components on all devices are written every 5 mins, correct?
Reads only occur when someone is browsing the website.
Am I correct about this?
Cheers,
Tristan
TRISTAN RHODES Network Engineer Weber State University 801.626.8549
On Mon, Jan 12, 2015 at 3:41 PM, Adam Armstrong <adama@memetic.org
http:///compose?To=adama@memetic.org>
wrote:
Not that rrdcached only speeds up RRD writing, which is why we don't really use it ourselves.
It actually slows down reading (generating graphs), too!
adam.
On 2015-01-12 10:42, Markus Klock wrote:
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive.
An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here:
http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
[1] [2]
SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster.
/Markus
2015-01-12 10:52 GMT+01:00 Louis Bailleul <louis.bailleul@phangos.fr
http:///compose?To=louis.bailleul@phangos.fr>:
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create.
What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" <adama@memetic.org
http:///compose?To=adama@memetic.org> wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs.
adam.
------ Original Message ------ From: "Louis Bailleul" <louis.bailleul@phangos.fr
http:///compose?To=louis.bailleul@phangos.fr>
To: "Observium Network Observation System" <observium@observium.org
http:///compose?To=observium@observium.org>
Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx%3E wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of the pages that are slow.
To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports.
I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds.
Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background.
From: Louis Bailleul <louis.bailleul@phangos.fr
http:///compose?To=louis.bailleul@phangos.fr>
Reply-To: "observium@observium.org
http:///compose?To=observium@observium.org" <observium@observium.org http:///compose?To=observium@observium.org>
Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org
http:///compose?To=observium@observium.org" <observium@observium.org http:///compose?To=observium@observium.org>
Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
FOUNDED IN 2007, IO PROVIDES THE DATA CENTER AS A SERVICE TO BUSINESSES AND GOVERNMENTS AROUND THE WORLD.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
observium mailing list observium@observium.org http:///compose?To=observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [1]
observium mailing list observium@observium.org http:///compose?To=observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [1]
Links:
[1] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [2]
http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
[1]
observium mailing list observium@observium.org http:///compose?To=observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2]
observium mailing list observium@observium.org http:///compose?To=observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2]
Links:
[1]
http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
[2] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http:///compose?To=observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http:///compose?To=observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http:///compose?To=observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi,
I used that page to compile the tool myself
and I can run the 1.4.9 and 1.4.7 side by side,
I just installed the bin into /opt/rrdtool-1.4.9/bin/rrdtool instead of the default folder
and changed the config.php to that folder
Simon
On 29 Jan 2015, at 3:54 pm, Tristan Rhodes tristanrhodes@weber.edu wrote:
I assume you compiled rrdtool from source?
http://oss.oetiker.ch/rrdtool/doc/rrdbuild.en.html http://oss.oetiker.ch/rrdtool/doc/rrdbuild.en.html
Are these the correct steps to do this?
- Stop observium cronjobs
- Uninstall rrdtool package (apt-get or yum)
- Compile new version of rrdtool following instructions above
- Move or link "rrdtool" to /usr/bin/rrdtool, or change "config.php" to where you compiled it:
$config['rrdtool'] = "/usr/bin/rrdtool";
I guess I'll try this and see how bad I break my system... :)
Cheers,
Tristan
Tristan Rhodes Network Engineer Weber State University 801.626.8549
On Thu, Jan 29, 2015 at 4:21 AM, Сережка Хомяков <xomka686@mail.ru mailto:xomka686@mail.ru> wrote:
Hi!
That's great! I'm on the way of testing RRDtool 1.4.9 too. Now, even RRDtool 1.5.0 is available.
-- Best, Sergey
Четверг, 29 января 2015, 9:50 UTC от "Louis Bailleul" <louis.bailleul@phangos.fr mailto:louis.bailleul@phangos.fr>: Hi,
Sorry for the delay but I couldn't test it before.
I finally managed to replace rrdtool 1.4.7 by rrdtool 1.4.9.
And I am glad I did!
There is indeed something broken with 1.4.7 because the speed up is quite amazing.
The web interface is now blazing fast, even for my switches with 700+ ports. Pages which used to render in between 8 and 12min now are displayed in 34 seconds.
Thanks a lot for pointing this out and keep up the good work on this amazing tool.
Best regards, Louis
January 13 2015 6:59 PM, "Adam Armstrong" <adama@memetic.org http://compose/?To=adama@memetic.org> wrote:
If rrd write performance (poller runtime) is an issue, you might benefit from rrdcached (or SSD/ramdisk storage).
If rrd read performance is an issue, you will be better served by ssd or ramdisk storage. rrdcached /might/ give some performance improvements from reducing write i/o load, but the problem with rrdcached is that in order to read an rrd you need to signal to rrdcached to flush its cache so you can read the rrd, adding an extra step, which may not be trivial in terms of performance on our graph-heavy pages.
I still say that the best performance comes from ramdisk, with ssd an option too.
I've run all of my Observium systems from ramdisk for years, with automated backup/restore scripts. If I reboot a server, I only lose data since the last backup (backing up rrd data with fast compression is pretty quick, so you can run it fairly frequently).
My dislike of rrdcached comes from it never really helping in any of the instances i had performance problems. Sure, it makes polling slightly quicker, but it doesn't really do anything for the loading of graphs, which is where the human<>computer irritation comes from.
adam.
On 2015-01-13 00:36, Tristan Rhodes wrote:
This is all good info. I haven't used rrdcached before but I think I will give it a shot.
Regarding writes versus reads, aren't there many, many times the amount of writes compared to reads?
For instance, the RRDs for all components on all devices are written every 5 mins, correct?
Reads only occur when someone is browsing the website.
Am I correct about this?
Cheers,
Tristan
TRISTAN RHODES Network Engineer Weber State University 801.626.8549 tel:801.626.8549
On Mon, Jan 12, 2015 at 3:41 PM, Adam Armstrong <adama@memetic.org http://compose/?To=adama@memetic.org> wrote:
Not that rrdcached only speeds up RRD writing, which is why we don't really use it ourselves.
It actually slows down reading (generating graphs), too!
adam.
On 2015-01-12 10:42, Markus Klock wrote:
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive.
An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [1] [2]
SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster.
/Markus
2015-01-12 10:52 GMT+01:00 Louis Bailleul <louis.bailleul@phangos.fr http://compose/?To=louis.bailleul@phangos.fr>:
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create.
What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" <adama@memetic.org http://compose/?To=adama@memetic.org> wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs.
adam.
------ Original Message ------ From: "Louis Bailleul" <louis.bailleul@phangos.fr http://compose/?To=louis.bailleul@phangos.fr> To: "Observium Network Observation System" <observium@observium.org http://compose/?To=observium@observium.org> Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans" <mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx mailto:22Tom%2520Laermans%2522%2520%253Ctom.laermans@powersource.cx%3E> wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of the pages that are slow.
To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports.
I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds.
Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background.
From: Louis Bailleul <louis.bailleul@phangos.fr http://compose/?To=louis.bailleul@phangos.fr> Reply-To: "observium@observium.org http://compose/?To=observium@observium.org" <observium@observium.org http://compose/?To=observium@observium.org> Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org http://compose/?To=observium@observium.org" <observium@observium.org http://compose/?To=observium@observium.org> Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
FOUNDED IN 2007, IO PROVIDES THE DATA CENTER AS A SERVICE TO BUSINESSES AND GOVERNMENTS AROUND THE WORLD.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
observium mailing list observium@observium.org http://compose/?To=observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [1]
observium mailing list observium@observium.org http://compose/?To=observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [1]
Links:
[1] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2] [2] http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [1]
observium mailing list observium@observium.org http://compose/?To=observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2]
observium mailing list observium@observium.org http://compose/?To=observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [2]
Links:
[1] http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [2] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://compose/?To=observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://compose/?To=observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://compose/?To=observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Perhaps someone would like to make 1.4.9 or 1.5.0-rc packages for ubuntu and/or centos/rhel?
:>
adam.
On 2015-01-29 16:01, Simon Smith wrote:
Hi,
I used that page to compile the tool myself
and I can run the 1.4.9 and 1.4.7 side by side,
I just installed the bin into /opt/rrdtool-1.4.9/bin/rrdtool instead of the default folder
and changed the config.php to that folder
Simon
On 29 Jan 2015, at 3:54 pm, Tristan Rhodes tristanrhodes@weber.edu wrote:
I assume you compiled rrdtool from source?
http://oss.oetiker.ch/rrdtool/doc/rrdbuild.en.html [1]
Are these the correct steps to do this?
- Stop observium cronjobs
- Uninstall rrdtool package (apt-get or yum)
- Compile new version of rrdtool following instructions above
- Move or link "rrdtool" to /usr/bin/rrdtool, or change
"config.php" to where you compiled it: $config['rrdtool'] = "/usr/bin/rrdtool";
I guess I'll try this and see how bad I break my system... :)
Cheers,
Tristan
TRISTAN RHODES Network Engineer Weber State University 801.626.8549
On Thu, Jan 29, 2015 at 4:21 AM, Сережка Хомяков xomka686@mail.ru wrote:
Hi!
That's great! I'm on the way of testing RRDtool 1.4.9 too. Now, even RRDtool 1.5.0 is available.
-- Best, Sergey
Четверг, 29 января 2015, 9:50 UTC от "Louis Bailleul" louis.bailleul@phangos.fr:
Hi,
Sorry for the delay but I couldn't test it before.
I finally managed to replace rrdtool 1.4.7 by rrdtool 1.4.9.
And I am glad I did!
There is indeed something broken with 1.4.7 because the speed up is quite amazing.
The web interface is now blazing fast, even for my switches with 700+ ports. Pages which used to render in between 8 and 12min now are displayed in 34 seconds.
Thanks a lot for pointing this out and keep up the good work on this amazing tool.
Best regards, Louis
January 13 2015 6:59 PM, "Adam Armstrong" <adama@memetic.org [2]> wrote:
If rrd write performance (poller runtime) is an issue, you might
benefit
from rrdcached (or SSD/ramdisk storage).
If rrd read performance is an issue, you will be better served by
ssd or
ramdisk storage. rrdcached /might/ give some performance
improvements
from reducing write i/o load, but the problem with rrdcached is
that in
order to read an rrd you need to signal to rrdcached to flush its
cache
so you can read the rrd, adding an extra step, which may not be
trivial
in terms of performance on our graph-heavy pages.
I still say that the best performance comes from ramdisk, with
ssd an
option too.
I've run all of my Observium systems from ramdisk for years, with
automated backup/restore scripts. If I reboot a server, I only
lose data
since the last backup (backing up rrd data with fast compression
is
pretty quick, so you can run it fairly frequently).
My dislike of rrdcached comes from it never really helping in any
of the
instances i had performance problems. Sure, it makes polling
slightly
quicker, but it doesn't really do anything for the loading of
graphs,
which is where the human<>computer irritation comes from.
adam.
On 2015-01-13 00:36, Tristan Rhodes wrote:
This is all good info. I haven't used rrdcached before but I
think I
will give it a shot.
Regarding writes versus reads, aren't there many, many times the amount of writes compared to reads?
For instance, the RRDs for all components on all devices are
written
every 5 mins, correct?
Reads only occur when someone is browsing the website.
Am I correct about this?
Cheers,
Tristan
TRISTAN RHODES Network Engineer Weber State University 801.626.8549 [3]
On Mon, Jan 12, 2015 at 3:41 PM, Adam Armstrong
<adama@memetic.org [2]>
wrote:
Not that rrdcached only speeds up RRD writing, which is why we
don't
really use it ourselves.
It actually slows down reading (generating graphs), too!
adam.
On 2015-01-12 10:42, Markus Klock wrote:
The ramdisk itself is not very reliable no, the trick is to make
a
backup of the ramdisk to a reglar harddrive or SSD once in a
while so
that in a case of power outage or unexpected reboot you could
just
copy the data back to the ramdisk from the harddrive.
An option to ramdisk would be to use rrdcached. In large
enviroments
rrdcached could cut I/O in half or more. A quick guide can be
found
here:
http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
[4]
[1] [2]
SSD disks + rrdcached could handle a pretty big enviroment. But
using
a ramdisk is still very very much faster.
/Markus
2015-01-12 10:52 GMT+01:00 Louis Bailleul
<louis.bailleul@phangos.fr [5]>:
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but
storing
large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can
create.
What do you think of the mysql tuning that have been suggested
by
Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" <adama@memetic.org [2]>
wrote:
Depending upon your deployment size, I usually recommend ramdisk
storage for RRDs.
adam.
------ Original Message ------ From: "Louis Bailleul" <louis.bailleul@phangos.fr [5]> To: "Observium Network Observation System"
<observium@observium.org [6]>
Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch
with 760
ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s
according
to firebug.
Generating the ports view page takes 1.421s but loading all the
graphs
takes 12m 33s and 493 HTTP request (maybe not accurate as
firebug says
the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds
and
the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page
finishes
loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause
but
maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of
call to
graph.php ?
For the record the disk system is an IBM V3700 SAS attached to
the ESXi
hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans"
mailto:%22Tom%20Laermans%22%20%3Ctom.laermans@powersource.cx%3E
wrote:
Hi,
You must note however, that the load time for the page as
displayed by
Observium is the generation time of the HTML (which is
influenced by
MySQL speed and CPU, mostly), which includes image tags to load
the
graphs - but not the loading of the graphs themselves (which is
heavily
influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of
the
pages that are slow.
To give you something to compare against, we’re running
Observium on an
ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage
– but
we’re at a pretty consistent 40-50% utilization at all times,
using
about 2GB of RAM. This is with ~200 devices and 16K ports.
I looked at a Cisco switch stack that has 115 active ports.
Graphs >
System loaded in .45 seconds, which is only 16 different graphs
between
four categories. Likewise, the Ports page loaded at .64 seconds
with
the miniature preview graphs. Clicking on a specific port, it
loaded 24
graphs in .49 seconds.
Now…I have some custom port lists, one of which is 36 graphs,
which
loaded in .53 seconds. However, I have another group that
has…a metric
shit-tonne of ports and graphs on it – hundreds of them. The
page
loaded in 1.07 seconds, but it took several seconds more for all
of the
individual graphs to populate, which I expect considering the
sheer
number of them present in that page. I don’t know if that’s
simply the
browser catching up, or if it’s still generating the actual
graphs in
the background.
From: Louis Bailleul <louis.bailleul@phangos.fr [5]> Reply-To: "observium@observium.org [6]" <observium@observium.org
[6]>
Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org [6]" <observium@observium.org [6]> Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of
10173
ports. I can't say that anything is wrong with it, but the web
interface seems
a bit slow especially when you want to display pages with a lot
of
graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores
and
3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS
disks.
Observium report that it's VM use on average 25% of its CPUs
with
spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50
IN.
Anyone could advise on what kind of hardware or software
tweaking could
help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or
the mysql
database ?
Best regards, Louis
FOUNDED IN 2007, IO PROVIDES THE DATA CENTER AS A SERVICE TO
BUSINESSES
AND GOVERNMENTS AROUND THE WORLD.
The communication contained in this e-mail is confidential and
is
intended only for the named recipient(s) and may contain
information
that is privileged, proprietary, attorney work product or exempt
from
disclosure under applicable law. If you have received this
message in
error, or are not the named recipient(s), please note that any
form of
distribution, copying or use of this communication or the
information
in it is strictly prohibited and may be unlawful. Please
immediately
notify the sender of the error, and delete this communication
including
any attached files from your system. Thank you for your
cooperation.
observium mailing list observium@observium.org [6] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
[7] [2] [1]
observium mailing list observium@observium.org [6] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
[7] [2] [1]
Links:
[1]
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [7] [2]
[2]
http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
[4]
[1]
observium mailing list observium@observium.org [6] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
[7] [2]
observium mailing list observium@observium.org [6] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
[7] [2]
Links:
[1]
http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html
[4]
[2]
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [7]
observium mailing list observium@observium.org [6] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
[7]
observium mailing list observium@observium.org [6] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [7]
observium mailing list observium@observium.org [6] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [7]
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [7]
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Links:
[1] http://oss.oetiker.ch/rrdtool/doc/rrdbuild.en.html [2] http:/compose?To=adama@memetic.org [3] tel:801.626.8549 [4] http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html [5] http:/compose?To=louis.bailleul@phangos.fr [6] http:/compose?To=observium@observium.org [7] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
hello,
On 01/30/2015 06:41 PM, Adam Armstrong wrote:
Perhaps someone would like to make 1.4.9 or 1.5.0-rc packages for ubuntu and/or centos/rhel?
it looks like on ubuntu it's enough to install rrdtool 1.4.8 deb from utopic (it should be compatible with both 12.04 LTS and 14.04 LTS without dependency problems!):
wget http://mirrors.kernel.org/ubuntu/pool/main/r/rrdtool/rrdtool_1.4.8-1.1ubuntu... dpkg -i rrdtool_1.4.8-1.1ubuntu1_amd64.deb
as for the rrdtool version, I think the improvement is in 1.4.8 as you can find on the changelog (1.4.9 has only bugfixes, not performance improvements):
* rrd_graph now uses a map to lookup variable names causing graphs with many items to be drawn magnitudes faster as the linear search of the variable tables is gone now.
http://oss.oetiker.ch/rrdtool/pub/CHANGES
Hi,
If you do not use rrdcached that's more than enough. But as suggested if you do not alter the configure script you can have 1.4.9 installed in /opt/rrdtool-1.4.9/ and thus keep the old version in case things goes south. If you use rrdcached and intend to delete your packaged version. Save the /etc/init.d/rrdcached | /etc/rc.d/init.d/rrdcached and /etc/default/rrdcached | /etc/sysconfig/rrdcached files.
Those files are not provided by the sources ^^.
Also depending on where you install the library (librrd.so) you might need to copy/symlink it in your ldpath or alter your ldconfig file. Best regards, Louis January 29 2015 3:53 PM, "Tristan Rhodes" wrote:
I assume you compiled rrdtool from source? http://oss.oetiker.ch/rrdtool/doc/rrdbuild.en.html (http://oss.oetiker.ch/rrdtool/doc/rrdbuild.en.html) Are these the correct steps to do this? 1) Stop observium cronjobs 2) Uninstall rrdtool package (apt-get or yum) 3) Compile new version of rrdtool following instructions above 4) Move or link "rrdtool" to /usr/bin/rrdtool, or change "config.php" to where you compiled it: $config['rrdtool'] = "/usr/bin/rrdtool"; I guess I'll try this and see how bad I break my system... :) Cheers, Tristan Tristan Rhodes Network Engineer Weber State University 801.626.8549
On Thu, Jan 29, 2015 at 4:21 AM, Сережка Хомяков wrote:
Hi!
That's great! I'm on the way of testing RRDtool 1.4.9 too. Now, even RRDtool 1.5.0 is available.
-- Best, Sergey Четверг, 29 января 2015, 9:50 UTC от "Louis Bailleul" : Hi,
Sorry for the delay but I couldn't test it before. I finally managed to replace rrdtool 1.4.7 by rrdtool 1.4.9.
And I am glad I did!
There is indeed something broken with 1.4.7 because the speed up is quite amazing.
The web interface is now blazing fast, even for my switches with 700+ ports. Pages which used to render in between 8 and 12min now are displayed in 34 seconds.
Thanks a lot for pointing this out and keep up the good work on this amazing tool. Best regards, Louis January 13 2015 6:59 PM, "Adam Armstrong" wrote:
If rrd write performance (poller runtime) is an issue, you might benefit from rrdcached (or SSD/ramdisk storage).
If rrd read performance is an issue, you will be better served by ssd or ramdisk storage. rrdcached /might/ give some performance improvements from reducing write i/o load, but the problem with rrdcached is that in order to read an rrd you need to signal to rrdcached to flush its cache so you can read the rrd, adding an extra step, which may not be trivial in terms of performance on our graph-heavy pages.
I still say that the best performance comes from ramdisk, with ssd an option too.
I've run all of my Observium systems from ramdisk for years, with automated backup/restore scripts. If I reboot a server, I only lose data since the last backup (backing up rrd data with fast compression is pretty quick, so you can run it fairly frequently).
My dislike of rrdcached comes from it never really helping in any of the instances i had performance problems. Sure, it makes polling slightly quicker, but it doesn't really do anything for the loading of graphs, which is where the humancomputer irritation comes from.
adam.
On 2015-01-13 00:36, Tristan Rhodes wrote:
This is all good info. I haven't used rrdcached before but I think I will give it a shot.
Regarding writes versus reads, aren't there many, many times the amount of writes compared to reads?
For instance, the RRDs for all components on all devices are written every 5 mins, correct?
Reads only occur when someone is browsing the website.
Am I correct about this?
Cheers,
Tristan
TRISTAN RHODES Network Engineer Weber State University 801.626.8549 (javascript:false)
On Mon, Jan 12, 2015 at 3:41 PM, Adam Armstrong wrote:
Not that rrdcached only speeds up RRD writing, which is why we don't really use it ourselves.
It actually slows down reading (generating graphs), too!
adam.
On 2015-01-12 10:42, Markus Klock wrote:
The ramdisk itself is not very reliable no, the trick is to make a backup of the ramdisk to a reglar harddrive or SSD once in a while so that in a case of power outage or unexpected reboot you could just copy the data back to the ramdisk from the harddrive.
An option to ramdisk would be to use rrdcached. In large enviroments rrdcached could cut I/O in half or more. A quick guide can be found here: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html (http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html) [1] [2]
SSD disks + rrdcached could handle a pretty big enviroment. But using a ramdisk is still very very much faster.
/Markus
2015-01-12 10:52 GMT+01:00 Louis Bailleul :
Hi Adam ,
Is ramdisk storage reliable enough ? Not that I have no faith in the reliability of our setup, but storing large amount of data in ram isn't a little risky ?
Currently our rrd dir is du -sh rrd/ 21G rrd/
That fit in a 24-32Go ram VM which is something that we can create.
What do you think of the mysql tuning that have been suggested by Eduardo and Alastair ?
Is there any other option than ramdisk storage ?
Best regards, Louis
January 9 2015 5:01 PM, "Adam Armstrong" wrote:
Depending upon your deployment size, I usually recommend ramdisk storage for RRDs.
adam.
------ Original Message ------ From: "Louis Bailleul" To: "Observium Network Observation System" Sent: 1/9/2015 10:51:40 AM Subject: Re: [Observium] Speed up graph rendering
Hi,
Thank you for the pointers.
To be more specific the page generation for an arista switch with 760 ports take 0.567 seconds. But the loading of all graphs (a lot of sensors) take 8m 53s according to firebug.
Generating the ports view page takes 1.421s but loading all the graphs takes 12m 33s and 493 HTTP request (maybe not accurate as firebug says the log limit was reached).
On a force10 switch with 48 ports page generates in 0.207seconds and the page loads in 2.73s. The ports view takes 0.311 seconds to generate and the page finishes loading graphs after 1m8s (205 HTTP requests in total).
So from what I understand the mysql speed might not be in cause but maybe the RRD disk I/O. What can I do to speed it up ? Moving the disk on faster storage will be enough ?
Could it be that apache need tuning to handle that amount of call to graph.php ?
For the record the disk system is an IBM V3700 SAS attached to the ESXi hosts.
Best regards, Louis
January 9 2015 3:11 PM, "Tom Laermans"
wrote:
Hi,
You must note however, that the load time for the page as displayed by Observium is the generation time of the HTML (which is influenced by MySQL speed and CPU, mostly), which includes image tags to load the graphs - but not the loading of the graphs themselves (which is heavily influenced by both CPU and RRD Disk I/O).
Tom
On 01/09/2015 04:08 PM, Pedersen, Sean wrote: It might help if you included a couple of references/examples of the pages that are slow.
To give you something to compare against, we’re running Observium on an ESX VM – 4x Xeon X5690 @ 3.5GHz, 8GB RAM, no idea on storage – but we’re at a pretty consistent 40-50% utilization at all times, using about 2GB of RAM. This is with ~200 devices and 16K ports.
I looked at a Cisco switch stack that has 115 active ports. Graphs > System loaded in .45 seconds, which is only 16 different graphs between four categories. Likewise, the Ports page loaded at .64 seconds with the miniature preview graphs. Clicking on a specific port, it loaded 24 graphs in .49 seconds.
Now…I have some custom port lists, one of which is 36 graphs, which loaded in .53 seconds. However, I have another group that has…a metric shit-tonne of ports and graphs on it – hundreds of them. The page loaded in 1.07 seconds, but it took several seconds more for all of the individual graphs to populate, which I expect considering the sheer number of them present in that page. I don’t know if that’s simply the browser catching up, or if it’s still generating the actual graphs in the background.
From: Louis Bailleul Reply-To: "observium@observium.org (http:///compose?To=observium@observium.org)" Date: Friday, January 9, 2015 at 7:46 AM To: "observium@observium.org (http:///compose?To=observium@observium.org)" Subject: [Observium] Speed up graph rendering
Hi,
I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs.
Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks.
Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN.
Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ?
Are they special requirements/tunnings for the rrd storage or the mysql database ?
Best regards, Louis
FOUNDED IN 2007, IO PROVIDES THE DATA CENTER AS A SERVICE TO BUSINESSES AND GOVERNMENTS AROUND THE WORLD.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
observium mailing list observium@observium.org (http:///compose?To=observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium) [2] [1]
observium mailing list observium@observium.org (http:///compose?To=observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium) [2] [1]
Links:
[1] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium) [2] [2] http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html (http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html) [1]
observium mailing list observium@observium.org (http:///compose?To=observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium) [2]
observium mailing list observium@observium.org (http:///compose?To=observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium) [2]
Links:
[1] http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html (http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html) [2] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
observium mailing list observium@observium.org (http:///compose?To=observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
observium mailing list observium@observium.org (http:///compose?To=observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
_______________________________________________ observium mailing list observium@observium.org (http:///compose?To=observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
_______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
participants (13)
-
Adam Armstrong -
Alastair Chamorro -
Eduardo Schoedler -
emilio brambilla -
Louis Bailleul -
Markus Klock -
Mathieu POUSSIN -
Mike Stupalov -
Pedersen, Sean -
Simon Smith -
Tom Laermans -
Tristan Rhodes -
Сережка Хомяков