- observium - lists.observium.org

Speeding up caching and database
by Louis Bailleul 16 Feb '16

16 Feb '16

Hi Team, I have a performance issue with the web interface. My Observium is monitoring 3095 devices from which 168 are switches (Just a bit less than 20000 ports total). My server is an IBM X3650 M4, 32G of ram, 6 cores and a raid 10 of SSD. I am more than happy with the polling performance has it finishing in under 200seconds. But the web interface is quite annoying : I have a minimum Time to first bit of ~2seconds. Once these two seconds have passed the page load almost instantly, graphs included. So the question is, can I speed things up ? And if yes how ? It looks like Observium is spending about a second in the database and says that the Cache took 2 seconds to process (?). Do I have some leverage on these points to make it faster ? Page load time (Just a device page) : Connection Setup TIME Queueing 5.24 ms Stalled 1.26 ms Request sent 0.80 ms Waiting (TTFB) 3.29 s Content Download 26.62 ms Explanation 3.33 s Observium Performance report : Page 2.281s Cache 2.047s MySQL Cell 50/0.0633s Row 35/0.0113s Rows 49/0.7822s Column 8/0.0564s Memory Cached 180MB Page 189MB Peak 229MB Versions: Observium 0.16.1.7535 (27th January 2016) OS Linux 3.10.0-327.el7.x86_64 [amd64] (CentOS 7.2.1511) Apache 2.4.6 (CentOS) PHP/5.4.16 PHP 5.4.16 Python 2.7.5 MySQL 5.5.44-MariaDB-log (extension: mysqli 5.5.44-MariaDB) SNMP NET-SNMP 5.7.2 RRDtool 1.4.8 Stats : Statistics DB size 606MB RRD size 134GB Devices 3095 Ports 19390 IPv4 Addresses 6148 IPv4 Networks 44 IPv6 Addresses 2959 IPv6 Networks 1 Services 0 Applications 0 Processors 61262 Memory 15563 Storage 11742 Disk I/O 97461 HR-MIB 96815 Entity-MIB 7292 Syslog Entries 0 Eventlog Entries 172938 Sensors 3863 Toner Mysqltuner output : -------- Performance Metrics ------------------------------------------------- [--] Up for: 2d 21h 57m 9s (260M q [1K qps], 5K conn, TX: 283B, RX: 45B) [--] Reads / Writes: 59% / 41% [--] Binary logging is disabled [--] Total buffers: 10.5G global + 4.7M per thread (650 max threads) [OK] Maximum reached memory usage: 11.2G (35.94% of installed RAM) [OK] Maximum possible memory usage: 13.5G (43.37% of installed RAM) [OK] Slow queries: 0% (14K/260M) [OK] Highest usage of available connections: 21% (142/650) [OK] Aborted connections: 0.00% (0/5477) [OK] Query cache efficiency: 32.3% (74M cached / 230M selects) [OK] Query cache prunes per day: 0 [OK] Sorts requiring temporary tables: 0% (129 temp sorts / 182K sorts) [OK] Temporary tables created on disk: 0% (201 on disk / 112K total) [OK] Thread cache hit rate: 97% (142 created / 5K connections) [OK] Table cache hit rate: 102% (922 open / 896 opened) [OK] Open file limit used: 5% (52/1K) [OK] Table locks acquired immediately: 97% (244M immediate / 249M locks) -------- MyISAM Metrics ----------------------------------------------------- [!!] Key buffer used: 18.0% (28M used / 157M cache) [OK] Key buffer size / total MyISAM indexes: 150.0M/99.0K [!!] Read Key buffer hit rate: 93.3% (15 cached / 1 reads) -------- InnoDB Metrics ----------------------------------------------------- [--] InnoDB is enabled. [OK] InnoDB buffer pool / data size: 8.0G/579.1M [!!] InnoDB buffer pool instances: 1 [!!] InnoDB Used buffer: 6.67% (34952 used/ 524287 total) [OK] InnoDB Read buffer efficiency: 100.00% (2079385726 hits/ 2079395120 total) [!!] InnoDB Write buffer efficiency: 0.00% (0 hits/ 1 total) [OK] InnoDB log waits: 0.00% (0 waits / 11253445 writes) -------- AriaDB Metrics ----------------------------------------------------- [--] AriaDB is disabled. -------- Replication Metrics ------------------------------------------------- [--] No replication slave(s) for this server. [--] This is a standalone server.. -------- Recommendations ----------------------------------------------------- General recommendations: Run OPTIMIZE TABLE to defragment tables for better performance Remove Anonymous User accounts - there are 2 anonymous accounts. Set up a Password for user with the following SQL statement ( SET PASSWORD FOR 'user'@'SpecificDNSorIp' = PASSWORD('secure_password'); ) Set up a Secure Password for user@host ( SET PASSWORD FOR 'user'@'SpecificDNSorIp' = PASSWORD('secure_password'); ) Restrict Host for user@% to user@SpecificDNSorIp Variables to adjust: innodb_buffer_pool_instances(=8) ------------------------------------------------------------------------------------- Thanks. Best regards, Louis [http://www.pgs.com/mediaFiles/Exclaimer%20graphics/PGS_LOGO_RGB_42x53px.jpg]<http://www.pgs.com> Louis Bailleul Computer Systems Engineer Imaging & Engineering | Imaging Telephone: +44 1932 376000 Direct: +44 1932 376842 VOIP: 74416842 Mobile: +44 7786 525307 Email: louis.bailleul(a)pgs.com<mailto:Louis.Bailleul@pgs.com> A Clearer Image | www.pgs.com<http://www.pgs.com> [http://www.pgs.com/mediaFiles/Exclaimer%20graphics/150731-ColourWave-660x14…] Address: 4 The Heights, Brooklands, Weybridge, Surrey, KT13 0NY, United Kingdom This e-mail, including any attachments and response string, may contain proprietary information which is confidential and may be legally privileged. It is for the intended recipient only. If you are not the intended recipient or transmission error has misdirected this e-mail, please notify the author by return e-mail and delete this message and any attachment immediately. If you are not the intended recipient you must not use, disclose, distribute, forward, copy, print or rely on this e-mail in any way except as permitted by the author.

4 6

Accounting for Power/Current
by chott＠praha1.net 16 Feb '16

16 Feb '16

Hello team, we are looking for solution how we can create accounting for Power/Current like Mettered PDU from APC Thank you Tom [email-header-praha1] Tomas Chott technický ředitel chott(a)praha1.net<mailto:chott@praha1.net> Phone: +420 245 004 005 Mobile: +420 733 123 135 Metropolitní síť Praha 1 z.s.p.o. Jindrisska 18 110 00 Praha 1 Czech Republic www.praha1.net<http://www.praha1.net> [Popis: Facebook]<http://www.facebook.com/praha1net> <http://www.linkedin.com/pub/tomas-chott/49/97a/1aa>

2 1

Sorting on traffic bug?
by Steffie Morris 13 Feb '16

13 Feb '16

Hi, when you are in the ports view of a switch, and you sort on the column traffic. speeds op 900Mbit are sorted before 1.5Gbit. probably because 900 is larger than 1.5 but the unit is different so i guess that needs to be taken into account somehow?

2 1

All ports at each polling showing "interface changed:" event log
by Matt Ladewig 13 Feb '16

13 Feb '16

We are at 7608. During each polling run, every port on our system shows an interface changed event log: https://s3.amazonaws.com/uploads.hipchat.com/131939/956300/RLSqGEGgF09oIZS/… Here is the port poller from one device showing this issue: https://s3.amazonaws.com/uploads.hipchat.com/131939/956300/4ADJVr2DdsmjgXT/… This error stands out: SQL[UPDATE `ports` set `port_label_base` ='Null',`port_label_num` ='0',`ifLastChange` ='2015-05-19 05:42:34' WHERE `port_id` = '3872'] SQL RUNTIME[0.00006795s] ERROR[Error in query: (Unknown column 'port_label_base' in 'field list') 1054] PORT updated rows= Looks like I'm missing some structure from my DB. Please let me know what is needed to sort this out. Thanks, Matt

2 6

Password reset not working
by Damien Burke 13 Feb '16

13 Feb '16

Hello, Password reset on https://www.observium.org/subs/index.php does not appear to be working. My observium username and password is not working on the svn anymore and I cannot login to account management. Is this a known issue?

3 4

Re: [Observium] After SVN update all my devices show down
by Aaron Hite 12 Feb '16

12 Feb '16

Here is an example of the test I ran. root@Observium:/opt/observium# ./discovery.php -h coln01-n7k-infra-01 ___ _ _ / _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___ | | | || '_ \ / __| / _ \| '__|\ \ / /| || | | || '_ ` _ \ | |_| || |_) |\__ \| __/| | \ V / | || |_| || | | | | | \___/ |_.__/ |___/ \___||_| \_/ |_| \__,_||_| |_| |_| Observium Professional 0.16.2.7609 http://www.observium.org ##### Starting discovery run at 2016-02-12 14:09:17 ##### ##### Finished discovery run at 2016-02-12 14:09:17 ##### WARNING: 0 devices discovered. Did you specify a device that does not exist? o Devices Discovered 0 o Discovery Time 0.003 secs o Memory usage 6MB (peak: 6.25MB) o MySQL Usage Cell[0/0s] Row[4/0s] Rows[2/0.001s] Column[0/0s] Update[0/0s] Insert[0/0s] Delete[0/0s] o RRDTool Usage root@Observium:/opt/observium# ping coln01-n7k-infra-01 PING coln01-n7k-infra-01 (216.46.59.23) 56(84) bytes of data. 64 bytes from coln01-n7k-infra-01 (216.46.59.23): icmp_seq=1 ttl=254 time=1.29 ms 64 bytes from coln01-n7k-infra-01 (216.46.59.23): icmp_seq=2 ttl=254 time=0.756 ms ^C --- coln01-n7k-infra-01 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.756/1.026/1.296/0.270 ms root@Observium:/opt/observium# Aaron From: observium <observium-bounces(a)observium.org<mailto:observium-bounces@observium.org>> on behalf of Aaron Hite <ahite(a)cosentry.com<mailto:ahite@cosentry.com>> Reply-To: Observium Network Observation System <observium(a)observium.org<mailto:observium@observium.org>> Date: Friday, February 12, 2016 at 2:06 PM To: Observium Network Observation System <observium(a)observium.org<mailto:observium@observium.org>> Subject: [Observium] After SVN update all my devices show down Guys, need some assistance here. I updated Observium this morning to Observium Professional 0.16.2.7609. Since that upgrade all my devices have shown down. If I go to the command line I am able to ping via DNS all the devices. I did notice that during the svn update it seemed that a lot of the MIB’s were updated. Any guidance on what direction I should be looking to bring my system back online? Thanks, Aaron

2 2

After SVN update all my devices show down
by Aaron Hite 12 Feb '16

12 Feb '16

Guys, need some assistance here. I updated Observium this morning to Observium Professional 0.16.2.7609. Since that upgrade all my devices have shown down. If I go to the command line I am able to ping via DNS all the devices. I did notice that during the svn update it seemed that a lot of the MIB’s were updated. Any guidance on what direction I should be looking to bring my system back online? Thanks, Aaron

1 0

Monitor Network services
by Matt Zillmann 12 Feb '16

12 Feb '16

Hello everyone, I am inquiring on whether it is possible to monitor some key network services on Windows servers using the subscription version of Observium. An example would be: IIS web service (UP/DOWN). Thank you, Matt.

1 0

PoE Cisco Small Business and 3750
by chott＠praha1.net 12 Feb '16

12 Feb '16

Hello team, in Cisco IOS like 3750, Observium shows PSE/PoE Power status but in Cisco SB no, i try add Custom OID and working, can you add support PoE for Cisco SB to default in template? There is OIDs Description OID Name Type Unit Devices POE_Max .1.3.6.1.2.1.105.1.3.1.1.2.1 pethMainPsePower.1 GAUGE W 12 POE_Usage .1.3.6.1.2.1.105.1.3.1.1.4.1 pethMainPseConsumptionPower.1 GAUGE W 12 Tom [email-header-praha1] Tomas Chott technický ředitel chott(a)praha1.net<mailto:chott@praha1.net> Phone: +420 245 004 005 Mobile: +420 733 123 135 Metropolitní síť Praha 1 z.s.p.o. Jindrisska 18 110 00 Praha 1 Czech Republic www.praha1.net<http://www.praha1.net> [Popis: Facebook]<http://www.facebook.com/praha1net> <http://www.linkedin.com/pub/tomas-chott/49/97a/1aa>

2 1

FreeBSD port update / "red" draw errors
by Dirk-Willem van Gulik 12 Feb '16

12 Feb '16

As discussed back in June — http://comments.gmane.org/gmane.network.observium.general/10286 - this issue also affects FreeBSD. Mathieu Arnold contributed a fix (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207057) which, together with a few other small patches & cleanups has been merged into the FreeBSD ports collection a few hours ago. May be an idea to apply this patch (below) to HEAD; I suspect there is a similar issue in generic_multi_data_separated. Dw diff --git a/net-mgmt/observium/files/patch-html_includes_graphs_generic__multi__bits__separated.inc.php b/net-mgmt/observium/files/patch-html_includes_graphs_generic__multi__bits__separated.inc.php new file mode 100644 index 0000000..b6054c8 --- /dev/null +++ b/net-mgmt/observium/files/patch-html_includes_graphs_generic__multi__bits__separated.inc.php @@ -0,0 +1,27 @@ +--- html/includes/graphs/generic_multi_bits_separated.inc.php.orig 2016-01-15 08:37:56 UTC ++++ html/includes/graphs/generic_multi_bits_separated.inc.php +@@ -76,13 +76,13 @@ foreach ($rrd_list as $rrd) + $rrd_options .= " VDEF:totout".$i."=outB".$i.",TOTAL"; + $rrd_options .= " VDEF:tot".$i."=octets".$i.",TOTAL"; + +- if ($i) { $stack="STACK"; } ++ if ($i) { $stack=":STACK"; } + + if ($vars['line_graph']) + { + $rrd_options .= " LINE1.25:inB".$i."#" . $colour_in . ":'" . $descr . "'"; + } else { +- $rrd_options .= " AREA:inB".$i."#" . $colour_in . ":'" . $descr . "':$stack"; ++ $rrd_options .= " AREA:inB".$i."#" . $colour_in . ":'" . $descr . "'$stack"; + } + $rrd_options .= " GPRINT:inB".$i.":LAST:%6.2lf%s$units"; + $rrd_options .= " GPRINT:inB".$i.":AVERAGE:%6.2lf%s$units"; +@@ -97,7 +97,7 @@ foreach ($rrd_list as $rrd) + $rrd_options .= " 'LINE1.25:outB".$i."_neg#" . $colour_out . ":" . $descr_out . "'"; + } else { + $rrd_options .= " 'HRULE:0#" . $colour_out.":".$descr_out."'"; +- $rrd_optionsb .= " 'AREA:outB".$i."_neg#" . $colour_out . "::$stack'"; ++ $rrd_optionsb .= " 'AREA:outB".$i."_neg#" . $colour_out . ":$stack'"; + } + $rrd_options .= " GPRINT:outB".$i.":LAST:%6.2lf%s$units"; + $rrd_options .= " GPRINT:outB".$i.":AVERAGE:%6.2lf%s$units”;

4 3