Have you tried to tune your mysql?

Try mysqltuner.pl, it's on github.

Disable mysql query cache, it have a giant lock and generally slows down things instead of speeds up.

--
Eduardo Schoedler


Em quinta-feira, 28 de julho de 2016, Lyndon Labuschagne <lyndonl@mobiletorque.co.za> escreveu:
Hi
Yes all up to date

Is there any particular things I should be looking at / for, I have totally killed off the DB and it is currently building back up, discovering hosts etc
a possible kick ass mysql config file or other hints


root@observium:/opt/observium# svn update
Updating '.':
At revision 8013.

root@observium:/opt/observium# ./discovery.php -u

  ___   _                              _
 / _ \ | |__   ___   ___  _ __ __   __(_) _   _  _ __ ___
| | | || '_ \ / __| / _ \| '__|\ \ / /| || | | || '_ ` _ \
| |_| || |_) |\__ \|  __/| |    \ V / | || |_| || | | | | |
 \___/ |_.__/ |___/ \___||_|     \_/  |_| \__,_||_| |_| |_|
                         Observium Professional 0.16.7.8003
                                   http://www.observium.org

-- Database is up to date.


> On 28 Jul 2016, at 14:39, Adam Armstrong <adama@memetic.org> wrote:
>
> Hmm. I don't understand how they can be particularly slow. They're generally small tables.
>
> Are you sure that's running the latest version? The ports table UPDATE seems fairly short.
>
> adam.
>> On 28/07/2016 09:06:42, Lyndon Labuschagne <lyndonl@mobiletorque.co.za> wrote:
>>
>> The continuing saga
>>
>> It seem that the "processors-state” integration waits for a long time, most other queries / read / writes are being completed in under 5 seconds, as shown below these are “Waiting” for 40+ seconds, these seems to be linked to JOINS in most cases
>> Does anyone have an idea on how to speed these up from a tuning perspective?
>>
>>
>>
>> 8603 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2740'
>> 8604 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2554'
>> 8609 observium localhost observium 40 Query query en UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2555'
>> 8610 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2154'
>> 8611 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1880'
>> 8612 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1916'
>> 8627 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1936'
>> 8629 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2161'
>> 8633 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2596'
>> 8634 observium localhost observium 37 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '864'
>> 8637 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '956'
>> 8638 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1073'
>> 8639 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '578'
>> 8640 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '513'
>> 8572 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '312'
>> 8630 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '430'
>> 8646 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1099'
>> 8647 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '573'
>> 8613 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='20824064',`mempool_perc` ='32.73',`mempool_free` ='42799104',`mempool_total` ='63623168' WHERE `mempool_id` = '2359'
>> 8615 observium localhost observium 12 Query query en UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='27287552',`mempool_perc` ='42.8',`mempool_free` ='36466688',`mempool_total` ='63754240' WHERE `mempool_id` = '1740'
>> 8628 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='22896640',`mempool_perc` ='35.93',`mempool_free` ='40824832',`mempool_total` ='63721472' WHERE `mempool_id` = '2273'
>> 8649 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '161'
>> 8650 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '847'
>> 8605 observium localhost observium 9 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1191'
>> 8625 observium localhost observium 7 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1057'
>> 8597 observium localhost observium 6 Query query en UPDATE `sensors-state` set `sensor_value` ='23.8',`sensor_event` ='ok',`sensor_status` ='',`sensor_last_change` ='1469690416',`sensor_polled` ='1469691635' WHERE `sensor_id` = '5102'
>> 8576 observium localhost observium 4 Query query en UPDATE `ports` set `poll_time` ='1469691627',`poll_period` ='703',`ifInOctets` ='88925',`ifInOctets_rate` ='0',`ifOutOctets` ='88925',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate`
>> 7746 observium localhost observium 2 Query query en INSERT INTO `vlans_fdb` (`device_id`,`vlan_id`,`port_id`,`mac_address`,`fdb_status`) VALUES ('701','205','13401','84b59cb47383','learned')
>> 8428 observium localhost observium 2 Query query en UPDATE `ports` set `poll_time` ='1469691592',`poll_period` ='328',`ifInOctets` ='0',`ifInOctets_rate` ='0',`ifOutOctets` ='0',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` ='0',`i
>>
>> 411 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2708','0','1469692947')
>> 421 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2404','0','1469692947')
>> 423 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2119','0','1469692947')
>> 425 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2004','0','1469692947')
>> 426 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2287','0','1469692947')
>> 429 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2888','0','1469692947')
>> 431 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2875','0','1469692947')
>> 432 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2661','0','1469692947')
>> 433 observium localhost observium 24 Query query en INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2759','0','1469692947')
>> 434 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2455','22','1469692947')
>> 435 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2982','0','1469692947')
>> 436 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2120','1','1469692947')
>> 437 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2452','30','1469692947')
>> 439 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1926','0','1469692947')
>> 441 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2447','36','1469692947')
>> 450 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2986','0','1469692947')
>> 451 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2267','0','1469692947')
>> 454 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2937','0','1469692947')
>> 456 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('3006','0','1469692947')
>> 457 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2916','0','1469692947')
>> 458 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2358','0','1469692947')
>> 459 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2825','0','1469692947')
>> 460 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2332','0','1469692947')
>> 471 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1866','0','1469692947')
>> 478 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2121','0','1469692947')
>> 482 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2549','0','1469692947')
>>
>> > On 25 Jul 2016, at 11:35, Lyndon Labuschagne wrote:
>> >
>> > This is a walk from the Server that is running at a decent poller speed
>> > It complains about LM-SENSORS-MIB, and yet the MIB exists, and is identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
>> >
>> > What I also find interesting although possibly nothing is that the returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04
>> > The old server which is running fine, is Ubuntu 15.04
>> >
>> > Both use snmpd version 5.7.3
>> >
>> > snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1
>> > MIB search path: /opt/observium/mibs
>> > Cannot find module (LM-SENSORS-MIB): At line 1 in (none)
>> > SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips
>> > SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4
>> > DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62 days, 23:05:46.00
>> > SNMPv2-MIB::sysContact.0 = STRING: networks
>> > SNMPv2-MIB::sysName.0 = STRING: domain.name.tld
>> > SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location
>> > SNMPv2-MIB::sysServices.0 = INTEGER: 79
>> > SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day, 3:39:21.00
>> > SNMPv2-MIB::sysORIndex.1 = INTEGER: 1
>> > SNMPv2-MIB::sysORIndex.2 = INTEGER: 2
>> > SNMPv2-MIB::sysORIndex.3 = INTEGER: 3
>> > SNMPv2-MIB::sysORIndex.4 = INTEGER: 4
>> > SNMPv2-MIB::sysORIndex.5 = INTEGER: 5
>> > SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB
>> > SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036
>> > SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB
>> > SNMPv2-MIB::sysORID.4 = OID: SNMPv2-SMI::enterprises.10002.1.1.1.31
>> > SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112
>> > SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP entities
>> > SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11 entities.
>> > SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe generic objects for network interface sub-layers
>> > SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe system resources
>> > SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module
>> > SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00
>> > SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00
>> > SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00
>> > SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00
>> > SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
>> >
>> > - Server that is showing issues
>> > snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1
>> > iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips"
>> > iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4
>> > iso.3.6.1.2.1.1.3.0 = Timeticks: (543981700) 62 days, 23:03:37.00
>> > iso.3.6.1.2.1.1.4.0 = STRING: "networks"
>> > iso.3.6.1.2.1.1.5.0 = STRING: "domain.name.tld"
>> > iso.3.6.1.2.1.1.6.0 = STRING: "Some GPS Location"
>> > iso.3.6.1.2.1.1.7.0 = INTEGER: 79
>> > iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00
>> > iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1
>> > iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2
>> > iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3
>> > iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4
>> > iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5
>> > iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1
>> > iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036
>> > iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31
>> > iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31
>> > iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112
>> > iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities"
>> > iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities."
>> > iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers"
>> > iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources"
>> > iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module "
>> > iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00
>> > iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00
>> > iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00
>> > iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00
>> > iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
>> >
>> >
>> >
>> >
>> >> On 25 Jul 2016, at 09:00, Lyndon Labuschagne wrote:
>> >>
>> >> May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not
>> >>
>> >> found this in debug mode
>> >>
>> >> It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.name.tld variable name in this MIB."
>> >>
>> >> the device is reachable via SNMP as shown in a portion of the snmpwalk below
>> >>
>> >> this is where I am seeing the slowing down during the ifindex scanning as previously posted
>> >>>> if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test.
>> >>>> The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
>> >>
>> >>
>> >> SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645']
>> >> SQL RUNTIME[0.00229788s]
>> >> SNMP STATUS[FALSE]
>> >> 6
>> >> CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
>> >>
>> >> CMD EXITCODE[1]
>> >> CMD RUNTIME[6.0817s]
>> >> STDOUT[
>> >>
>> >> ]
>> >> STDERR[
>> >> Error in packet
>> >> Reason: (noSuchName) There is no such variable name in this MIB.
>> >> Failed object: ifInMulticastPkts.6
>> >>
>> >> Timeout: No Response from udp:dns.name.tld:161.
>> >> ]
>> >> SNMP ERROR[#1002 - Request timeout]
>> >>
>> >> SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';]
>> >> SQL RUNTIME[0.00058079s]
>> >>
>> >> SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650']
>> >> SQL RUNTIME[0.00175095s]
>> >> SNMP STATUS[FALSE]
>> >> 7
>> >> CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
>> >>
>> >> CMD EXITCODE[1]
>> >> CMD RUNTIME[6.0846s]
>> >> STDOUT[
>> >>
>> >> ]
>> >> STDERR[
>> >> Error in packet
>> >> Reason: (noSuchName) There is no such variable name in this MIB.
>> >> Failed object: ifInMulticastPkts.7
>> >>
>> >> Timeout: No Response from udp:dns.name.tld:161.
>> >> ]
>> >> SNMP ERROR[#1002 - Request timeout]
>> >>
>> >> SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';]
>> >> SQL RUNTIME[0.00056005s]
>> >>
>> >> SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656']
>> >> SQL RUNTIME[0.00169611s]
>> >> SNMP STATUS[FALSE]
>> >> 8
>> >> CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
>> >>
>> >> CMD EXITCODE[1]
>> >> CMD RUNTIME[6.0792s]
>> >> STDOUT[
>> >>
>> >> ]
>> >> STDERR[
>> >> Error in packet
>> >> Reason: (noSuchName) There is no such variable name in this MIB.
>> >> Failed object: ifInMulticastPkts.8
>> >>
>> >> Timeout: No Response from udp:dns.name.tld:161.
>> >> ]
>> >> SNMP ERROR[#1002 - Request timeout]
>> >>
>> >> SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';]
>> >> SQL RUNTIME[0.00071502s]
>> >>
>> >> SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662']
>> >> SQL RUNTIME[0.00218582s]
>> >> SNMP STATUS[FALSE]
>> >> 9
>> >> CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
>> >>
>> >> CMD EXITCODE[1]
>> >> CMD RUNTIME[6.0803s]
>> >> STDOUT[
>> >>
>> >>
>> >> This is a manual snmpwalk
>> >>
>> >> root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld
>> >> iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips"
>> >> iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4
>> >> iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00
>> >> iso.3.6.1.2.1.1.4.0 = STRING: "networks"
>> >> iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld"
>> >> iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords"
>> >> iso.3.6.1.2.1.1.7.0 = INTEGER: 79
>> >> iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00
>> >> iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1
>> >> iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2
>> >> iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3
>> >> iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4
>> >> iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5
>> >> iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1
>> >> iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036
>> >> iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31
>> >> iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31
>> >> iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112
>> >> iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities"
>> >> iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities."
>> >> iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers"
>> >> iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources"
>> >> iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module "
>> >> iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00
>> >> iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00
>> >> iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00
>> >> iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00
>> >> iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
>> >> iso.3.6.1.2.1.2.1.0 = INTEGER: 9
>> >> iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1
>> >> iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2
>> >> iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3
>> >> iso.3.6.1.2.1.2.2.1.1.4 = INTEGER: 4
>> >> iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5
>> >> iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6
>> >> iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7
>> >> iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8
>> >> iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9
>> >> iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo"
>> >> iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0"
>> >> iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1"
>> >> iso.3.6.1.2.1.2.2.1.2.4 = STRING: "wifi0"
>> >> iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0"
>> >> iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36"
>> >> iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36"
>> >> iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0"
>> >> iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1"
>> >> iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24
>> >> iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6
>> >> iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6
>> >> iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6
>> >> iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6
>> >> iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6
>> >> iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6
>> >> iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6
>> >> iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6
>> >> iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436
>> >> iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020
>> >> iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500
>> >> iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286
>> >> iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020
>> >> iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020
>> >> iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020
>> >> iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020
>> >> iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020
>> >> iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0
>> >> iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000
>> >> iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0
>> >> iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0
>> >> iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0
>> >> iso.3.6.1.2.1.2.2.1.5.6 = Gauge32: 100000000
>> >> iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>> On 25 Jul 2016, at 07:24, Lyndon Labuschagne wrote:
>> >>>
>> >>> Im going to fire up the server again today and test more, will let you know,
>> >>>
>> >>> if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
>> >>>
>> >>>
>> >>>> On 25 Jul 2016, at 00:30, Adam Armstrong wrote:
>> >>>>
>> >>>> If do you have sufficient buffers and caching for innodb in mysql?
>> >>>>
>> >>>> We have some /very/ large installs which saw little impact from the changes.
>> >>>>
>> >>>> Adam.
>> >>>>
>> >>>> Sent from BlueMail
>> >>>>
>> >>>> On 24 Jul 2016, at 20:55, Lyndon Labuschagne wrote:
>> >>>> I have been performance tuning all day with little to no affect
>> >>>>
>> >>>>
>> >>>> its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
>> >>>>
>> >>>>
>> >>>>
>> >>>> On 24 Jul 2016, at 21:47, Adam Armstrong wrote:
>> >>>>
>> >>>> We've changed the way the ports table works, removing memory tables.
>> >>>>
>> >>>> It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
>> >>>>
>> >>>> Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
>> >>>>
>> >>>> Perhaps you could investigate mysql performance tuning? :)
>> >>>>
>> >>>> Adam.
>> >>>>
>> >>>> Sent from BlueMail
>> >>>>
>> >>>>
>> >>>> On 24 Jul 2016, at 19:58, Lyndon Labuschagne wrote:
>> >>>> i went back
>> >>>> to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device,
>> >>>>
>> >>>> until I upgraded the db by using discovery -u
>> >>>>
>> >>>> back in the same boat again
>> >>>>
>> >>>> On 24 Jul 2016, at 09:56, Lyndon Labuschagne wrote:
>> >>>>
>> >>>> Ok if I poll a router / switch / server etc it seems to be ok,
>> >>>> a full scan on a 7606 takes around 45 seconds,
>> >>>>
>> >>>>
>> >>>> if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test.
>> >>>> The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
>> >>>>
>> >>>> o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts
>> >>>> ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts
>> >>>> ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
>> >>>>
>> >>>>
>> >>>>
>> >>>> The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
>> >>>>
>> >>>> On 24 Jul 2016, at 09:16, Lyndon Labuschagne wrote:
>> >>>>
>> >>>> I know its bad form to reply to your own email but here is some more information re drive performance
>> >>>>
>> >>>>
>> >>>> root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
>> >>>>
>> >>>>
>> >>>> /dev/mapper/observium--vg-root:
>> >>>> Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec
>> >>>> Timing buffered disk reads: 950 MB in 3.01 seconds
>> >>>> = 315.94 MB/sec
>> >>>>
>> >>>> root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output
>> >>>> 10240+0 records in
>> >>>> 10240+0 records out
>> >>>> 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
>> >>>>
>> >>>>
>> >>>> On 24 Jul 2016, at 09:13, Lyndon Labuschagne wrote:
>> >>>>
>> >>>> Hi All
>> >>>>
>> >>>> I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc
>> >>>> The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
>> >>>>
>> >>>>
>> >>>> Will all of this my average poller-wrapper time is around 60 to 70 seconds per device
>> >>>> if I manually run an snmpwalk a single device takes around 6 seconds
>> >>>>
>> >>>> I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
>> >>>>
>> >>>> I have around 900 devices and the VM server was way
>> >>>> faster, any idea where to start looking for a bottleneck?
>> >>>>
>> >>>>
>> >>>> Regards
>> >>>> Lyndon
>> >>>>
>> >>>> observium mailing list
>> >>>> observium@observium.org
>> >>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>> >>>>
>> >>>>
>> >>>> observium mailing list
>> >>>> observium@observium.org
>> >>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>> >>>>
>> >>>>
>> >>>> observium mailing list
>> >>>> observium@observium.org
>> >>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>> >>>>
>> >>>>
>> >>>> observium mailing list
>> >>>> observium@observium.org
>> >>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>> >>>>
>> >>>> observium mailing list
>> >>>> observium@observium.org
>> >>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>> >>>>
>> >>>>
>> >>>> observium mailing list
>> >>>> observium@observium.org
>> >>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>> >>>> _______________________________________________
>> >>>> observium mailing list
>> >>>> observium@observium.org
>> >>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>> >>>
>> >>> _______________________________________________
>> >>> observium mailing list
>> >>> observium@observium.org
>> >>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>> >>
>> >> _______________________________________________
>> >> observium mailing list
>> >> observium@observium.org
>> >> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>> >
>> > _______________________________________________
>> > observium mailing list
>> > observium@observium.org
>> > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>>
>> _______________________________________________
>> observium mailing list
>> observium@observium.org
>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
> _______________________________________________
> observium mailing list
> observium@observium.org
> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium


--
Eduardo Schoedler