Hmm. I don't understand how they can be particularly slow. They're generally small tables.

On 28/07/2016 09:06:42, Lyndon Labuschagne <lyndonl@mobiletorque.co.za> wrote:

The continuing saga

It seem that the "processors-state” integration waits for a long time, most other queries / read / writes are being completed in under 5 seconds, as shown below these are “Waiting” for 40+ seconds, these seems to be linked to JOINS in most cases
Does anyone have an idea on how to speed these up from a tuning perspective?



8603 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2740'
8604 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2554'
8609 observium localhost observium 40 Query query en UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2555'
8610 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2154'
8611 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1880'
8612 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1916'
8627 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1936'
8629 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2161'
8633 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2596'
8634 observium localhost observium 37 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '864'
8637 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '956'
8638 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1073'
8639 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '578'
8640 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '513'
8572 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '312'
8630 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '430'
8646 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1099'
8647 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '573'
8613 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='20824064',`mempool_perc` ='32.73',`mempool_free` ='42799104',`mempool_total` ='63623168' WHERE `mempool_id` = '2359'
8615 observium localhost observium 12 Query query en UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='27287552',`mempool_perc` ='42.8',`mempool_free` ='36466688',`mempool_total` ='63754240' WHERE `mempool_id` = '1740'
8628 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='22896640',`mempool_perc` ='35.93',`mempool_free` ='40824832',`mempool_total` ='63721472' WHERE `mempool_id` = '2273'
8649 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '161'
8650 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '847'
8605 observium localhost observium 9 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1191'
8625 observium localhost observium 7 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1057'
8597 observium localhost observium 6 Query query en UPDATE `sensors-state` set `sensor_value` ='23.8',`sensor_event` ='ok',`sensor_status` ='',`sensor_last_change` ='1469690416',`sensor_polled` ='1469691635' WHERE `sensor_id` = '5102'
8576 observium localhost observium 4 Query query en UPDATE `ports` set `poll_time` ='1469691627',`poll_period` ='703',`ifInOctets` ='88925',`ifInOctets_rate` ='0',`ifOutOctets` ='88925',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate`
7746 observium localhost observium 2 Query query en INSERT INTO `vlans_fdb` (`device_id`,`vlan_id`,`port_id`,`mac_address`,`fdb_status`) VALUES ('701','205','13401','84b59cb47383','learned')
8428 observium localhost observium 2 Query query en UPDATE `ports` set `poll_time` ='1469691592',`poll_period` ='328',`ifInOctets` ='0',`ifInOctets_rate` ='0',`ifOutOctets` ='0',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` ='0',`i

411 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2708','0','1469692947')
421 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2404','0','1469692947')
423 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2119','0','1469692947')
425 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2004','0','1469692947')
426 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2287','0','1469692947')
429 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2888','0','1469692947')
431 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2875','0','1469692947')
432 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2661','0','1469692947')
433 observium localhost observium 24 Query query en INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2759','0','1469692947')
434 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2455','22','1469692947')
435 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2982','0','1469692947')
436 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2120','1','1469692947')
437 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2452','30','1469692947')
439 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1926','0','1469692947')
441 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2447','36','1469692947')
450 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2986','0','1469692947')
451 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2267','0','1469692947')
454 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2937','0','1469692947')
456 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('3006','0','1469692947')
457 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2916','0','1469692947')
458 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2358','0','1469692947')
459 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2825','0','1469692947')
460 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2332','0','1469692947')
471 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1866','0','1469692947')
478 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2121','0','1469692947')
482 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2549','0','1469692947')

> On 25 Jul 2016, at 11:35, Lyndon Labuschagne wrote:
>
> This is a walk from the Server that is running at a decent poller speed
> It complains about LM-SENSORS-MIB, and yet the MIB exists, and is identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
>
> What I also find interesting although possibly nothing is that the returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04
> The old server which is running fine, is Ubuntu 15.04
>
> Both use snmpd version 5.7.3
>
> snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1
> MIB search path: /opt/observium/mibs
> Cannot find module (LM-SENSORS-MIB): At line 1 in (none)
> SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips
> SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4
> DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62 days, 23:05:46.00
> SNMPv2-MIB::sysContact.0 = STRING: networks
> SNMPv2-MIB::sysName.0 = STRING: domain.name.tld
> SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location
> SNMPv2-MIB::sysServices.0 = INTEGER: 79
> SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day, 3:39:21.00
> SNMPv2-MIB::sysORIndex.1 = INTEGER: 1
> SNMPv2-MIB::sysORIndex.2 = INTEGER: 2
> SNMPv2-MIB::sysORIndex.3 = INTEGER: 3
> SNMPv2-MIB::sysORIndex.4 = INTEGER: 4
> SNMPv2-MIB::sysORIndex.5 = INTEGER: 5
> SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB
> SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036
> SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB
> SNMPv2-MIB::sysORID.4 = OID: SNMPv2-SMI::enterprises.10002.1.1.1.31
> SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112
> SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP entities
> SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11 entities.
> SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe generic objects for network interface sub-layers
> SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe system resources
> SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module
> SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00
> SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00
> SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00
> SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00
> SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
>
> - Server that is showing issues
> snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1
> iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips"
> iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4
> iso.3.6.1.2.1.1.3.0 = Timeticks: (543981700) 62 days, 23:03:37.00
> iso.3.6.1.2.1.1.4.0 = STRING: "networks"
> iso.3.6.1.2.1.1.5.0 = STRING: "domain.name.tld"
> iso.3.6.1.2.1.1.6.0 = STRING: "Some GPS Location"
> iso.3.6.1.2.1.1.7.0 = INTEGER: 79
> iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00
> iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1
> iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2
> iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3
> iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4
> iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5
> iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1
> iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036
> iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31
> iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31
> iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112
> iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities"
> iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities."
> iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers"
> iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources"
> iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module "
> iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00
> iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00
> iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00
> iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00
> iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
>
>
>
>
>> On 25 Jul 2016, at 09:00, Lyndon Labuschagne wrote:
>>
>> May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not
>>
>> found this in debug mode
>>
>> It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.name.tld variable name in this MIB."
>>
>> the device is reachable via SNMP as shown in a portion of the snmpwalk below
>>
>> this is where I am seeing the slowing down during the ifindex scanning as previously posted
>>>> if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test.
>>>> The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
>>
>>
>> SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645']
>> SQL RUNTIME[0.00229788s]
>> SNMP STATUS[FALSE]
>> 6
>> CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
>>
>> CMD EXITCODE[1]
>> CMD RUNTIME[6.0817s]
>> STDOUT[
>>
>> ]
>> STDERR[
>> Error in packet
>> Reason: (noSuchName) There is no such variable name in this MIB.
>> Failed object: ifInMulticastPkts.6
>>
>> Timeout: No Response from udp:dns.name.tld:161.
>> ]
>> SNMP ERROR[#1002 - Request timeout]
>>
>> SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';]
>> SQL RUNTIME[0.00058079s]
>>
>> SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650']
>> SQL RUNTIME[0.00175095s]
>> SNMP STATUS[FALSE]
>> 7
>> CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
>>
>> CMD EXITCODE[1]
>> CMD RUNTIME[6.0846s]
>> STDOUT[
>>
>> ]
>> STDERR[
>> Error in packet
>> Reason: (noSuchName) There is no such variable name in this MIB.
>> Failed object: ifInMulticastPkts.7
>>
>> Timeout: No Response from udp:dns.name.tld:161.
>> ]
>> SNMP ERROR[#1002 - Request timeout]
>>
>> SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';]
>> SQL RUNTIME[0.00056005s]
>>
>> SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656']
>> SQL RUNTIME[0.00169611s]
>> SNMP STATUS[FALSE]
>> 8
>> CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
>>
>> CMD EXITCODE[1]
>> CMD RUNTIME[6.0792s]
>> STDOUT[
>>
>> ]
>> STDERR[
>> Error in packet
>> Reason: (noSuchName) There is no such variable name in this MIB.
>> Failed object: ifInMulticastPkts.8
>>
>> Timeout: No Response from udp:dns.name.tld:161.
>> ]
>> SNMP ERROR[#1002 - Request timeout]
>>
>> SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';]
>> SQL RUNTIME[0.00071502s]
>>
>> SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662']
>> SQL RUNTIME[0.00218582s]
>> SNMP STATUS[FALSE]
>> 9
>> CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
>>
>> CMD EXITCODE[1]
>> CMD RUNTIME[6.0803s]
>> STDOUT[
>>
>>
>> This is a manual snmpwalk
>>
>> root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld
>> iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips"
>> iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4
>> iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00
>> iso.3.6.1.2.1.1.4.0 = STRING: "networks"
>> iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld"
>> iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords"
>> iso.3.6.1.2.1.1.7.0 = INTEGER: 79
>> iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00
>> iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1
>> iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2
>> iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3
>> iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4
>> iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5
>> iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1
>> iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036
>> iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31
>> iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31
>> iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112
>> iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities"
>> iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities."
>> iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers"
>> iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources"
>> iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module "
>> iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00
>> iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00
>> iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00
>> iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00
>> iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
>> iso.3.6.1.2.1.2.1.0 = INTEGER: 9
>> iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1
>> iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2
>> iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3
>> iso.3.6.1.2.1.2.2.1.1.4 = INTEGER: 4
>> iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5
>> iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6
>> iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7
>> iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8
>> iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9
>> iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo"
>> iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0"
>> iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1"
>> iso.3.6.1.2.1.2.2.1.2.4 = STRING: "wifi0"
>> iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0"
>> iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36"
>> iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36"
>> iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0"
>> iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1"
>> iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24
>> iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6
>> iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6
>> iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6
>> iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6
>> iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6
>> iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6
>> iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6
>> iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6
>> iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436
>> iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020
>> iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500
>> iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286
>> iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020
>> iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020
>> iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020
>> iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020
>> iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020
>> iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0
>> iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000
>> iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0
>> iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0
>> iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0
>> iso.3.6.1.2.1.2.2.1.5.6 = Gauge32: 100000000
>> iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0
>>
>>
>>
>>
>>
>>> On 25 Jul 2016, at 07:24, Lyndon Labuschagne wrote:
>>>
>>> Im going to fire up the server again today and test more, will let you know,
>>>
>>> if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
>>>
>>>
>>>> On 25 Jul 2016, at 00:30, Adam Armstrong wrote:
>>>>
>>>> If do you have sufficient buffers and caching for innodb in mysql?
>>>>
>>>> We have some /very/ large installs which saw little impact from the changes.
>>>>
>>>> Adam.
>>>>
>>>> Sent from BlueMail
>>>>
>>>> On 24 Jul 2016, at 20:55, Lyndon Labuschagne wrote:
>>>> I have been performance tuning all day with little to no affect
>>>>
>>>>
>>>> its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
>>>>
>>>>
>>>>
>>>> On 24 Jul 2016, at 21:47, Adam Armstrong wrote:
>>>>
>>>> We've changed the way the ports table works, removing memory tables.
>>>>
>>>> It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
>>>>
>>>> Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
>>>>
>>>> Perhaps you could investigate mysql performance tuning? :)
>>>>
>>>> Adam.
>>>>
>>>> Sent from BlueMail
>>>>
>>>>
>>>> On 24 Jul 2016, at 19:58, Lyndon Labuschagne wrote:
>>>> i went back
>>>> to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device,
>>>>
>>>> until I upgraded the db by using discovery -u
>>>>
>>>> back in the same boat again
>>>>
>>>> On 24 Jul 2016, at 09:56, Lyndon Labuschagne wrote:
>>>>
>>>> Ok if I poll a router / switch / server etc it seems to be ok,
>>>> a full scan on a 7606 takes around 45 seconds,
>>>>
>>>>
>>>> if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test.
>>>> The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
>>>>
>>>> o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts
>>>> ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts
>>>> ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
>>>>
>>>>
>>>>
>>>> The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
>>>>
>>>> On 24 Jul 2016, at 09:16, Lyndon Labuschagne wrote:
>>>>
>>>> I know its bad form to reply to your own email but here is some more information re drive performance
>>>>
>>>>
>>>> root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
>>>>
>>>>
>>>> /dev/mapper/observium--vg-root:
>>>> Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec
>>>> Timing buffered disk reads: 950 MB in 3.01 seconds
>>>> = 315.94 MB/sec
>>>>
>>>> root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output
>>>> 10240+0 records in
>>>> 10240+0 records out
>>>> 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
>>>>
>>>>
>>>> On 24 Jul 2016, at 09:13, Lyndon Labuschagne wrote:
>>>>
>>>> Hi All
>>>>
>>>> I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc
>>>> The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
>>>>
>>>>
>>>> Will all of this my average poller-wrapper time is around 60 to 70 seconds per device
>>>> if I manually run an snmpwalk a single device takes around 6 seconds
>>>>
>>>> I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
>>>>
>>>> I have around 900 devices and the VM server was way
>>>> faster, any idea where to start looking for a bottleneck?
>>>>
>>>>
>>>> Regards
>>>> Lyndon
>>>>
>>>> observium mailing list
>>>> observium@observium.org
>>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>>>>
>>>>
>>>> observium mailing list
>>>> observium@observium.org
>>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>>>>
>>>>
>>>> observium mailing list
>>>> observium@observium.org
>>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>>>>
>>>>
>>>> observium mailing list
>>>> observium@observium.org
>>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>>>>
>>>> observium mailing list
>>>> observium@observium.org
>>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>>>>
>>>>
>>>> observium mailing list
>>>> observium@observium.org
>>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>>>> _______________________________________________
>>>> observium mailing list
>>>> observium@observium.org
>>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>>>
>>> _______________________________________________
>>> observium mailing list
>>> observium@observium.org
>>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>>
>> _______________________________________________
>> observium mailing list
>> observium@observium.org
>> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
>
> _______________________________________________
> observium mailing list
> observium@observium.org
> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium