May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not 

found this in debug mode

It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.name.tld variable name in this MIB."

the device is reachable via SNMP as shown in a portion of the snmpwalk below

this is where I am seeing the slowing down during the ifindex scanning as previously posted 
  if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. 
  The issue pops up when doing the ifindex -  ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds) 


SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645']
SQL RUNTIME[0.00229788s]
SNMP STATUS[FALSE]
6 
CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]

CMD EXITCODE[1]
CMD RUNTIME[6.0817s]
STDOUT[

]
STDERR[
Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: ifInMulticastPkts.6

Timeout: No Response from udp:dns.name.tld:161.
]
SNMP ERROR[#1002 - Request timeout]

SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';]
SQL RUNTIME[0.00058079s]

SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650']
SQL RUNTIME[0.00175095s]
SNMP STATUS[FALSE]
7 
CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]

CMD EXITCODE[1]
CMD RUNTIME[6.0846s]
STDOUT[

]
STDERR[
Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: ifInMulticastPkts.7

Timeout: No Response from udp:dns.name.tld:161.
]
SNMP ERROR[#1002 - Request timeout]

SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';]
SQL RUNTIME[0.00056005s]

SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656']
SQL RUNTIME[0.00169611s]
SNMP STATUS[FALSE]
8 
CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]

CMD EXITCODE[1]
CMD RUNTIME[6.0792s]
STDOUT[

]
STDERR[
Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: ifInMulticastPkts.8

Timeout: No Response from udp:dns.name.tld:161.
]
SNMP ERROR[#1002 - Request timeout]

SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';]
SQL RUNTIME[0.00071502s]

SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662']
SQL RUNTIME[0.00218582s]
SNMP STATUS[FALSE]
9 
CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]

CMD EXITCODE[1]
CMD RUNTIME[6.0803s]
STDOUT[


This is a manual snmpwalk 

root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld
iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips"
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4
iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00
iso.3.6.1.2.1.1.4.0 = STRING: "networks"
iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld"
iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords"
iso.3.6.1.2.1.1.7.0 = INTEGER: 79
iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00
iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1
iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2
iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3
iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4
iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5
iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1
iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036
iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31
iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31
iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112
iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities"
iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities."
iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers"
iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources"
iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module "
iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00
iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00
iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00
iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00
iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
iso.3.6.1.2.1.2.1.0 = INTEGER: 9
iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1
iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2
iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3
iso.3.6.1.2.1.2.2.1.1.4 = INTEGER: 4
iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5
iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7
iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8
iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9
iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo"
iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0"
iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1"
iso.3.6.1.2.1.2.2.1.2.4 = STRING: "wifi0"
iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0"
iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36"
iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36"
iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0"
iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1"
iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24
iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436
iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020
iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500
iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286
iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020
iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020
iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020
iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020
iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020
iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0
iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000
iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0
iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0
iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0
iso.3.6.1.2.1.2.2.1.5.6 = Gauge32: 100000000
iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0





On 25 Jul 2016, at 07:24, Lyndon Labuschagne <lyndonl@mobiletorque.co.za> wrote:

Im going to fire up the server again today and test more, will let you know, 

if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB


On 25 Jul 2016, at 00:30, Adam Armstrong <adama@memetic.org> wrote:

If do you have sufficient buffers and caching for innodb in mysql?

We have some /very/ large installs which saw little impact from the changes.

Adam.

Sent from BlueMail

On 24 Jul 2016, at 20:55, Lyndon Labuschagne <lyndonl@mobiletorque.co.za> wrote:
I have been performance tuning all day with little to no affect


its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM



 On 24 Jul 2016, at 21:47, Adam Armstrong <adama@memetic.org> wrote:
 
 We've changed the way the ports table works, removing memory tables.
 
 It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
 
 Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
 
 Perhaps you could investigate mysql performance tuning? :)
 
 Adam.
 
 Sent from BlueMail
 
 
 On 24 Jul 2016, at 19:58, Lyndon Labuschagne <lyndonl@mobiletorque.co.za> wrote:
 i went back
to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device, 

 until I upgraded the db by using discovery -u
 
 back in the same boat again 
 
  On 24 Jul 2016, at 09:56, Lyndon Labuschagne <lyndonl@mobiletorque.co.za> wrote:
  
  Ok if I poll a router / switch / server etc it seems to be ok,
  a full scan on a 7606 takes around 45 seconds, 
  
  
  if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. 
  The issue pops up when doing the ifindex -  ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds) 
  
  o Caching Oids         ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts
 ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts
ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus 

 
  
  The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version 
  
  On 24 Jul 2016, at 09:16, Lyndon Labuschagne <lyndonl@mobiletorque.co.za> wrote:
  
  I know its bad form to reply to your own email but here is some more information re drive performance 
  
 
 root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
 
  
  /dev/mapper/observium--vg-root:
  Timing cached reads:   14428 MB in  2.00 seconds = 7218.63 MB/sec
  Timing buffered disk reads: 950 MB in  3.01 seconds
= 315.94 MB/sec

  root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output
  10240+0 records in
  10240+0 records out
  83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
  
  
  On 24 Jul 2016, at 09:13, Lyndon Labuschagne <lyndonl@mobiletorque.co.za> wrote:
  
  Hi All
  
  I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc 
  The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
 
 
  Will all of this my average poller-wrapper time is around 60 to 70 seconds per device 
  if I manually run an snmpwalk a single device takes around 6 seconds 
  
  I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
  
  I have around 900 devices and the VM server was way
faster, any idea where to start looking for a bottleneck?

  
  Regards
  Lyndon
 
  observium mailing list
  observium@observium.org
  http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
  
 
  observium mailing list
  observium@observium.org
  http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
  
 
  observium mailing list
  observium@observium.org
  http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
 
 
 observium mailing list
 observium@observium.org
 http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

 observium mailing list
 observium@observium.org
 http://postman.memetic.org/cgi-bin/mailman/listinfo/observium


observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium