Slow poller speeds after migration
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device, until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, 19:58, at 19:58, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device, until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne
lyndonl@mobiletorque.co.za wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be
slow, even when the radio is directly connected to the hosting switch as a test.
The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9
(that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName
ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly
older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne
lyndonl@mobiletorque.co.za wrote:
I know its bad form to reply to your own email but here is some more
information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt
/dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output
bs=8k count=10k; rm -f /tmp/output
10240+0 records in 10240+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne
lyndonl@mobiletorque.co.za wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was
taking a bit of a pounding, graph dropouts etc
The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of
RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70
seconds per device
if I manually run an snmpwalk a single device takes around 6
seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing
much of a difference either way
I have around 900 devices and the VM server was way faster, any
idea where to start looking for a bottleneck?
Regards Lyndon _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong adama@memetic.org wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote: i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device, until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
If do you have sufficient buffers and caching for innodb in mysql?
We have some /very/ large installs which saw little impact from the changes.
Adam.
Sent from BlueMail
On 24 Jul 2016, 20:55, at 20:55, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong adama@memetic.org wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now
writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install
or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne
lyndonl@mobiletorque.co.za wrote:
i went back to use the old database server on the VM and all was much
better, running pollers at 10 to 15 seconds per device,
until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne
lyndonl@mobiletorque.co.za wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be
slow, even when the radio is directly connected to the hosting switch as a test.
The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9
(that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName
ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts
ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts
ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a
slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne
lyndonl@mobiletorque.co.za wrote:
I know its bad form to reply to your own email but here is some more
information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt
/dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output
bs=8k count=10k; rm -f /tmp/output
10240+0 records in 10240+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne
lyndonl@mobiletorque.co.za wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was
taking a bit of a pounding, graph dropouts etc
The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of
RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70
seconds per device
if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing
much of a difference either way
I have around 900 devices and the VM server was way faster, any idea
where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Im going to fire up the server again today and test more, will let you know,
if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
On 25 Jul 2016, at 00:30, Adam Armstrong <adama@memetic.orgmailto:adama@memetic.org> wrote:
If do you have sufficient buffers and caching for innodb in mysql?
We have some /very/ large installs which saw little impact from the changes.
Adam.
Sent from BlueMailhttp://www.bluemail.me/r
On 24 Jul 2016, at 20:55, Lyndon Labuschagne <lyndonl@mobiletorque.co.zamailto:lyndonl@mobiletorque.co.za> wrote:
I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong <adama@memetic.orgmailto:adama@memetic.org> wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne <lyndonl@mobiletorque.co.zamailto:lyndonl@mobiletorque.co.za> wrote: i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device, until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne <lyndonl@mobiletorque.co.zamailto:lyndonl@mobiletorque.co.za> wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne <lyndonl@mobiletorque.co.zamailto:lyndonl@mobiletorque.co.za> wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne <lyndonl@mobiletorque.co.zamailto:lyndonl@mobiletorque.co.za> wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium ________________________________
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
________________________________
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not
found this in debug mode
It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.name.tld variable name in this MIB."
the device is reachable via SNMP as shown in a portion of the snmpwalk below
this is where I am seeing the slowing down during the ifindex scanning as previously posted if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645'] SQL RUNTIME[0.00229788s] SNMP STATUS[FALSE] 6 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
CMD EXITCODE[1] CMD RUNTIME[6.0817s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.6
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';] SQL RUNTIME[0.00058079s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650'] SQL RUNTIME[0.00175095s] SNMP STATUS[FALSE] 7 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
CMD EXITCODE[1] CMD RUNTIME[6.0846s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.7
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';] SQL RUNTIME[0.00056005s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656'] SQL RUNTIME[0.00169611s] SNMP STATUS[FALSE] 8 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
CMD EXITCODE[1] CMD RUNTIME[6.0792s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.8
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';] SQL RUNTIME[0.00071502s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662'] SQL RUNTIME[0.00218582s] SNMP STATUS[FALSE] 9 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
CMD EXITCODE[1] CMD RUNTIME[6.0803s] STDOUT[
This is a manual snmpwalk
root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.2.1.0 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.2.2.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7 iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8 iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo" iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0" iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1" iso.3.6.1.2.1.2.2.1.2.4 = STRING: "wifi0" iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0" iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36" iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36" iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0" iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1" iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24 iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436 iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500 iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286 iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.6 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0
On 25 Jul 2016, at 07:24, Lyndon Labuschagne <lyndonl@mobiletorque.co.zamailto:lyndonl@mobiletorque.co.za> wrote:
Im going to fire up the server again today and test more, will let you know,
if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
On 25 Jul 2016, at 00:30, Adam Armstrong <adama@memetic.orgmailto:adama@memetic.org> wrote:
If do you have sufficient buffers and caching for innodb in mysql?
We have some /very/ large installs which saw little impact from the changes.
Adam.
Sent from BlueMail
On 24 Jul 2016, at 20:55, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote: I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong adama@memetic.org wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote: i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device,
until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec
root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
This is a walk from the Server that is running at a decent poller speed It complains about LM-SENSORS-MIB, and yet the MIB exists, and is identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
What I also find interesting although possibly nothing is that the returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04 The old server which is running fine, is Ubuntu 15.04
Both use snmpd version 5.7.3
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 MIB search path: /opt/observium/mibs Cannot find module (LM-SENSORS-MIB): At line 1 in (none) SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62 days, 23:05:46.00 SNMPv2-MIB::sysContact.0 = STRING: networks SNMPv2-MIB::sysName.0 = STRING: domain.name.tld SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location SNMPv2-MIB::sysServices.0 = INTEGER: 79 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORIndex.1 = INTEGER: 1 SNMPv2-MIB::sysORIndex.2 = INTEGER: 2 SNMPv2-MIB::sysORIndex.3 = INTEGER: 3 SNMPv2-MIB::sysORIndex.4 = INTEGER: 4 SNMPv2-MIB::sysORIndex.5 = INTEGER: 5 SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036 SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB SNMPv2-MIB::sysORID.4 = OID: SNMPv2-SMI::enterprises.10002.1.1.1.31 SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112 SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP entities SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11 entities. SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe generic objects for network interface sub-layers SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe system resources SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
- Server that is showing issues snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543981700) 62 days, 23:03:37.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "domain.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: "Some GPS Location" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
On 25 Jul 2016, at 09:00, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not
found this in debug mode
It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.name.tld variable name in this MIB."
the device is reachable via SNMP as shown in a portion of the snmpwalk below
this is where I am seeing the slowing down during the ifindex scanning as previously posted
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645'] SQL RUNTIME[0.00229788s] SNMP STATUS[FALSE] 6 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
CMD EXITCODE[1] CMD RUNTIME[6.0817s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.6
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';] SQL RUNTIME[0.00058079s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650'] SQL RUNTIME[0.00175095s] SNMP STATUS[FALSE] 7 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
CMD EXITCODE[1] CMD RUNTIME[6.0846s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.7
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';] SQL RUNTIME[0.00056005s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656'] SQL RUNTIME[0.00169611s] SNMP STATUS[FALSE] 8 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
CMD EXITCODE[1] CMD RUNTIME[6.0792s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.8
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';] SQL RUNTIME[0.00071502s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662'] SQL RUNTIME[0.00218582s] SNMP STATUS[FALSE] 9 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
CMD EXITCODE[1] CMD RUNTIME[6.0803s] STDOUT[
This is a manual snmpwalk
root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.2.1.0 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.2.2.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7 iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8 iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo" iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0" iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1" iso.3.6.1.2.1.2.2.1.2.4 = STRING: "wifi0" iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0" iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36" iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36" iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0" iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1" iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24 iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436 iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500 iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286 iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.6 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0
On 25 Jul 2016, at 07:24, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Im going to fire up the server again today and test more, will let you know,
if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
On 25 Jul 2016, at 00:30, Adam Armstrong adama@memetic.org wrote:
If do you have sufficient buffers and caching for innodb in mysql?
We have some /very/ large installs which saw little impact from the changes.
Adam.
Sent from BlueMail
On 24 Jul 2016, at 20:55, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote: I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong adama@memetic.org wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote: i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device,
until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec
root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
The continuing saga
It seem that the "processors-state” integration waits for a long time, most other queries / read / writes are being completed in under 5 seconds, as shown below these are “Waiting” for 40+ seconds, these seems to be linked to JOINS in most cases Does anyone have an idea on how to speed these up from a tuning perspective?
8603 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2740' 8604 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2554' 8609 observium localhost observium 40 Query query en UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2555' 8610 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2154' 8611 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1880' 8612 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1916' 8627 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1936' 8629 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2161' 8633 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2596' 8634 observium localhost observium 37 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '864' 8637 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '956' 8638 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1073' 8639 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '578' 8640 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '513' 8572 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '312' 8630 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '430' 8646 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1099' 8647 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '573' 8613 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='20824064',`mempool_perc` ='32.73',`mempool_free` ='42799104',`mempool_total` ='63623168' WHERE `mempool_id` = '2359' 8615 observium localhost observium 12 Query query en UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='27287552',`mempool_perc` ='42.8',`mempool_free` ='36466688',`mempool_total` ='63754240' WHERE `mempool_id` = '1740' 8628 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='22896640',`mempool_perc` ='35.93',`mempool_free` ='40824832',`mempool_total` ='63721472' WHERE `mempool_id` = '2273' 8649 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '161' 8650 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '847' 8605 observium localhost observium 9 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1191' 8625 observium localhost observium 7 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1057' 8597 observium localhost observium 6 Query query en UPDATE `sensors-state` set `sensor_value` ='23.8',`sensor_event` ='ok',`sensor_status` ='',`sensor_last_change` ='1469690416',`sensor_polled` ='1469691635' WHERE `sensor_id` = '5102' 8576 observium localhost observium 4 Query query en UPDATE `ports` set `poll_time` ='1469691627',`poll_period` ='703',`ifInOctets` ='88925',`ifInOctets_rate` ='0',`ifOutOctets` ='88925',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` 7746 observium localhost observium 2 Query query en INSERT INTO `vlans_fdb` (`device_id`,`vlan_id`,`port_id`,`mac_address`,`fdb_status`) VALUES ('701','205','13401','84b59cb47383','learned') 8428 observium localhost observium 2 Query query en UPDATE `ports` set `poll_time` ='1469691592',`poll_period` ='328',`ifInOctets` ='0',`ifInOctets_rate` ='0',`ifOutOctets` ='0',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` ='0',`i
411 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2708','0','1469692947') 421 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2404','0','1469692947') 423 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2119','0','1469692947') 425 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2004','0','1469692947') 426 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2287','0','1469692947') 429 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2888','0','1469692947') 431 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2875','0','1469692947') 432 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2661','0','1469692947') 433 observium localhost observium 24 Query query en INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2759','0','1469692947') 434 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2455','22','1469692947') 435 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2982','0','1469692947') 436 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2120','1','1469692947') 437 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2452','30','1469692947') 439 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1926','0','1469692947') 441 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2447','36','1469692947') 450 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2986','0','1469692947') 451 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2267','0','1469692947') 454 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2937','0','1469692947') 456 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('3006','0','1469692947') 457 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2916','0','1469692947') 458 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2358','0','1469692947') 459 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2825','0','1469692947') 460 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2332','0','1469692947') 471 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1866','0','1469692947') 478 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2121','0','1469692947') 482 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2549','0','1469692947')
On 25 Jul 2016, at 11:35, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
This is a walk from the Server that is running at a decent poller speed It complains about LM-SENSORS-MIB, and yet the MIB exists, and is identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
What I also find interesting although possibly nothing is that the returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04 The old server which is running fine, is Ubuntu 15.04
Both use snmpd version 5.7.3
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 MIB search path: /opt/observium/mibs Cannot find module (LM-SENSORS-MIB): At line 1 in (none) SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62 days, 23:05:46.00 SNMPv2-MIB::sysContact.0 = STRING: networks SNMPv2-MIB::sysName.0 = STRING: domain.name.tld SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location SNMPv2-MIB::sysServices.0 = INTEGER: 79 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORIndex.1 = INTEGER: 1 SNMPv2-MIB::sysORIndex.2 = INTEGER: 2 SNMPv2-MIB::sysORIndex.3 = INTEGER: 3 SNMPv2-MIB::sysORIndex.4 = INTEGER: 4 SNMPv2-MIB::sysORIndex.5 = INTEGER: 5 SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036 SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB SNMPv2-MIB::sysORID.4 = OID: SNMPv2-SMI::enterprises.10002.1.1.1.31 SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112 SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP entities SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11 entities. SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe generic objects for network interface sub-layers SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe system resources SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
- Server that is showing issues
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543981700) 62 days, 23:03:37.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "domain.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: "Some GPS Location" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
On 25 Jul 2016, at 09:00, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not
found this in debug mode
It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.name.tld variable name in this MIB."
the device is reachable via SNMP as shown in a portion of the snmpwalk below
this is where I am seeing the slowing down during the ifindex scanning as previously posted
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645'] SQL RUNTIME[0.00229788s] SNMP STATUS[FALSE] 6 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
CMD EXITCODE[1] CMD RUNTIME[6.0817s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.6
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';] SQL RUNTIME[0.00058079s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650'] SQL RUNTIME[0.00175095s] SNMP STATUS[FALSE] 7 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
CMD EXITCODE[1] CMD RUNTIME[6.0846s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.7
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';] SQL RUNTIME[0.00056005s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656'] SQL RUNTIME[0.00169611s] SNMP STATUS[FALSE] 8 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
CMD EXITCODE[1] CMD RUNTIME[6.0792s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.8
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';] SQL RUNTIME[0.00071502s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662'] SQL RUNTIME[0.00218582s] SNMP STATUS[FALSE] 9 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
CMD EXITCODE[1] CMD RUNTIME[6.0803s] STDOUT[
This is a manual snmpwalk
root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.2.1.0 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.2.2.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7 iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8 iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo" iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0" iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1" iso.3.6.1.2.1.2.2.1.2.4 = STRING: "wifi0" iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0" iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36" iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36" iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0" iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1" iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24 iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436 iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500 iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286 iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.6 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0
On 25 Jul 2016, at 07:24, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Im going to fire up the server again today and test more, will let you know,
if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
On 25 Jul 2016, at 00:30, Adam Armstrong adama@memetic.org wrote:
If do you have sufficient buffers and caching for innodb in mysql?
We have some /very/ large installs which saw little impact from the changes.
Adam.
Sent from BlueMail
On 24 Jul 2016, at 20:55, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote: I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong adama@memetic.org wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote: i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device,
until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec
root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hmm. I don't understand how they can be particularly slow. They're generally small tables.
Are you sure that's running the latest version? The ports table UPDATE seems fairly short.
adam. On 28/07/2016 09:06:42, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote: The continuing saga
It seem that the "processors-state” integration waits for a long time, most other queries / read / writes are being completed in under 5 seconds, as shown below these are “Waiting” for 40+ seconds, these seems to be linked to JOINS in most cases Does anyone have an idea on how to speed these up from a tuning perspective?
8603 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2740' 8604 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2554' 8609 observium localhost observium 40 Query query en UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2555' 8610 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2154' 8611 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1880' 8612 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1916' 8627 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1936' 8629 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2161' 8633 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2596' 8634 observium localhost observium 37 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '864' 8637 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '956' 8638 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1073' 8639 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '578' 8640 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '513' 8572 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '312' 8630 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '430' 8646 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1099' 8647 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '573' 8613 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='20824064',`mempool_perc` ='32.73',`mempool_free` ='42799104',`mempool_total` ='63623168' WHERE `mempool_id` = '2359' 8615 observium localhost observium 12 Query query en UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='27287552',`mempool_perc` ='42.8',`mempool_free` ='36466688',`mempool_total` ='63754240' WHERE `mempool_id` = '1740' 8628 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='22896640',`mempool_perc` ='35.93',`mempool_free` ='40824832',`mempool_total` ='63721472' WHERE `mempool_id` = '2273' 8649 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '161' 8650 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '847' 8605 observium localhost observium 9 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1191' 8625 observium localhost observium 7 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1057' 8597 observium localhost observium 6 Query query en UPDATE `sensors-state` set `sensor_value` ='23.8',`sensor_event` ='ok',`sensor_status` ='',`sensor_last_change` ='1469690416',`sensor_polled` ='1469691635' WHERE `sensor_id` = '5102' 8576 observium localhost observium 4 Query query en UPDATE `ports` set `poll_time` ='1469691627',`poll_period` ='703',`ifInOctets` ='88925',`ifInOctets_rate` ='0',`ifOutOctets` ='88925',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` 7746 observium localhost observium 2 Query query en INSERT INTO `vlans_fdb` (`device_id`,`vlan_id`,`port_id`,`mac_address`,`fdb_status`) VALUES ('701','205','13401','84b59cb47383','learned') 8428 observium localhost observium 2 Query query en UPDATE `ports` set `poll_time` ='1469691592',`poll_period` ='328',`ifInOctets` ='0',`ifInOctets_rate` ='0',`ifOutOctets` ='0',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` ='0',`i
411 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2708','0','1469692947') 421 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2404','0','1469692947') 423 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2119','0','1469692947') 425 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2004','0','1469692947') 426 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2287','0','1469692947') 429 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2888','0','1469692947') 431 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2875','0','1469692947') 432 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2661','0','1469692947') 433 observium localhost observium 24 Query query en INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2759','0','1469692947') 434 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2455','22','1469692947') 435 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2982','0','1469692947') 436 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2120','1','1469692947') 437 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2452','30','1469692947') 439 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1926','0','1469692947') 441 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2447','36','1469692947') 450 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2986','0','1469692947') 451 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2267','0','1469692947') 454 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2937','0','1469692947') 456 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('3006','0','1469692947') 457 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2916','0','1469692947') 458 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2358','0','1469692947') 459 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2825','0','1469692947') 460 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2332','0','1469692947') 471 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1866','0','1469692947') 478 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2121','0','1469692947') 482 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2549','0','1469692947')
On 25 Jul 2016, at 11:35, Lyndon Labuschagne wrote:
This is a walk from the Server that is running at a decent poller speed It complains about LM-SENSORS-MIB, and yet the MIB exists, and is identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
What I also find interesting although possibly nothing is that the returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04 The old server which is running fine, is Ubuntu 15.04
Both use snmpd version 5.7.3
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 MIB search path: /opt/observium/mibs Cannot find module (LM-SENSORS-MIB): At line 1 in (none) SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62 days, 23:05:46.00 SNMPv2-MIB::sysContact.0 = STRING: networks SNMPv2-MIB::sysName.0 = STRING: domain.name.tld SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location SNMPv2-MIB::sysServices.0 = INTEGER: 79 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORIndex.1 = INTEGER: 1 SNMPv2-MIB::sysORIndex.2 = INTEGER: 2 SNMPv2-MIB::sysORIndex.3 = INTEGER: 3 SNMPv2-MIB::sysORIndex.4 = INTEGER: 4 SNMPv2-MIB::sysORIndex.5 = INTEGER: 5 SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036 SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB SNMPv2-MIB::sysORID.4 = OID: SNMPv2-SMI::enterprises.10002.1.1.1.31 SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112 SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP entities SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11 entities. SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe generic objects for network interface sub-layers SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe system resources SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
- Server that is showing issues
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543981700) 62 days, 23:03:37.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "domain.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: "Some GPS Location" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
On 25 Jul 2016, at 09:00, Lyndon Labuschagne wrote:
May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not
found this in debug mode
It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.name.tld variable name in this MIB."
the device is reachable via SNMP as shown in a portion of the snmpwalk below
this is where I am seeing the slowing down during the ifindex scanning as previously posted
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645'] SQL RUNTIME[0.00229788s] SNMP STATUS[FALSE] 6 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
CMD EXITCODE[1] CMD RUNTIME[6.0817s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.6
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';] SQL RUNTIME[0.00058079s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650'] SQL RUNTIME[0.00175095s] SNMP STATUS[FALSE] 7 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
CMD EXITCODE[1] CMD RUNTIME[6.0846s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.7
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';] SQL RUNTIME[0.00056005s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656'] SQL RUNTIME[0.00169611s] SNMP STATUS[FALSE] 8 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
CMD EXITCODE[1] CMD RUNTIME[6.0792s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.8
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';] SQL RUNTIME[0.00071502s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662'] SQL RUNTIME[0.00218582s] SNMP STATUS[FALSE] 9 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
CMD EXITCODE[1] CMD RUNTIME[6.0803s] STDOUT[
This is a manual snmpwalk
root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.2.1.0 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.2.2.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7 iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8 iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo" iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0" iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1" iso.3.6.1.2.1.2.2.1.2.4 = STRING: "wifi0" iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0" iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36" iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36" iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0" iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1" iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24 iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436 iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500 iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286 iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.6 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0
On 25 Jul 2016, at 07:24, Lyndon Labuschagne wrote:
Im going to fire up the server again today and test more, will let you know,
if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
On 25 Jul 2016, at 00:30, Adam Armstrong wrote:
If do you have sufficient buffers and caching for innodb in mysql?
We have some /very/ large installs which saw little impact from the changes.
Adam.
Sent from BlueMail
On 24 Jul 2016, at 20:55, Lyndon Labuschagne wrote: I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne wrote: i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device,
until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec
root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Yes all up to date
Is there any particular things I should be looking at / for, I have totally killed off the DB and it is currently building back up, discovering hosts etc a possible kick ass mysql config file or other hints
root@observium:/opt/observium# svn update Updating '.': At revision 8013.
root@observium:/opt/observium# ./discovery.php -u
___ _ _ / _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___ | | | || '_ \ / __| / _ | '__|\ \ / /| || | | || '_ ` _ \ | |_| || |_) |__ | __/| | \ V / | || |_| || | | | | | ___/ |_.__/ |___/ ___||_| _/ |_| __,_||_| |_| |_| Observium Professional 0.16.7.8003 http://www.observium.org
-- Database is up to date.
On 28 Jul 2016, at 14:39, Adam Armstrong adama@memetic.org wrote:
Hmm. I don't understand how they can be particularly slow. They're generally small tables.
Are you sure that's running the latest version? The ports table UPDATE seems fairly short.
adam.
On 28/07/2016 09:06:42, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
The continuing saga
It seem that the "processors-state” integration waits for a long time, most other queries / read / writes are being completed in under 5 seconds, as shown below these are “Waiting” for 40+ seconds, these seems to be linked to JOINS in most cases Does anyone have an idea on how to speed these up from a tuning perspective?
8603 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2740' 8604 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2554' 8609 observium localhost observium 40 Query query en UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2555' 8610 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2154' 8611 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1880' 8612 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1916' 8627 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1936' 8629 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2161' 8633 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2596' 8634 observium localhost observium 37 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '864' 8637 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '956' 8638 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1073' 8639 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '578' 8640 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '513' 8572 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '312' 8630 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '430' 8646 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1099' 8647 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '573' 8613 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='20824064',`mempool_perc` ='32.73',`mempool_free` ='42799104',`mempool_total` ='63623168' WHERE `mempool_id` = '2359' 8615 observium localhost observium 12 Query query en UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='27287552',`mempool_perc` ='42.8',`mempool_free` ='36466688',`mempool_total` ='63754240' WHERE `mempool_id` = '1740' 8628 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='22896640',`mempool_perc` ='35.93',`mempool_free` ='40824832',`mempool_total` ='63721472' WHERE `mempool_id` = '2273' 8649 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '161' 8650 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '847' 8605 observium localhost observium 9 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1191' 8625 observium localhost observium 7 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1057' 8597 observium localhost observium 6 Query query en UPDATE `sensors-state` set `sensor_value` ='23.8',`sensor_event` ='ok',`sensor_status` ='',`sensor_last_change` ='1469690416',`sensor_polled` ='1469691635' WHERE `sensor_id` = '5102' 8576 observium localhost observium 4 Query query en UPDATE `ports` set `poll_time` ='1469691627',`poll_period` ='703',`ifInOctets` ='88925',`ifInOctets_rate` ='0',`ifOutOctets` ='88925',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` 7746 observium localhost observium 2 Query query en INSERT INTO `vlans_fdb` (`device_id`,`vlan_id`,`port_id`,`mac_address`,`fdb_status`) VALUES ('701','205','13401','84b59cb47383','learned') 8428 observium localhost observium 2 Query query en UPDATE `ports` set `poll_time` ='1469691592',`poll_period` ='328',`ifInOctets` ='0',`ifInOctets_rate` ='0',`ifOutOctets` ='0',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` ='0',`i
411 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2708','0','1469692947') 421 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2404','0','1469692947') 423 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2119','0','1469692947') 425 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2004','0','1469692947') 426 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2287','0','1469692947') 429 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2888','0','1469692947') 431 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2875','0','1469692947') 432 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2661','0','1469692947') 433 observium localhost observium 24 Query query en INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2759','0','1469692947') 434 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2455','22','1469692947') 435 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2982','0','1469692947') 436 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2120','1','1469692947') 437 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2452','30','1469692947') 439 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1926','0','1469692947') 441 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2447','36','1469692947') 450 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2986','0','1469692947') 451 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2267','0','1469692947') 454 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2937','0','1469692947') 456 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('3006','0','1469692947') 457 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2916','0','1469692947') 458 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2358','0','1469692947') 459 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2825','0','1469692947') 460 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2332','0','1469692947') 471 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1866','0','1469692947') 478 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2121','0','1469692947') 482 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2549','0','1469692947')
On 25 Jul 2016, at 11:35, Lyndon Labuschagne wrote:
This is a walk from the Server that is running at a decent poller speed It complains about LM-SENSORS-MIB, and yet the MIB exists, and is identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
What I also find interesting although possibly nothing is that the returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04 The old server which is running fine, is Ubuntu 15.04
Both use snmpd version 5.7.3
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 MIB search path: /opt/observium/mibs Cannot find module (LM-SENSORS-MIB): At line 1 in (none) SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62 days, 23:05:46.00 SNMPv2-MIB::sysContact.0 = STRING: networks SNMPv2-MIB::sysName.0 = STRING: domain.name.tld SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location SNMPv2-MIB::sysServices.0 = INTEGER: 79 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORIndex.1 = INTEGER: 1 SNMPv2-MIB::sysORIndex.2 = INTEGER: 2 SNMPv2-MIB::sysORIndex.3 = INTEGER: 3 SNMPv2-MIB::sysORIndex.4 = INTEGER: 4 SNMPv2-MIB::sysORIndex.5 = INTEGER: 5 SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036 SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB SNMPv2-MIB::sysORID.4 = OID: SNMPv2-SMI::enterprises.10002.1.1.1.31 SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112 SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP entities SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11 entities. SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe generic objects for network interface sub-layers SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe system resources SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
- Server that is showing issues
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543981700) 62 days, 23:03:37.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "domain.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: "Some GPS Location" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
On 25 Jul 2016, at 09:00, Lyndon Labuschagne wrote:
May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not
found this in debug mode
It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.name.tld variable name in this MIB."
the device is reachable via SNMP as shown in a portion of the snmpwalk below
this is where I am seeing the slowing down during the ifindex scanning as previously posted
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645'] SQL RUNTIME[0.00229788s] SNMP STATUS[FALSE] 6 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
CMD EXITCODE[1] CMD RUNTIME[6.0817s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.6
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';] SQL RUNTIME[0.00058079s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650'] SQL RUNTIME[0.00175095s] SNMP STATUS[FALSE] 7 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
CMD EXITCODE[1] CMD RUNTIME[6.0846s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.7
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';] SQL RUNTIME[0.00056005s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656'] SQL RUNTIME[0.00169611s] SNMP STATUS[FALSE] 8 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
CMD EXITCODE[1] CMD RUNTIME[6.0792s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.8
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';] SQL RUNTIME[0.00071502s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662'] SQL RUNTIME[0.00218582s] SNMP STATUS[FALSE] 9 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
CMD EXITCODE[1] CMD RUNTIME[6.0803s] STDOUT[
This is a manual snmpwalk
root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.2.1.0 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.2.2.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7 iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8 iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo" iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0" iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1" iso.3.6.1.2.1.2.2.1.2.4 = STRING: "wifi0" iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0" iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36" iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36" iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0" iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1" iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24 iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436 iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500 iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286 iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.6 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0
On 25 Jul 2016, at 07:24, Lyndon Labuschagne wrote:
Im going to fire up the server again today and test more, will let you know,
if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
On 25 Jul 2016, at 00:30, Adam Armstrong wrote:
If do you have sufficient buffers and caching for innodb in mysql?
We have some /very/ large installs which saw little impact from the changes.
Adam.
Sent from BlueMail
On 24 Jul 2016, at 20:55, Lyndon Labuschagne wrote: I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne wrote: i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device,
until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec
root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Oh. Stable!
It's entirely possibly you don't yet have the changes which have the potential to make things "slow" ;)
Adam.
Sent from BlueMail
On 28 Jul 2016, 07:49, at 07:49, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
Hi Yes all up to date
Is there any particular things I should be looking at / for, I have totally killed off the DB and it is currently building back up, discovering hosts etc a possible kick ass mysql config file or other hints
root@observium:/opt/observium# svn update Updating '.': At revision 8013.
root@observium:/opt/observium# ./discovery.php -u
/ _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___ | | | || '_ \ / __| / _ | '__|\ \ / /| || | | || '_ ` _ \ | |_| || |_) |__ | __/| | \ V / | || |_| || | | | | | ___/ |_.__/ |___/ ___||_| _/ |_| __,_||_| |_| |_| Observium Professional 0.16.7.8003 http://www.observium.org
-- Database is up to date.
On 28 Jul 2016, at 14:39, Adam Armstrong adama@memetic.org wrote:
Hmm. I don't understand how they can be particularly slow. They're
generally small tables.
Are you sure that's running the latest version? The ports table
UPDATE seems fairly short.
adam.
On 28/07/2016 09:06:42, Lyndon Labuschagne
lyndonl@mobiletorque.co.za wrote:
The continuing saga
It seem that the "processors-state” integration waits for a long
time, most other queries / read / writes are being completed in under 5 seconds, as shown below these are “Waiting” for 40+ seconds, these seems to be linked to JOINS in most cases
Does anyone have an idea on how to speed these up from a tuning
perspective?
8603 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2740'
8604 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2554'
8609 observium localhost observium 40 Query query en UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2555'
8610 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2154'
8611 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1880'
8612 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1916'
8627 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1936'
8629 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2161'
8633 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2596'
8634 observium localhost observium 37 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '864'
8637 observium localhost observium 35 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '956'
8638 observium localhost observium 35 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1073'
8639 observium localhost observium 35 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '578'
8640 observium localhost observium 35 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '513'
8572 observium localhost observium 14 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '312'
8630 observium localhost observium 14 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '430'
8646 observium localhost observium 13 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1099'
8647 observium localhost observium 13 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '573'
8613 observium localhost observium 12 Query Waiting UPDATE
`mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='20824064',`mempool_perc` ='32.73',`mempool_free` ='42799104',`mempool_total` ='63623168' WHERE `mempool_id` = '2359'
8615 observium localhost observium 12 Query query en UPDATE
`mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='27287552',`mempool_perc` ='42.8',`mempool_free` ='36466688',`mempool_total` ='63754240' WHERE `mempool_id` = '1740'
8628 observium localhost observium 12 Query Waiting UPDATE
`mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='22896640',`mempool_perc` ='35.93',`mempool_free` ='40824832',`mempool_total` ='63721472' WHERE `mempool_id` = '2273'
8649 observium localhost observium 10 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '161'
8650 observium localhost observium 10 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '847'
8605 observium localhost observium 9 Query Waiting SELECT * FROM
`mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1191'
8625 observium localhost observium 7 Query Waiting SELECT * FROM
`mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1057'
8597 observium localhost observium 6 Query query en UPDATE
`sensors-state` set `sensor_value` ='23.8',`sensor_event` ='ok',`sensor_status` ='',`sensor_last_change` ='1469690416',`sensor_polled` ='1469691635' WHERE `sensor_id` = '5102'
8576 observium localhost observium 4 Query query en UPDATE `ports`
set `poll_time` ='1469691627',`poll_period` ='703',`ifInOctets` ='88925',`ifInOctets_rate` ='0',`ifOutOctets` ='88925',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate`
7746 observium localhost observium 2 Query query en INSERT INTO
`vlans_fdb` (`device_id`,`vlan_id`,`port_id`,`mac_address`,`fdb_status`) VALUES ('701','205','13401','84b59cb47383','learned')
8428 observium localhost observium 2 Query query en UPDATE `ports`
set `poll_time` ='1469691592',`poll_period` ='328',`ifInOctets` ='0',`ifInOctets_rate` ='0',`ifOutOctets` ='0',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` ='0',`i
411 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2708','0','1469692947')
421 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2404','0','1469692947')
423 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2119','0','1469692947')
425 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2004','0','1469692947')
426 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2287','0','1469692947')
429 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2888','0','1469692947')
431 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2875','0','1469692947')
432 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2661','0','1469692947')
433 observium localhost observium 24 Query query en INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2759','0','1469692947')
434 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2455','22','1469692947')
435 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2982','0','1469692947')
436 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2120','1','1469692947')
437 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2452','30','1469692947')
439 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1926','0','1469692947')
441 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2447','36','1469692947')
450 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2986','0','1469692947')
451 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2267','0','1469692947')
454 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2937','0','1469692947')
456 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('3006','0','1469692947')
457 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2916','0','1469692947')
458 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2358','0','1469692947')
459 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2825','0','1469692947')
460 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2332','0','1469692947')
471 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1866','0','1469692947')
478 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2121','0','1469692947')
482 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2549','0','1469692947')
On 25 Jul 2016, at 11:35, Lyndon Labuschagne wrote:
This is a walk from the Server that is running at a decent poller
speed
It complains about LM-SENSORS-MIB, and yet the MIB exists, and is
identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
What I also find interesting although possibly nothing is that the
returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04
The old server which is running fine, is Ubuntu 15.04
Both use snmpd version 5.7.3
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 MIB search path: /opt/observium/mibs Cannot find module (LM-SENSORS-MIB): At line 1 in (none) SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65 #1 Thu Jul 16
11:50:24 EEST 2015 mips
SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62
days, 23:05:46.00
SNMPv2-MIB::sysContact.0 = STRING: networks SNMPv2-MIB::sysName.0 = STRING: domain.name.tld SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location SNMPv2-MIB::sysServices.0 = INTEGER: 79 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day,
3:39:21.00
SNMPv2-MIB::sysORIndex.1 = INTEGER: 1 SNMPv2-MIB::sysORIndex.2 = INTEGER: 2 SNMPv2-MIB::sysORIndex.3 = INTEGER: 3 SNMPv2-MIB::sysORIndex.4 = INTEGER: 4 SNMPv2-MIB::sysORIndex.5 = INTEGER: 5 SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036 SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB SNMPv2-MIB::sysORID.4 = OID:
SNMPv2-SMI::enterprises.10002.1.1.1.31
SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112 SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP
entities
SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11
entities.
SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe
generic objects for network interface sub-layers
SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe
system resources
SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00
SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00
SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00
SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00
SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
- Server that is showing issues
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16
11:50:24 EEST 2015 mips"
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543981700) 62 days, 23:03:37.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "domain.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: "Some GPS Location" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP
entities"
iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11
entities."
iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe
generic objects for network interface sub-layers"
iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe
system resources"
iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
On 25 Jul 2016, at 09:00, Lyndon Labuschagne wrote:
May not betheDB after all most of the SQL RUNTIME stats are
0.000xxxx so its probably not
found this in debug mode
It seems the poller / mib is freaking out about "Reason:
(noSuchName) There is no such dns.name.tld variable name in this MIB."
the device is reachable via SNMP as shown in a portion of the
snmpwalk below
this is where I am seeing the slowing down during the ifindex
scanning as previously posted
> if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems
to be slow, even when the radio is directly connected to the hosting switch as a test.
> The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7
8 9 (that takes 40 to 50 seconds)
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated`
='1469428547' WHERE `error_id` = '6645']
SQL RUNTIME[0.00229788s] SNMP STATUS[FALSE] 6 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M
/opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
CMD EXITCODE[1] CMD RUNTIME[6.0817s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.6
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND
`error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';]
SQL RUNTIME[0.00058079s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated`
='1469428553' WHERE `error_id` = '6650']
SQL RUNTIME[0.00175095s] SNMP STATUS[FALSE] 7 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M
/opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
CMD EXITCODE[1] CMD RUNTIME[6.0846s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.7
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND
`error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';]
SQL RUNTIME[0.00056005s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated`
='1469428559' WHERE `error_id` = '6656']
SQL RUNTIME[0.00169611s] SNMP STATUS[FALSE] 8 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M
/opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
CMD EXITCODE[1] CMD RUNTIME[6.0792s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.8
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND
`error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';]
SQL RUNTIME[0.00071502s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated`
='1469428565' WHERE `error_id` = '6662']
SQL RUNTIME[0.00218582s] SNMP STATUS[FALSE] 9 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M
/opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
CMD EXITCODE[1] CMD RUNTIME[6.0803s] STDOUT[
This is a manual snmpwalk
root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16
11:50:24 EEST 2015 mips"
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00
iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP
entities"
iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11
entities."
iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe
generic objects for network interface sub-layers"
iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe
system resources"
iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module "
iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.2.1.0 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.2.2.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7 iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8 iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo" iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0" iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1" iso.3.6.1.2.1.2.2.1.2.4 = STRING: "wifi0" iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0" iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36" iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36" iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0" iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1" iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24 iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436 iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500 iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286 iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.6 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0
On 25 Jul 2016, at 07:24, Lyndon Labuschagne wrote:
Im going to fire up the server again today and test more, will
let you know,
if there anyway of running the poller and but not writing to the
DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
> On 25 Jul 2016, at 00:30, Adam Armstrong wrote: > > If do you have sufficient buffers and caching for innodb in
mysql?
> > We have some /very/ large installs which saw little impact from
the changes.
> > Adam. > > Sent from BlueMail > > On 24 Jul 2016, at 20:55, Lyndon Labuschagne wrote: > I have been performance tuning all day with little to no affect
> > > its just very strange how a XEN VM out performs bare metal, it
may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
> > > > On 24 Jul 2016, at 21:47, Adam Armstrong wrote: > > We've changed the way the ports table works, removing memory
tables.
> > It's likely that you now need more mysql resources, since were
now writing to an innodb table instead of a memory table.
> > Odd, really. It didn't seem to have a huge effect on my dev
install or some other very large installs.
> > Perhaps you could investigate mysql performance tuning? :) > > Adam. > > Sent from BlueMail > > > On 24 Jul 2016, at 19:58, Lyndon Labuschagne wrote: > i went back > to use the old database server on the VM and all was much
better, running pollers at 10 to 15 seconds per device,
> > until I upgraded the db by using discovery -u > > back in the same boat again > > On 24 Jul 2016, at 09:56, Lyndon Labuschagne wrote: > > Ok if I poll a router / switch / server etc it seems to be ok, > a full scan on a 7606 takes around 45 seconds, > > > if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems
to be slow, even when the radio is directly connected to the hosting switch as a test.
> The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7
8 9 (that takes 40 to 50 seconds)
> > o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName
ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts
> ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts
ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts
> ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts
ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
> > > > The the other thing that had changed is that I was running a
slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
> > On 24 Jul 2016, at 09:16, Lyndon Labuschagne wrote: > > I know its bad form to reply to your own email but here is some
more information re drive performance
> > > root@observium:/opt/observium/rrd# hdparm -Tt
/dev/mapper/observium--vg-root
> > > /dev/mapper/observium--vg-root: > Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec > Timing buffered disk reads: 950 MB in 3.01 seconds > = 315.94 MB/sec > > root@observium:/opt/observium/rrd# dd if=/dev/zero
of=/tmp/output bs=8k count=10k; rm -f /tmp/output
> 10240+0 records in > 10240+0 records out > 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s > > > On 24 Jul 2016, at 09:13, Lyndon Labuschagne wrote: > > Hi All > > I have moved from a VM server to a physical server. as the VM
was taking a bit of a pounding, graph dropouts etc
> The physical server is a dual quad core Xeon, 2.66Ghz with
128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
> > > Will all of this my average poller-wrapper time is around 60 to
70 seconds per device
> if I manually run an snmpwalk a single device takes around 6
seconds
> > I have also disabled the ramdisk, rrdcached etc and I am not
seeing much of a difference either way
> > I have around 900 devices and the VM server was way > faster, any idea where to start looking for a bottleneck? > > > Regards > Lyndon > > observium mailing list > observium@observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > > observium mailing list > observium@observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > > observium mailing list > observium@observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > > observium mailing list > observium@observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > observium mailing list > observium@observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > > observium mailing list > observium@observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > _______________________________________________ > observium mailing list > observium@observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
so the speeds are going to get worse?
On 28 Jul 2016, at 17:29, Adam Armstrong <adama@memetic.orgmailto:adama@memetic.org> wrote:
Oh. Stable!
It's entirely possibly you don't yet have the changes which have the potential to make things "slow" ;)
Adam.
Sent from BlueMailhttp://www.bluemail.me/r
On 28 Jul 2016, at 07:49, Lyndon Labuschagne <lyndonl@mobiletorque.co.zamailto:lyndonl@mobiletorque.co.za> wrote:
Hi Yes all up to date
Is there any particular things I should be looking at / for, I have totally killed off the DB and it is currently building back up, discovering hosts etc a possible kick ass mysql config file or other hints
root@observium:/opt/observium# svn update Updating '.': At revision 8013.
root@observium:/opt/observium# ./discovery.php -u
___ _ _ / _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___ | | | || '_ \ / __| / _ | '__|\ \ / /| || | | || '_ ` _ \ | |_| || |_) |__ | __/| | \ V / | || |_| || | | | | | ___/ |_.__/ |___/ ___||_| _/ |_| __,_||_| |_| |_| Observium Professional 0.16.7.8003 http://www.observium.orghttp://www.observium.org/
-- Database is up to date.
On 28 Jul 2016, at 14:39, Adam Armstrong <adama@memetic.orgmailto:adama@memetic.org> wrote:
Hmm. I don't understand how they can be particularly slow. They're generally small tables.
Are you sure that's running the latest version? The ports table UPDATE seems fairly short.
adam. On 28/07/2016 09:06:42, Lyndon Labuschagne <lyndonl@mobiletorque.co.zamailto:lyndonl@mobiletorque.co.za> wrote:
The continuing saga
It seem that the "processors-state” integration waits for a long time, most other queries / read / writes are being completed in under 5 seconds, as shown below these are “Waiting” for 40+ seconds, these seems to be linked to JOINS in most cases Does anyone have an idea on how to speed these up from a tuning perspective?
8603 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2740' 8604 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2554' 8609 observium localhost observium 40 Query query en UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2555' 8610 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2154' 8611 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1880' 8612 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1916' 8627 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1936' 8629 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2161' 8633 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2596' 8634 observium localhost observium 37 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '864' 8637 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '956' 8638 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1073' 8639 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '578' 8640 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '513' 8572 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '312' 8630 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '430' 8646 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1099' 8647 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '573' 8613 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='20824064',`mempool_perc` ='32.73',`mempool_free` ='42799104',`mempool_total` ='63623168' WHERE `mempool_id` = '2359' 8615 observium localhost observium 12 Query query en UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='27287552',`mempool_perc` ='42.8',`mempool_free` ='36466688',`mempool_total` ='63754240' WHERE `mempool_id` = '1740' 8628 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='22896640',`mempool_perc` ='35.93',`mempool_free` ='40824832',`mempool_total` ='63721472' WHERE `mempool_id` = '2273' 8649 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '161' 8650 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '847' 8605 observium localhost observium 9 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1191' 8625 observium localhost observium 7 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1057' 8597 observium localhost observium 6 Query query en UPDATE `sensors-state` set `sensor_value` ='23.8',`sensor_event` ='ok',`sensor_status` ='',`sensor_last_change` ='1469690416',`sensor_polled` ='1469691635' WHERE `sensor_id` = '5102' 8576 observium localhost observium 4 Query query en UPDATE `ports` set `poll_time` ='1469691627',`poll_period` ='703',`ifInOctets` ='88925',`ifInOctets_rate` ='0',`ifOutOctets` ='88925',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` 7746 observium localhost observium 2 Query query en INSERT INTO `vlans_fdb` (`device_id`,`vlan_id`,`port_id`,`mac_address`,`fdb_status`) VALUES ('701','205','13401','84b59cb47383','learned') 8428 observium localhost observium 2 Query query en UPDATE `ports` set `poll_time` ='1469691592',`poll_period` ='328',`ifInOctets` ='0',`ifInOctets_rate` ='0',`ifOutOctets` ='0',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` ='0',`i
411 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2708','0','1469692947') 421 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2404','0','1469692947') 423 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2119','0','1469692947') 425 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2004','0','1469692947') 426 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2287','0','1469692947') 429 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2888','0','1469692947') 431 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2875','0','1469692947') 432 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2661','0','1469692947') 433 observium localhost observium 24 Query query en INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2759','0','1469692947') 434 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2455','22','1469692947') 435 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2982','0','1469692947') 436 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2120','1','1469692947') 437 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2452','30','1469692947') 439 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1926','0','1469692947') 441 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2447','36','1469692947') 450 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2986','0','1469692947') 451 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2267','0','1469692947') 454 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2937','0','1469692947') 456 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('3006','0','1469692947') 457 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2916','0','1469692947') 458 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2358','0','1469692947') 459 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2825','0','1469692947') 460 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2332','0','1469692947') 471 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1866','0','1469692947') 478 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2121','0','1469692947') 482 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2549','0','1469692947')
On 25 Jul 2016, at 11:35, Lyndon Labuschagne wrote:
This is a walk from the Server that is running at a decent poller speed It complains about LM-SENSORS-MIB, and yet the MIB exists, and is identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
What I also find interesting although possibly nothing is that the returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04 The old server which is running fine, is Ubuntu 15.04
Both use snmpd version 5.7.3
snmpwalk -v1 -c SNMPKEY domain.namehttp://domain.name/.tld .1.3.6.1http://1.3.6.1/.2.1.1 MIB search path: /opt/observium/mibs Cannot find module (LM-SENSORS-MIB): At line 1 in (none)
SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65http://2.6.32.65/ #1 Thu Jul 16 11:50:24 EEST 2015 mips SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62 days, 23:05:46.00 SNMPv2-MIB::sysContact.0 = STRING: networks SNMPv2-MIB::sysName.0 = STRING: domain.namehttp://domain.name/.tld SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location SNMPv2-MIB::sysServices.0 = INTEGER: 79 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORIndex.1 = INTEGER: 1 SNMPv2-MIB::sysORIndex.2 = INTEGER: 2 SNMPv2-MIB::sysORIndex.3 = INTEGER: 3 SNMPv2-MIB::sysORIndex.4 = INTEGER: 4 SNMPv2-MIB::sysORIndex.5 = INTEGER: 5 SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036 SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB SNMPv2-MIB::sysORID.4 = OID: SNMPv2-SMI::enterprises.10002.1.1.1http://002.1.1.1/.31 SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112 SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP entities SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11 entities. SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe generic objects for network interface sub-layers SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe system resources SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
- Server that is showing issues snmpwalk -v1 -c SNMPKEY domain.namehttp://domain.name/.tld .1.3.6.1http://1.3.6.1/.2.1.1 iso.3.6.1.2http://3.6.1.2/.1.1.1.0http://1.1.1.0/ = STRING: "Linux 2.6.32.65http://2.6.32.65/ #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2http://3.6.1.2/.1.1.2.0http://1.1.2.0/ = OID: iso.3.6.1.4http://3.6.1.4/.1.41112.1.4 iso.3.6.1.2http://3.6.1.2/.1.1.3.0http://1.1.3.0/ = Timeticks: (543981700) 62 days, 23:03:37.00 iso.3.6.1.2http://3.6.1.2/.1.1.4.0http://1.1.4.0/ = STRING: "networks" iso.3.6.1.2http://3.6.1.2/.1.1.5.0http://1.1.5.0/ = STRING: "domain.namehttp://domain.name/.tld" iso.3.6.1.2http://3.6.1.2/.1.1.6.0http://1.1.6.0/ = STRING: "Some GPS Location" iso.3.6.1.2http://3.6.1.2/.1.1.7.0http://1.1.7.0/ = INTEGER: 79 iso.3.6.1.2http://3.6.1.2/.1.1.8.0http://1.1.8.0/ = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.1 = INTEGER: 1 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.2 = INTEGER: 2 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.3 = INTEGER: 3 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.4 = INTEGER: 4 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.5 = INTEGER: 5 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.1 = OID: iso.3.6.1.6http://3.6.1.6/.3.1 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.2 = OID: iso.2.840.10036 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.3 = OID: iso.3.6.1.2http://3.6.1.2/.1.31 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.4 = OID: iso.3.6.1.4http://3.6.1.4/.1.10002.1.1.1http://002.1.1.1/.31 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.5 = OID: iso.3.6.1.4http://3.6.1.4/.1.41112 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
On 25 Jul 2016, at 09:00, Lyndon Labuschagne wrote:
May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not
found this in debug mode
It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.namehttp://dns.name/.tld variable name in this MIB."
the device is reachable via SNMP as shown in a portion of the snmpwalk below
this is where I am seeing the slowing down during the ifindex scanning as previously posted if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645'] SQL RUNTIME[0.00229788s] SNMP STATUS[FALSE] 6 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.namehttp://dns.name/.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
CMD EXITCODE[1] CMD RUNTIME[6.0817s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.6
Timeout: No Response from udp:dns.namehttp://dns.name/.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';] SQL RUNTIME[0.00058079s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650'] SQL RUNTIME[0.00175095s] SNMP STATUS[FALSE] 7 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.namehttp://dns.name/.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
CMD EXITCODE[1] CMD RUNTIME[6.0846s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: ifInMulticastPkts.7
Timeout: No Response from udp:dns.namehttp://dns.name/.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';] SQL RUNTIME[0.00056005s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656'] SQL RUNTIME[0.00169611s] SNMP STATUS[FALSE] 8 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.namehttp://dns.name/.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
CMD EXITCODE[1] CMD RUNTIME[6.0792s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.8
Timeout: No Response from udp:dns.namehttp://dns.name/.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';] SQL RUNTIME[0.00071502s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662'] SQL RUNTIME[0.00218582s] SNMP STATUS[FALSE] 9 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.namehttp://dns.name/.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
CMD EXITCODE[1] CMD RUNTIME[6.0803s] STDOUT[
This is a manual snmpwalk
root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.namehttp://dns.name/.tld iso.3.6.1.2http://3.6.1.2/.1.1.1.0http://1.1.1.0/ = STRING: "Linux 2.6.32.65http://2.6.32.65/ #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2http://3.6.1.2/.1.1.2.0http://1.1.2.0/ = OID: iso.3.6.1.4http://3.6.1.4/.1.41112.1.4 iso.3.6.1.2http://3.6.1.2/.1.1.3.0http://1.1.3.0/ = Timeticks: (543077500) 62 days, 20:32:55.00 iso.3.6.1.2http://3.6.1.2/.1.1.4.0http://1.1.4.0/ = STRING: "networks" iso.3.6.1.2http://3.6.1.2/.1.1.5.0http://1.1.5.0/ = STRING: "vdns.namehttp://vdns.name/.tld" iso.3.6.1.2http://3.6.1.2/.1.1.6.0http://1.1.6.0/ = STRING: “Some GPS Co Ords" iso.3.6.1.2http://3.6.1.2/.1.1.7.0http://1.1.7.0/ = INTEGER: 79 iso.3.6.1.2http://3.6.1.2/.1.1.8.0http://1.1.8.0/ = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.1 = INTEGER: 1 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.2 = INTEGER: 2 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.3 = INTEGER: 3 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.4 = INTEGER: 4 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.5 = INTEGER: 5 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.1 = OID: iso.3.6.1.6http://3.6.1.6/.3.1 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.2 = OID: iso.2.840.10036 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.3 = OID: iso.3.6.1.2http://3.6.1.2/.1.31 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.4 = OID: iso.3.6.1.4http://3.6.1.4/.1.10002.1.1.1http://002.1.1.1/.31 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.5 = OID: iso.3.6.1.4http://3.6.1.4/.1.41112 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.2.1.0http://1.2.1.0/ = INTEGER: 9 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.1 = INTEGER: 1 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.2 = INTEGER: 2 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.3 = INTEGER: 3
iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.4 = INTEGER: 4 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.5 = INTEGER: 5 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.6 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.7 = INTEGER: 7 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.8 = INTEGER: 8 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.9 = INTEGER: 9 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.1 = STRING: "lo" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.2 = STRING: "eth0" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.3 = STRING: "eth1" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.4 = STRING: "wifi0" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.5 = STRING: "ath0" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.6 = STRING: "eth0.36" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.7 = STRING: "ath0.36" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.8 = STRING: "br0" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.9 = STRING: "br1" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.1 = INTEGER: 24 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.2 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.3 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.4 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.5 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.6 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.7 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.8 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.9 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.1 = INTEGER: 16436 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.2 = INTEGER: 2020 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.3 = INTEGER: 1500 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.4 = INTEGER: 2286 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.5 = INTEGER: 2020 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.6 = INTEGER: 2020 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.7 = INTEGER: 2020 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.8 = INTEGER: 2020 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.9 = INTEGER: 2020 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.5.1 = Gauge32: 0 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.5.2 = Gauge32: 100000000 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.5.3 = Gauge32: 0 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.5.4 = Gauge32: 0 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.5.5 = Gauge32: 0 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.5.6 = Gauge32: 100000000 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.5.7 = Gauge32: 0
On 25 Jul 2016, at 07:24, Lyndon Labuschagne wrote:
Im going to fire up the server again today and test more, will let you know,
if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
On 25 Jul 2016, at 00:30, Adam Armstrong wrote:
If do you have sufficient buffers and caching for innodb in mysql?
We have some /very/ large installs which saw little impact from the changes.
Adam.
Sent from BlueMail
On 24 Jul 2016, at 20:55, Lyndon Labuschagne wrote: I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne wrote: i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device,
until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec
root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out
83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium ________________________________
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
________________________________
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
________________________________
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
________________________________
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
________________________________
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium ________________________________
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
________________________________
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
I have no idea. I'm not sure why the issues you have now exist :)
It sounds like you probably have some pretty severe I/O bottleneck somewhere.
Adam.
Sent from BlueMail
On 28 Jul 2016, 11:18, at 11:18, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
so the speeds are going to get worse?
On 28 Jul 2016, at 17:29, Adam Armstrong <adama@memetic.orgmailto:adama@memetic.org> wrote:
Oh. Stable!
It's entirely possibly you don't yet have the changes which have the potential to make things "slow" ;)
Adam.
Sent from BlueMailhttp://www.bluemail.me/r
On 28 Jul 2016, at 07:49, Lyndon Labuschagne <lyndonl@mobiletorque.co.zamailto:lyndonl@mobiletorque.co.za> wrote:
Hi Yes all up to date
Is there any particular things I should be looking at / for, I have totally killed off the DB and it is currently building back up, discovering hosts etc a possible kick ass mysql config file or other hints
root@observium:/opt/observium# svn update Updating '.': At revision 8013.
root@observium:/opt/observium# ./discovery.php -u
/ _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___ | | | || '_ \ / __| / _ | '__|\ \ / /| || | | || '_ ` _ \ | |_| || |_) |__ | __/| | \ V / | || |_| || | | | | | ___/ |_.__/ |___/ ___||_| _/ |_| __,_||_| |_| |_| Observium Professional 0.16.7.8003 http://www.observium.orghttp://www.observium.org/
-- Database is up to date.
On 28 Jul 2016, at 14:39, Adam Armstrong <adama@memetic.orgmailto:adama@memetic.org> wrote:
Hmm. I don't understand how they can be particularly slow. They're generally small tables.
Are you sure that's running the latest version? The ports table UPDATE seems fairly short.
adam. On 28/07/2016 09:06:42, Lyndon Labuschagne <lyndonl@mobiletorque.co.zamailto:lyndonl@mobiletorque.co.za> wrote:
The continuing saga
It seem that the "processors-state” integration waits for a long time, most other queries / read / writes are being completed in under 5 seconds, as shown below these are “Waiting” for 40+ seconds, these seems to be linked to JOINS in most cases Does anyone have an idea on how to speed these up from a tuning perspective?
8603 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2740' 8604 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2554' 8609 observium localhost observium 40 Query query en UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2555' 8610 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2154' 8611 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1880' 8612 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1916' 8627 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1936' 8629 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2161' 8633 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2596' 8634 observium localhost observium 37 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '864' 8637 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '956' 8638 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1073' 8639 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '578' 8640 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '513' 8572 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '312' 8630 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '430' 8646 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1099' 8647 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '573' 8613 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='20824064',`mempool_perc` ='32.73',`mempool_free` ='42799104',`mempool_total` ='63623168' WHERE `mempool_id` = '2359' 8615 observium localhost observium 12 Query query en UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='27287552',`mempool_perc` ='42.8',`mempool_free` ='36466688',`mempool_total` ='63754240' WHERE `mempool_id` = '1740' 8628 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='22896640',`mempool_perc` ='35.93',`mempool_free` ='40824832',`mempool_total` ='63721472' WHERE `mempool_id` = '2273' 8649 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '161' 8650 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '847' 8605 observium localhost observium 9 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1191' 8625 observium localhost observium 7 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1057' 8597 observium localhost observium 6 Query query en UPDATE `sensors-state` set `sensor_value` ='23.8',`sensor_event` ='ok',`sensor_status` ='',`sensor_last_change` ='1469690416',`sensor_polled` ='1469691635' WHERE `sensor_id` = '5102' 8576 observium localhost observium 4 Query query en UPDATE `ports` set `poll_time` ='1469691627',`poll_period` ='703',`ifInOctets` ='88925',`ifInOctets_rate` ='0',`ifOutOctets` ='88925',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` 7746 observium localhost observium 2 Query query en INSERT INTO `vlans_fdb` (`device_id`,`vlan_id`,`port_id`,`mac_address`,`fdb_status`) VALUES ('701','205','13401','84b59cb47383','learned') 8428 observium localhost observium 2 Query query en UPDATE `ports` set `poll_time` ='1469691592',`poll_period` ='328',`ifInOctets` ='0',`ifInOctets_rate` ='0',`ifOutOctets` ='0',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` ='0',`i
411 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2708','0','1469692947') 421 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2404','0','1469692947') 423 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2119','0','1469692947') 425 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2004','0','1469692947') 426 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2287','0','1469692947') 429 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2888','0','1469692947') 431 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2875','0','1469692947') 432 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2661','0','1469692947') 433 observium localhost observium 24 Query query en INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2759','0','1469692947') 434 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2455','22','1469692947') 435 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2982','0','1469692947') 436 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2120','1','1469692947') 437 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2452','30','1469692947') 439 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1926','0','1469692947') 441 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2447','36','1469692947') 450 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2986','0','1469692947') 451 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2267','0','1469692947') 454 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2937','0','1469692947') 456 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('3006','0','1469692947') 457 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2916','0','1469692947') 458 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2358','0','1469692947') 459 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2825','0','1469692947') 460 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2332','0','1469692947') 471 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1866','0','1469692947') 478 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2121','0','1469692947') 482 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2549','0','1469692947')
On 25 Jul 2016, at 11:35, Lyndon Labuschagne wrote:
This is a walk from the Server that is running at a decent poller speed It complains about LM-SENSORS-MIB, and yet the MIB exists, and is identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
What I also find interesting although possibly nothing is that the returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04 The old server which is running fine, is Ubuntu 15.04
Both use snmpd version 5.7.3
snmpwalk -v1 -c SNMPKEY domain.namehttp://domain.name/.tld .1.3.6.1http://1.3.6.1/.2.1.1 MIB search path: /opt/observium/mibs Cannot find module (LM-SENSORS-MIB): At line 1 in (none)
SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65http://2.6.32.65/ #1 Thu Jul 16 11:50:24 EEST 2015 mips SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62 days, 23:05:46.00 SNMPv2-MIB::sysContact.0 = STRING: networks SNMPv2-MIB::sysName.0 = STRING: domain.namehttp://domain.name/.tld SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location SNMPv2-MIB::sysServices.0 = INTEGER: 79 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORIndex.1 = INTEGER: 1 SNMPv2-MIB::sysORIndex.2 = INTEGER: 2 SNMPv2-MIB::sysORIndex.3 = INTEGER: 3 SNMPv2-MIB::sysORIndex.4 = INTEGER: 4 SNMPv2-MIB::sysORIndex.5 = INTEGER: 5 SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036 SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB SNMPv2-MIB::sysORID.4 = OID: SNMPv2-SMI::enterprises.10002.1.1.1http://002.1.1.1/.31 SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112 SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP entities SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11 entities. SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe generic objects for network interface sub-layers SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe system resources SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
- Server that is showing issues
snmpwalk -v1 -c SNMPKEY domain.namehttp://domain.name/.tld .1.3.6.1http://1.3.6.1/.2.1.1 iso.3.6.1.2http://3.6.1.2/.1.1.1.0http://1.1.1.0/ = STRING: "Linux 2.6.32.65http://2.6.32.65/ #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2http://3.6.1.2/.1.1.2.0http://1.1.2.0/ = OID: iso.3.6.1.4http://3.6.1.4/.1.41112.1.4 iso.3.6.1.2http://3.6.1.2/.1.1.3.0http://1.1.3.0/ = Timeticks: (543981700) 62 days, 23:03:37.00 iso.3.6.1.2http://3.6.1.2/.1.1.4.0http://1.1.4.0/ = STRING: "networks" iso.3.6.1.2http://3.6.1.2/.1.1.5.0http://1.1.5.0/ = STRING: "domain.namehttp://domain.name/.tld" iso.3.6.1.2http://3.6.1.2/.1.1.6.0http://1.1.6.0/ = STRING: "Some GPS Location" iso.3.6.1.2http://3.6.1.2/.1.1.7.0http://1.1.7.0/ = INTEGER: 79 iso.3.6.1.2http://3.6.1.2/.1.1.8.0http://1.1.8.0/ = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.1 = INTEGER: 1 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.2 = INTEGER: 2 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.3 = INTEGER: 3 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.4 = INTEGER: 4 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.5 = INTEGER: 5 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.1 = OID: iso.3.6.1.6http://3.6.1.6/.3.1 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.2 = OID: iso.2.840.10036 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.3 = OID: iso.3.6.1.2http://3.6.1.2/.1.31 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.4 = OID: iso.3.6.1.4http://3.6.1.4/.1.10002.1.1.1http://002.1.1.1/.31 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.5 = OID: iso.3.6.1.4http://3.6.1.4/.1.41112 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
On 25 Jul 2016, at 09:00, Lyndon Labuschagne wrote:
May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not
found this in debug mode
It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.namehttp://dns.name/.tld variable name in this MIB."
the device is reachable via SNMP as shown in a portion of the snmpwalk below
this is where I am seeing the slowing down during the ifindex scanning as previously posted if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645'] SQL RUNTIME[0.00229788s] SNMP STATUS[FALSE] 6 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.namehttp://dns.name/.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
CMD EXITCODE[1] CMD RUNTIME[6.0817s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.6
Timeout: No Response from udp:dns.namehttp://dns.name/.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';] SQL RUNTIME[0.00058079s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650'] SQL RUNTIME[0.00175095s] SNMP STATUS[FALSE] 7 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.namehttp://dns.name/.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
CMD EXITCODE[1] CMD RUNTIME[6.0846s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: ifInMulticastPkts.7
Timeout: No Response from udp:dns.namehttp://dns.name/.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';] SQL RUNTIME[0.00056005s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656'] SQL RUNTIME[0.00169611s] SNMP STATUS[FALSE] 8 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.namehttp://dns.name/.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
CMD EXITCODE[1] CMD RUNTIME[6.0792s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.8
Timeout: No Response from udp:dns.namehttp://dns.name/.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';] SQL RUNTIME[0.00071502s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662'] SQL RUNTIME[0.00218582s] SNMP STATUS[FALSE] 9 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.namehttp://dns.name/.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
CMD EXITCODE[1] CMD RUNTIME[6.0803s] STDOUT[
This is a manual snmpwalk
root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.namehttp://dns.name/.tld iso.3.6.1.2http://3.6.1.2/.1.1.1.0http://1.1.1.0/ = STRING: "Linux 2.6.32.65http://2.6.32.65/ #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2http://3.6.1.2/.1.1.2.0http://1.1.2.0/ = OID: iso.3.6.1.4http://3.6.1.4/.1.41112.1.4 iso.3.6.1.2http://3.6.1.2/.1.1.3.0http://1.1.3.0/ = Timeticks: (543077500) 62 days, 20:32:55.00 iso.3.6.1.2http://3.6.1.2/.1.1.4.0http://1.1.4.0/ = STRING: "networks" iso.3.6.1.2http://3.6.1.2/.1.1.5.0http://1.1.5.0/ = STRING: "vdns.namehttp://vdns.name/.tld" iso.3.6.1.2http://3.6.1.2/.1.1.6.0http://1.1.6.0/ = STRING: “Some GPS Co Ords" iso.3.6.1.2http://3.6.1.2/.1.1.7.0http://1.1.7.0/ = INTEGER: 79 iso.3.6.1.2http://3.6.1.2/.1.1.8.0http://1.1.8.0/ = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.1 = INTEGER: 1 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.2 = INTEGER: 2 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.3 = INTEGER: 3 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.4 = INTEGER: 4 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.1.5 = INTEGER: 5 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.1 = OID: iso.3.6.1.6http://3.6.1.6/.3.1 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.2 = OID: iso.2.840.10036 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.3 = OID: iso.3.6.1.2http://3.6.1.2/.1.31 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.4 = OID: iso.3.6.1.4http://3.6.1.4/.1.10002.1.1.1http://002.1.1.1/.31 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.2.5 = OID: iso.3.6.1.4http://3.6.1.4/.1.41112 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.1.9.1http://1.1.9.1/.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2http://3.6.1.2/.1.2.1.0http://1.2.1.0/ = INTEGER: 9 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.1 = INTEGER: 1 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.2 = INTEGER: 2 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.3 = INTEGER: 3
iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.4 = INTEGER: 4 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.5 = INTEGER: 5 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.6 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.7 = INTEGER: 7 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.8 = INTEGER: 8 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.1.9 = INTEGER: 9 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.1 = STRING: "lo" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.2 = STRING: "eth0" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.3 = STRING: "eth1" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.4 = STRING: "wifi0" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.5 = STRING: "ath0" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.6 = STRING: "eth0.36" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.7 = STRING: "ath0.36" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.8 = STRING: "br0" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.2.9 = STRING: "br1" iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.1 = INTEGER: 24 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.2 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.3 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.4 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.5 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.6 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.7 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.8 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.3.9 = INTEGER: 6 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.1 = INTEGER: 16436 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.2 = INTEGER: 2020 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.3 = INTEGER: 1500 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.4 = INTEGER: 2286 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.5 = INTEGER: 2020 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.6 = INTEGER: 2020 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.7 = INTEGER: 2020 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.8 = INTEGER: 2020 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.4.9 = INTEGER: 2020 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.5.1 = Gauge32: 0 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.5.2 = Gauge32: 100000000 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.5.3 = Gauge32: 0 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.5.4 = Gauge32: 0 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.5.5 = Gauge32: 0 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.5.6 = Gauge32: 100000000 iso.3.6.1.2http://3.6.1.2/.1.2.2.1http://1.2.2.1/.5.7 = Gauge32: 0
On 25 Jul 2016, at 07:24, Lyndon Labuschagne wrote:
Im going to fire up the server again today and test more, will let you know,
if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
On 25 Jul 2016, at 00:30, Adam Armstrong wrote:
If do you have sufficient buffers and caching for innodb in mysql?
We have some /very/ large installs which saw little impact from the changes.
Adam.
Sent from BlueMail
On 24 Jul 2016, at 20:55, Lyndon Labuschagne wrote: I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne wrote: i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device,
until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec
root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out
83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium ________________________________
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium ________________________________
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
I would have assumed the same thing but after numerous tests I just cant see it happening especially not on the Mysql side
After I dropped the DB though things are looking better with a fresh start
MySQL test 30 seconds limit
OLTP test statistics: queries performed: read: 591948 write: 211360 other: 84547 total: 887855 transactions: 42265 (1407.82 per sec.) deadlocks: 17 (0.57 per sec.) read/write requests: 803308 (26757.71 per sec.) other operations: 84547 (2816.21 per sec.)
root@observium:~# sysbench --test=cpu --cpu-max-prime=20000 run sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options: Number of threads: 1
Doing CPU performance benchmark
Threads started! Done.
Maximum prime number checked in CPU test: 20000
Test execution summary: total time: 30.9549s total number of events: 10000 total time taken by event execution: 30.9459 per-request statistics: min: 2.49ms avg: 3.09ms max: 6.87ms approx. 95 percentile: 4.41ms
Threads fairness: events (avg/stddev): 10000.0000/0.00 execution time (avg/stddev): 30.9459/0.00
Running the test with following options: Number of threads: 1 Initializing random number generator from timer.
Extra file open flags: 0 128 files, 1.1719Gb each 150Gb total file size Block size 16Kb Number of random requests for random IO: 0 Read/Write ratio for combined random IO test: 1.50 Periodic FSYNC enabled, calling fsync() each 100 requests. Calling fsync() at the end of test, Enabled. Using synchronous I/O mode Doing random r/w test Threads started! Time limit exceeded, exiting... Done.
Operations performed: 20837 Read, 13891 Write, 44416 Other = 79144 Total Read 325.58Mb Written 217.05Mb Total transferred 542.62Mb (1.8087Mb/sec) 115.76 Requests/sec executed
Test execution summary: total time: 300.0011s total number of events: 34728 total time taken by event execution: 124.5346 per-request statistics: min: 0.00ms avg: 3.59ms max: 1548.47ms approx. 95 percentile: 16.26ms
Threads fairness: events (avg/stddev): 34728.0000/0.00 execution time (avg/stddev): 124.5346/0.00
On 28 Jul 2016, at 19:01, Adam Armstrong adama@memetic.org wrote:
I have no idea. I'm not sure why the issues you have now exist :)
It sounds like you probably have some pretty severe I/O bottleneck somewhere.
Adam.
Sent from BlueMail
On 28 Jul 2016, at 11:18, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote: so the speeds are going to get worse?
On 28 Jul 2016, at 17:29, Adam Armstrong adama@memetic.org wrote:
Oh. Stable!
It's entirely possibly you don't yet have the changes which have the potential to make things "slow" ;)
Adam.
Sent from BlueMail
On 28 Jul 2016, at 07:49, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote: Hi Yes all up to date
Is there any particular things I should be looking at / for, I have totally killed off the DB and it is currently building back up, discovering hosts etc a possible kick ass mysql config file or other hints
root@observium:/opt/observium# svn update Updating '.': At revision 8013.
root@observium:/opt/observium# ./discovery.php -u
/ _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___ | | | || '_ \ / __| / _ | '__|\ \ / /| || | | || '_ ` _ \ | |_| || |_) |__ | __/| | \ V / | || |_| || | | | | | ___/ |_.__/ |___/ ___||_| _/ |_| __,_||_| |_| |_| Observium Professional 0.16.7.8003 http://www.observium.org
-- Database is up to date.
On 28 Jul 2016, at 14:39, Adam Armstrong adama@memetic.org wrote:
Hmm. I don't understand how they can be particularly slow. They're generally small tables.
Are you sure that's running the latest version? The ports table UPDATE seems fairly short.
adam. On 28/07/2016 09:06:42, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote:
The continuing saga
It seem that the "processors-state” integration waits for a long time, most other queries / read / writes are being completed in under 5 seconds, as shown below these are “Waiting” for 40+ seconds, these seems to be linked to JOINS in most cases Does anyone have an idea on how to speed these up from a tuning perspective?
8603 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2740'
8604 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2554' 8609 observium localhost observium 40 Query query en UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2555' 8610 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2154' 8611 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1880' 8612 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1916'
8627 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1936'
8629 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2161' 8633 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2596' 8634 observium localhost observium 37 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '864' 8637 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '956'
8638 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1073' 8639 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '578' 8640 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '513' 8572 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '312'
8630 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '430' 8646 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1099' 8647 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '573' 8613 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='20824064',`mempool_perc` ='32.73',`mempool_free` ='42799104',`mempool_total` ='63623168' WHERE `mempool_id` = '2359'
8615 observium localhost observium 12 Query query en UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='27287552',`mempool_perc` ='42.8',`mempool_free` ='36466688',`mempool_total` ='63754240' WHERE `mempool_id` = '1740' 8628 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='22896640',`mempool_perc` ='35.93',`mempool_free` ='40824832',`mempool_total` ='63721472' WHERE `mempool_id` = '2273' 8649 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '161' 8650 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '847'
8605 observium localhost observium 9 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1191' 8625 observium localhost observium 7 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1057' 8597 observium localhost observium 6 Query query en UPDATE `sensors-state` set `sensor_value` ='23.8',`sensor_event` ='ok',`sensor_status` ='',`sensor_last_change` ='1469690416',`sensor_polled` ='1469691635' WHERE `sensor_id` = '5102' 8576 observium localhost observium 4 Query query en UPDATE `ports` set `poll_time` ='1469691627',`poll_period` ='703',`ifInOctets` ='88925',`ifInOctets_rate` ='0',`ifOutOctets` ='88925',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` 7746 observium localhost observium 2 Query query en INSERT INTO `vlans_fdb` (`device_id`,`vlan_id`,`port_id`,`mac_address`,`fdb_status`) VALUES ('701','205','13401','84b59cb47383','learned')
8428 observium localhost observium 2 Query query en UPDATE `ports` set `poll_time` ='1469691592',`poll_period` ='328',`ifInOctets` ='0',`ifInOctets_rate` ='0',`ifOutOctets` ='0',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` ='0',`i
411 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2708','0','1469692947') 421 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2404','0','1469692947') 423 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2119','0','1469692947')
425 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2004','0','1469692947')
426 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2287','0','1469692947') 429 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2888','0','1469692947') 431 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2875','0','1469692947') 432 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2661','0','1469692947') 433 observium localhost observium 24 Query query en INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2759','0','1469692947')
434 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2455','22','1469692947') 435 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2982','0','1469692947') 436 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2120','1','1469692947') 437 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2452','30','1469692947') 439 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1926','0','1469692947') 441 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2447','36','1469692947')
450 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2986','0','1469692947') 451 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2267','0','1469692947') 454 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2937','0','1469692947') 456 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('3006','0','1469692947') 457 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2916','0','1469692947')
458 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2358','0','1469692947')
459 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2825','0','1469692947') 460 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2332','0','1469692947') 471 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1866','0','1469692947') 478 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2121','0','1469692947') 482 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2549','0','1469692947')
On 25 Jul 2016, at 11:35, Lyndon Labuschagne wrote:
This is a walk from the Server that is running at a decent poller speed It complains about LM-SENSORS-MIB, and yet the MIB exists, and is identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
What I also find interesting although possibly nothing is that the returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04 The old server which is running fine, is Ubuntu 15.04
Both use snmpd version 5.7.3
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 MIB search path: /opt/observium/mibs Cannot find module (LM-SENSORS-MIB): At line 1 in (none)
SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62 days, 23:05:46.00 SNMPv2-MIB::sysContact.0 = STRING: networks SNMPv2-MIB::sysName.0 = STRING: domain.name.tld SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location SNMPv2-MIB::sysServices.0 = INTEGER: 79 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORIndex.1 = INTEGER: 1 SNMPv2-MIB::sysORIndex.2 = INTEGER: 2 SNMPv2-MIB::sysORIndex.3 = INTEGER: 3 SNMPv2-MIB::sysORIndex.4 = INTEGER: 4 SNMPv2-MIB::sysORIndex.5 = INTEGER: 5 SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB
SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036 SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB SNMPv2-MIB::sysORID.4 = OID: SNMPv2-SMI::enterprises.10 002.1.1.1.31 SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112 SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP entities SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11 entities. SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe generic objects for network interface sub-layers SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe system resources SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
- Server that is showing issues
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543981700) 62 days, 23:03:37.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "domain.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: "Some GPS Location" iso.3.6.1.2.1.1.7.0 = INTEGER: 79
iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1
iso. 3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1 .3.2 = STRING: "The MIB module for IEEE 802.11 entities."
iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
On 25 Jul 2016, at 09:00, Lyndon Labuschagne wrote:
May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not
found this in debug mode
It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.name.tld variable name in this MIB."
the device is reachable via SNMP as shown in a portion of the snmpwalk below
this is where I am seeing the slowing down during the ifindex scanning as previously posted if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645']
SQL RUNTIME[0.00229788s] SNMP STATUS[FALSE] 6 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
CMD EXITCODE[1] CMD RUNTIME[6.0817s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: ifInMulticastPkts.6
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';]
SQL RUNTIME[0.00058079s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650']
SQL RUNTIME[0.00175095s] SNMP STATUS[FALSE] 7 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
CMD EXITCODE[1] CMD RUNTIME[6.0846s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: ifInMulticastPkts.7
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';] SQL RUNTIME[0.00056005s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656']
SQL RUNTIME[0.00169611s] SNMP STATUS[FALSE] 8 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
CMD EXITCODE[1] CMD RUNTIME[6.0792s] STDOUT[
] STDERR[ Error in packet
Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.8
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';] SQL RUNTIME[0.00071502s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662']
SQL RUNTIME[0.00218582s]
SNMP STATUS[FALSE] 9 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
CMD EXITCODE[1] CMD RUNTIME[6.0803s] STDOUT[
This is a manual snmpwalk
root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.2.1.0 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3
iso. 3.6.1.2.1.2.2.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7 iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8 iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo" iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0" iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1" iso.3.6.1.2.1.2.2.1 .2.4 = STRING: "wifi0"
iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0" iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36" iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36" iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0" iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1" iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24 iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436 iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500 iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286 iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0 iso.3.6.1.2.1.2.2.1 .5.6 = Gauge32: 100000000
iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0
On 25 Jul 2016, at 07:24, Lyndon Labuschagne wrote:
Im going to fire up the server again today and test more, will let you know,
if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
On 25 Jul 2016, at 00:30, Adam Armstrong wrote:
If do you have sufficient buffers and caching for innodb in mysql?
We have some /very/ large installs which saw little impact from the changes.
Adam.
Sent from BlueMail
On 24 Jul 2016, at 20:55, Lyndon Labuschagne wrote: I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne wrote: i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device,
until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec
root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out
83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list
observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list
observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
How big is this installation?
Normally you wouldn't have you think about performance issues until quite a lot of ports.
adam. On 28/07/2016 18:41:06, Lyndon Labuschagne lyndonl@mobiletorque.co.za wrote: I would have assumed the same thing but after numerous tests I just cant see it happening especially not on the Mysql side
After I dropped the DB though things are looking better with a fresh start
MySQL test 30 seconds limit
OLTP test statistics: queries performed: read: 591948 write: 211360 other: 84547 total: 887855 transactions: 42265 (1407.82 per sec.) deadlocks: 17 (0.57 per sec.) read/write requests: 803308 (26757.71 per sec.) other operations: 84547 (2816.21 per sec.)
root@observium:~# sysbench --test=cpu --cpu-max-prime=20000 run sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options: Number of threads: 1
Doing CPU performance benchmark
Threads started! Done.
Maximum prime number checked in CPU test: 20000
Test execution summary: total time: 30.9549s total number of events: 10000 total time taken by event execution: 30.9459 per-request statistics: min: 2.49ms avg: 3.09ms max: 6.87ms approx. 95 percentile: 4.41ms
Threads fairness: events (avg/stddev): 10000.0000/0.00 execution time (avg/stddev): 30.9459/0.00
Running the test with following options: Number of threads: 1 Initializing random number generator from timer.
Extra file open flags: 0 128 files, 1.1719Gb each 150Gb total file size Block size 16Kb Number of random requests for random IO: 0 Read/Write ratio for combined random IO test: 1.50 Periodic FSYNC enabled, calling fsync() each 100 requests. Calling fsync() at the end of test, Enabled. Using synchronous I/O mode Doing random r/w test Threads started! Time limit exceeded, exiting... Done.
Operations performed: 20837 Read, 13891 Write, 44416 Other = 79144 Total Read 325.58Mb Written 217.05Mb Total transferred 542.62Mb (1.8087Mb/sec) 115.76 Requests/sec executed
Test execution summary: total time: 300.0011s total number of events: 34728 total time taken by event execution: 124.5346 per-request statistics: min: 0.00ms avg: 3.59ms max: 1548.47ms approx. 95 percentile: 16.26ms
Threads fairness: events (avg/stddev): 34728.0000/0.00 execution time (avg/stddev): 124.5346/0.00
On 28 Jul 2016, at 19:01, Adam Armstrong wrote:
I have no idea. I'm not sure why the issues you have now exist :)
It sounds like you probably have some pretty severe I/O bottleneck somewhere.
Adam.
Sent from BlueMail
On 28 Jul 2016, at 11:18, Lyndon Labuschagne wrote: so the speeds are going to get worse?
On 28 Jul 2016, at 17:29, Adam Armstrong wrote:
Oh. Stable!
It's entirely possibly you don't yet have the changes which have the potential to make things "slow" ;)
Adam.
Sent from BlueMail
On 28 Jul 2016, at 07:49, Lyndon Labuschagne wrote: Hi Yes all up to date
Is there any particular things I should be looking at / for, I have totally killed off the DB and it is currently building back up, discovering hosts etc a possible kick ass mysql config file or other hints
root@observium:/opt/observium# svn update Updating '.': At revision 8013.
root@observium:/opt/observium# ./discovery.php -u
/ _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___ | | | || '_ \ / __| / _ | '__|\ \ / /| || | | || '_ ` _ \ | |_| || |_) |__ | __/| | \ V / | || |_| || | | | | | ___/ |_.__/ |___/ ___||_| _/ |_| __,_||_| |_| |_| Observium Professional 0.16.7.8003 http://www.observium.org
-- Database is up to date.
On 28 Jul 2016, at 14:39, Adam Armstrong wrote:
Hmm. I don't understand how they can be particularly slow. They're generally small tables.
Are you sure that's running the latest version? The ports table UPDATE seems fairly short.
adam. On 28/07/2016 09:06:42, Lyndon Labuschagne wrote:
The continuing saga
It seem that the "processors-state” integration waits for a long time, most other queries / read / writes are being completed in under 5 seconds, as shown below these are “Waiting” for 40+ seconds, these seems to be linked to JOINS in most cases Does anyone have an idea on how to speed these up from a tuning perspective?
8603 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2740'
8604 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2554' 8609 observium localhost observium 40 Query query en UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2555' 8610 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2154' 8611 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1880' 8612 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1916'
8627 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1936'
8629 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2161' 8633 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2596' 8634 observium localhost observium 37 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '864' 8637 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '956'
8638 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1073' 8639 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '578' 8640 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '513' 8572 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '312'
8630 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '430' 8646 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1099' 8647 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '573' 8613 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='20824064',`mempool_perc` ='32.73',`mempool_free` ='42799104',`mempool_total` ='63623168' WHERE `mempool_id` = '2359'
8615 observium localhost observium 12 Query query en UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='27287552',`mempool_perc` ='42.8',`mempool_free` ='36466688',`mempool_total` ='63754240' WHERE `mempool_id` = '1740' 8628 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='22896640',`mempool_perc` ='35.93',`mempool_free` ='40824832',`mempool_total` ='63721472' WHERE `mempool_id` = '2273' 8649 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '161' 8650 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '847'
8605 observium localhost observium 9 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1191' 8625 observium localhost observium 7 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1057' 8597 observium localhost observium 6 Query query en UPDATE `sensors-state` set `sensor_value` ='23.8',`sensor_event` ='ok',`sensor_status` ='',`sensor_last_change` ='1469690416',`sensor_polled` ='1469691635' WHERE `sensor_id` = '5102' 8576 observium localhost observium 4 Query query en UPDATE `ports` set `poll_time` ='1469691627',`poll_period` ='703',`ifInOctets` ='88925',`ifInOctets_rate` ='0',`ifOutOctets` ='88925',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` 7746 observium localhost observium 2 Query query en INSERT INTO `vlans_fdb` (`device_id`,`vlan_id`,`port_id`,`mac_address`,`fdb_status`) VALUES ('701','205','13401','84b59cb47383','learned')
8428 observium localhost observium 2 Query query en UPDATE `ports` set `poll_time` ='1469691592',`poll_period` ='328',`ifInOctets` ='0',`ifInOctets_rate` ='0',`ifOutOctets` ='0',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` ='0',`i
411 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2708','0','1469692947') 421 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2404','0','1469692947') 423 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2119','0','1469692947')
425 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2004','0','1469692947')
426 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2287','0','1469692947') 429 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2888','0','1469692947') 431 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2875','0','1469692947') 432 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2661','0','1469692947') 433 observium localhost observium 24 Query query en INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2759','0','1469692947')
434 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2455','22','1469692947') 435 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2982','0','1469692947') 436 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2120','1','1469692947') 437 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2452','30','1469692947') 439 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1926','0','1469692947') 441 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2447','36','1469692947')
450 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2986','0','1469692947') 451 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2267','0','1469692947') 454 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2937','0','1469692947') 456 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('3006','0','1469692947') 457 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2916','0','1469692947')
458 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2358','0','1469692947')
459 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2825','0','1469692947') 460 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2332','0','1469692947') 471 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1866','0','1469692947') 478 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2121','0','1469692947') 482 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2549','0','1469692947')
On 25 Jul 2016, at 11:35, Lyndon Labuschagne wrote:
This is a walk from the Server that is running at a decent poller speed It complains about LM-SENSORS-MIB, and yet the MIB exists, and is identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
What I also find interesting although possibly nothing is that the returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04 The old server which is running fine, is Ubuntu 15.04
Both use snmpd version 5.7.3
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 MIB search path: /opt/observium/mibs Cannot find module (LM-SENSORS-MIB): At line 1 in (none)
SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62 days, 23:05:46.00 SNMPv2-MIB::sysContact.0 = STRING: networks SNMPv2-MIB::sysName.0 = STRING: domain.name.tld SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location SNMPv2-MIB::sysServices.0 = INTEGER: 79 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORIndex.1 = INTEGER: 1 SNMPv2-MIB::sysORIndex.2 = INTEGER: 2 SNMPv2-MIB::sysORIndex.3 = INTEGER: 3 SNMPv2-MIB::sysORIndex.4 = INTEGER: 4 SNMPv2-MIB::sysORIndex.5 = INTEGER: 5 SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB
SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036 SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB SNMPv2-MIB::sysORID.4 = OID: SNMPv2-SMI::enterprises.10 002.1.1.1.31 SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112 SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP entities SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11 entities. SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe generic objects for network interface sub-layers SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe system resources SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
- Server that is showing issues
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543981700) 62 days, 23:03:37.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "domain.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: "Some GPS Location" iso.3.6.1.2.1.1.7.0 = INTEGER: 79
iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1
iso. 3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1 .3.2 = STRING: "The MIB module for IEEE 802.11 entities."
iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
On 25 Jul 2016, at 09:00, Lyndon Labuschagne wrote:
May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not
found this in debug mode
It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.name.tld variable name in this MIB."
the device is reachable via SNMP as shown in a portion of the snmpwalk below
this is where I am seeing the slowing down during the ifindex scanning as previously posted if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645']
SQL RUNTIME[0.00229788s] SNMP STATUS[FALSE] 6 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
CMD EXITCODE[1] CMD RUNTIME[6.0817s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: ifInMulticastPkts.6
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';]
SQL RUNTIME[0.00058079s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650']
SQL RUNTIME[0.00175095s] SNMP STATUS[FALSE] 7 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
CMD EXITCODE[1] CMD RUNTIME[6.0846s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: ifInMulticastPkts.7
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';] SQL RUNTIME[0.00056005s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656']
SQL RUNTIME[0.00169611s] SNMP STATUS[FALSE] 8 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
CMD EXITCODE[1] CMD RUNTIME[6.0792s] STDOUT[
] STDERR[ Error in packet
Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.8
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';] SQL RUNTIME[0.00071502s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662']
SQL RUNTIME[0.00218582s]
SNMP STATUS[FALSE] 9 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
CMD EXITCODE[1] CMD RUNTIME[6.0803s] STDOUT[
This is a manual snmpwalk
root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.2.1.0 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3
iso. 3.6.1.2.1.2.2.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7 iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8 iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo" iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0" iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1" iso.3.6.1.2.1.2.2.1 .2.4 = STRING: "wifi0"
iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0" iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36" iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36" iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0" iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1" iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24 iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436 iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500 iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286 iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0 iso.3.6.1.2.1.2.2.1 .5.6 = Gauge32: 100000000
iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0
On 25 Jul 2016, at 07:24, Lyndon Labuschagne wrote:
Im going to fire up the server again today and test more, will let you know,
if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
On 25 Jul 2016, at 00:30, Adam Armstrong wrote:
If do you have sufficient buffers and caching for innodb in mysql?
We have some /very/ large installs which saw little impact from the changes.
Adam.
Sent from BlueMail
On 24 Jul 2016, at 20:55, Lyndon Labuschagne wrote: I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne wrote: i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device,
until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec
root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out
83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list
observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list
observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
How big is big? around 18500 ports
On 28 Jul 2016, at 20:03, Adam Armstrong <adama@memetic.orgmailto:adama@memetic.org> wrote:
How big is this installation?
Normally you wouldn't have you think about performance issues until quite a lot of ports.
adam.
On 28/07/2016 18:41:06, Lyndon Labuschagne <lyndonl@mobiletorque.co.zamailto:lyndonl@mobiletorque.co.za> wrote:
I would have assumed the same thing but after numerous tests I just cant see it happening especially not on the Mysql side
After I dropped the DB though things are looking better with a fresh start
MySQL test 30 seconds limit
OLTP test statistics: queries performed: read: 591948 write: 211360 other: 84547 total: 887855 transactions: 42265 (1407.82 per sec.) deadlocks: 17 (0.57 per sec.) read/write requests: 803308 (26757.71 per sec.) other operations: 84547 (2816.21 per sec.)
root@observium:~# sysbench --test=cpu --cpu-max-prime=20000 run sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options: Number of threads: 1
Doing CPU performance benchmark
Threads started! Done.
Maximum prime number checked in CPU test: 20000
Test execution summary: total time: 30.9549s total number of events: 10000 total time taken by event execution: 30.9459 per-request statistics: min: 2.49ms avg: 3.09ms max: 6.87ms approx. 95 percentile: 4.41ms
Threads fairness: events (avg/stddev): 10000.0000/0.00 execution time (avg/stddev): 30.9459/0.00
Running the test with following options: Number of threads: 1 Initializing random number generator from timer.
Extra file open flags: 0 128 files, 1.1719Gb each 150Gb total file size Block size 16Kb Number of random requests for random IO: 0 Read/Write ratio for combined random IO test: 1.50 Periodic FSYNC enabled, calling fsync() each 100 requests. Calling fsync() at the end of test, Enabled. Using synchronous I/O mode Doing random r/w test Threads started! Time limit exceeded, exiting... Done.
Operations performed: 20837 Read, 13891 Write, 44416 Other = 79144 Total Read 325.58Mb Written 217.05Mb Total transferred 542.62Mb (1.8087Mb/sec) 115.76 Requests/sec executed
Test execution summary: total time: 300.0011s total number of events: 34728 total time taken by event execution: 124.5346 per-request statistics: min: 0.00ms avg: 3.59ms max: 1548.47ms approx. 95 percentile: 16.26ms
Threads fairness: events (avg/stddev): 34728.0000/0.00 execution time (avg/stddev): 124.5346/0.00
On 28 Jul 2016, at 19:01, Adam Armstrong wrote:
I have no idea. I'm not sure why the issues you have now exist :)
It sounds like you probably have some pretty severe I/O bottleneck somewhere.
Adam.
Sent from BlueMail
On 28 Jul 2016, at 11:18, Lyndon Labuschagne wrote: so the speeds are going to get worse?
On 28 Jul 2016, at 17:29, Adam Armstrong wrote:
Oh. Stable!
It's entirely possibly you don't yet have the changes which have the potential to make things "slow" ;)
Adam.
Sent from BlueMail
On 28 Jul 2016, at 07:49, Lyndon Labuschagne wrote: Hi Yes all up to date
Is there any particular things I should be looking at / for, I have totally killed off the DB and it is currently building back up, discovering hosts etc a possible kick ass mysql config file or other hints
root@observium:/opt/observium# svn update Updating '.': At revision 8013.
root@observium:/opt/observium# ./discovery.php -u
/ _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___ | | | || '_ \ / __| / _ | '__|\ \ / /| || | | || '_ ` _ \ | |_| || |_) |__ | __/| | \ V / | || |_| || | | | | | ___/ |_.__/ |___/ ___||_| _/ |_| __,_||_| |_| |_| Observium Professional 0.16.7.8003 http://www.observium.org
-- Database is up to date.
On 28 Jul 2016, at 14:39, Adam Armstrong wrote:
Hmm. I don't understand how they can be particularly slow. They're generally small tables.
Are you sure that's running the latest version? The ports table UPDATE seems fairly short.
adam. On 28/07/2016 09:06:42, Lyndon Labuschagne wrote:
The continuing saga
It seem that the "processors-state” integration waits for a long time, most other queries / read / writes are being completed in under 5 seconds, as shown below these are “Waiting” for 40+ seconds, these seems to be linked to JOINS in most cases Does anyone have an idea on how to speed these up from a tuning perspective?
8603 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2740'
8604 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2554' 8609 observium localhost observium 40 Query query en UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2555' 8610 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2154' 8611 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1880' 8612 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1916'
8627 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1936'
8629 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2161' 8633 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2596' 8634 observium localhost observium 37 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '864' 8637 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '956'
8638 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1073' 8639 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '578' 8640 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '513' 8572 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '312'
8630 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '430' 8646 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1099' 8647 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '573' 8613 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='20824064',`mempool_perc` ='32.73',`mempool_free` ='42799104',`mempool_total` ='63623168' WHERE `mempool_id` = '2359'
8615 observium localhost observium 12 Query query en UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='27287552',`mempool_perc` ='42.8',`mempool_free` ='36466688',`mempool_total` ='63754240' WHERE `mempool_id` = '1740' 8628 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='22896640',`mempool_perc` ='35.93',`mempool_free` ='40824832',`mempool_total` ='63721472' WHERE `mempool_id` = '2273' 8649 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '161' 8650 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '847'
8605 observium localhost observium 9 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1191' 8625 observium localhost observium 7 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1057' 8597 observium localhost observium 6 Query query en UPDATE `sensors-state` set `sensor_value` ='23.8',`sensor_event` ='ok',`sensor_status` ='',`sensor_last_change` ='1469690416',`sensor_polled` ='1469691635' WHERE `sensor_id` = '5102' 8576 observium localhost observium 4 Query query en UPDATE `ports` set `poll_time` ='1469691627',`poll_period` ='703',`ifInOctets` ='88925',`ifInOctets_rate` ='0',`ifOutOctets` ='88925',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` 7746 observium localhost observium 2 Query query en INSERT INTO `vlans_fdb` (`device_id`,`vlan_id`,`port_id`,`mac_address`,`fdb_status`) VALUES ('701','205','13401','84b59cb47383','learned')
8428 observium localhost observium 2 Query query en UPDATE `ports` set `poll_time` ='1469691592',`poll_period` ='328',`ifInOctets` ='0',`ifInOctets_rate` ='0',`ifOutOctets` ='0',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` ='0',`i
411 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2708','0','1469692947') 421 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2404','0','1469692947') 423 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2119','0','1469692947')
425 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2004','0','1469692947')
426 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2287','0','1469692947') 429 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2888','0','1469692947') 431 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2875','0','1469692947') 432 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2661','0','1469692947') 433 observium localhost observium 24 Query query en INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2759','0','1469692947')
434 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2455','22','1469692947') 435 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2982','0','1469692947') 436 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2120','1','1469692947') 437 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2452','30','1469692947') 439 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1926','0','1469692947') 441 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2447','36','1469692947')
450 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2986','0','1469692947') 451 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2267','0','1469692947') 454 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2937','0','1469692947') 456 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('3006','0','1469692947') 457 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2916','0','1469692947')
458 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2358','0','1469692947')
459 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2825','0','1469692947') 460 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2332','0','1469692947') 471 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1866','0','1469692947') 478 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2121','0','1469692947') 482 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2549','0','1469692947')
On 25 Jul 2016, at 11:35, Lyndon Labuschagne wrote:
This is a walk from the Server that is running at a decent poller speed It complains about LM-SENSORS-MIB, and yet the MIB exists, and is identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
What I also find interesting although possibly nothing is that the returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04 The old server which is running fine, is Ubuntu 15.04
Both use snmpd version 5.7.3
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 MIB search path: /opt/observium/mibs Cannot find module (LM-SENSORS-MIB): At line 1 in (none)
SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62 days, 23:05:46.00 SNMPv2-MIB::sysContact.0 = STRING: networks SNMPv2-MIB::sysName.0 = STRING: domain.name.tld SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location SNMPv2-MIB::sysServices.0 = INTEGER: 79 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORIndex.1 = INTEGER: 1 SNMPv2-MIB::sysORIndex.2 = INTEGER: 2 SNMPv2-MIB::sysORIndex.3 = INTEGER: 3 SNMPv2-MIB::sysORIndex.4 = INTEGER: 4 SNMPv2-MIB::sysORIndex.5 = INTEGER: 5 SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB
SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036 SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB SNMPv2-MIB::sysORID.4 = OID: SNMPv2-SMI::enterprises.10 002.1.1.1.31 SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112 SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP entities SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11 entities. SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe generic objects for network interface sub-layers SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe system resources SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
- Server that is showing issues
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543981700) 62 days, 23:03:37.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "domain.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: "Some GPS Location" iso.3.6.1.2.1.1.7.0 = INTEGER: 79
iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1
iso. 3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1 .3.2 = STRING: "The MIB module for IEEE 802.11 entities."
iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
On 25 Jul 2016, at 09:00, Lyndon Labuschagne wrote:
May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not
found this in debug mode
It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.name.tld variable name in this MIB."
the device is reachable via SNMP as shown in a portion of the snmpwalk below
this is where I am seeing the slowing down during the ifindex scanning as previously posted if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645']
SQL RUNTIME[0.00229788s] SNMP STATUS[FALSE] 6 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
CMD EXITCODE[1] CMD RUNTIME[6.0817s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: ifInMulticastPkts.6
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';]
SQL RUNTIME[0.00058079s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650']
SQL RUNTIME[0.00175095s] SNMP STATUS[FALSE] 7 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
CMD EXITCODE[1] CMD RUNTIME[6.0846s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: ifInMulticastPkts.7
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';] SQL RUNTIME[0.00056005s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656']
SQL RUNTIME[0.00169611s] SNMP STATUS[FALSE] 8 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
CMD EXITCODE[1] CMD RUNTIME[6.0792s] STDOUT[
] STDERR[ Error in packet
Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.8
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';] SQL RUNTIME[0.00071502s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662']
SQL RUNTIME[0.00218582s]
SNMP STATUS[FALSE] 9 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
CMD EXITCODE[1] CMD RUNTIME[6.0803s] STDOUT[
This is a manual snmpwalk
root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.2.1.0 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3
iso. 3.6.1.2.1.2.2.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7 iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8 iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo" iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0" iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1" iso.3.6.1.2.1.2.2.1 .2.4 = STRING: "wifi0"
iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0" iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36" iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36" iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0" iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1" iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24 iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436 iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500 iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286 iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0 iso.3.6.1.2.1.2.2.1 .5.6 = Gauge32: 100000000
iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0
On 25 Jul 2016, at 07:24, Lyndon Labuschagne wrote:
Im going to fire up the server again today and test more, will let you know,
if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
On 25 Jul 2016, at 00:30, Adam Armstrong wrote:
If do you have sufficient buffers and caching for innodb in mysql?
We have some /very/ large installs which saw little impact from the changes.
Adam.
Sent from BlueMail
On 24 Jul 2016, at 20:55, Lyndon Labuschagne wrote: I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne wrote: i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device,
until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec
root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out
83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list
observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list
observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Have you tried to tune your mysql?
Try mysqltuner.pl, it's on github.
Disable mysql query cache, it have a giant lock and generally slows down things instead of speeds up.
-- Eduardo Schoedler
Em quinta-feira, 28 de julho de 2016, Lyndon Labuschagne < lyndonl@mobiletorque.co.za> escreveu:
Hi Yes all up to date
Is there any particular things I should be looking at / for, I have totally killed off the DB and it is currently building back up, discovering hosts etc a possible kick ass mysql config file or other hints
root@observium:/opt/observium# svn update Updating '.': At revision 8013.
root@observium:/opt/observium# ./discovery.php -u
/ _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___ | | | || '_ \ / __| / _ | '__|\ \ / /| || | | || '_ ` _ \ | |_| || |_) |__ | __/| | \ V / | || |_| || | | | | | ___/ |_.__/ |___/ ___||_| _/ |_| __,_||_| |_| |_| Observium Professional 0.16.7.8003 http://www.observium.org
-- Database is up to date.
On 28 Jul 2016, at 14:39, Adam Armstrong <adama@memetic.org
javascript:;> wrote:
Hmm. I don't understand how they can be particularly slow. They're
generally small tables.
Are you sure that's running the latest version? The ports table UPDATE
seems fairly short.
adam.
On 28/07/2016 09:06:42, Lyndon Labuschagne <lyndonl@mobiletorque.co.za
javascript:;> wrote:
The continuing saga
It seem that the "processors-state” integration waits for a long time,
most other queries / read / writes are being completed in under 5 seconds, as shown below these are “Waiting” for 40+ seconds, these seems to be linked to JOINS in most cases
Does anyone have an idea on how to speed these up from a tuning
perspective?
8603 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2740'
8604 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2554'
8609 observium localhost observium 40 Query query en UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2555'
8610 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2154'
8611 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1880'
8612 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1916'
8627 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1936'
8629 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2161'
8633 observium localhost observium 40 Query Waiting UPDATE
`processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2596'
8634 observium localhost observium 37 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '864'
8637 observium localhost observium 35 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '956'
8638 observium localhost observium 35 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1073'
8639 observium localhost observium 35 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '578'
8640 observium localhost observium 35 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '513'
8572 observium localhost observium 14 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '312'
8630 observium localhost observium 14 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '430'
8646 observium localhost observium 13 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1099'
8647 observium localhost observium 13 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '573'
8613 observium localhost observium 12 Query Waiting UPDATE
`mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='20824064',`mempool_perc` ='32.73',`mempool_free` ='42799104',`mempool_total` ='63623168' WHERE `mempool_id` = '2359'
8615 observium localhost observium 12 Query query en UPDATE
`mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='27287552',`mempool_perc` ='42.8',`mempool_free` ='36466688',`mempool_total` ='63754240' WHERE `mempool_id` = '1740'
8628 observium localhost observium 12 Query Waiting UPDATE
`mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='22896640',`mempool_perc` ='35.93',`mempool_free` ='40824832',`mempool_total` ='63721472' WHERE `mempool_id` = '2273'
8649 observium localhost observium 10 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '161'
8650 observium localhost observium 10 Query Waiting SELECT
`processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '847'
8605 observium localhost observium 9 Query Waiting SELECT * FROM
`mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1191'
8625 observium localhost observium 7 Query Waiting SELECT * FROM
`mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1057'
8597 observium localhost observium 6 Query query en UPDATE
`sensors-state` set `sensor_value` ='23.8',`sensor_event` ='ok',`sensor_status` ='',`sensor_last_change` ='1469690416',`sensor_polled` ='1469691635' WHERE `sensor_id` = '5102'
8576 observium localhost observium 4 Query query en UPDATE `ports` set
`poll_time` ='1469691627',`poll_period` ='703',`ifInOctets` ='88925',`ifInOctets_rate` ='0',`ifOutOctets` ='88925',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate`
7746 observium localhost observium 2 Query query en INSERT INTO
`vlans_fdb` (`device_id`,`vlan_id`,`port_id`,`mac_address`,`fdb_status`) VALUES ('701','205','13401','84b59cb47383','learned')
8428 observium localhost observium 2 Query query en UPDATE `ports` set
`poll_time` ='1469691592',`poll_period` ='328',`ifInOctets` ='0',`ifInOctets_rate` ='0',`ifOutOctets` ='0',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` ='0',`i
411 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2708','0','1469692947')
421 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2404','0','1469692947')
423 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2119','0','1469692947')
425 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2004','0','1469692947')
426 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2287','0','1469692947')
429 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2888','0','1469692947')
431 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2875','0','1469692947')
432 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2661','0','1469692947')
433 observium localhost observium 24 Query query en INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2759','0','1469692947')
434 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2455','22','1469692947')
435 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2982','0','1469692947')
436 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2120','1','1469692947')
437 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2452','30','1469692947')
439 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1926','0','1469692947')
441 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2447','36','1469692947')
450 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2986','0','1469692947')
451 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2267','0','1469692947')
454 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2937','0','1469692947')
456 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('3006','0','1469692947')
457 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2916','0','1469692947')
458 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2358','0','1469692947')
459 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2825','0','1469692947')
460 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2332','0','1469692947')
471 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1866','0','1469692947')
478 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2121','0','1469692947')
482 observium localhost observium 24 Query Waiting INSERT INTO
`processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2549','0','1469692947')
On 25 Jul 2016, at 11:35, Lyndon Labuschagne wrote:
This is a walk from the Server that is running at a decent poller
speed
It complains about LM-SENSORS-MIB, and yet the MIB exists, and is
identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
What I also find interesting although possibly nothing is that the
returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04
The old server which is running fine, is Ubuntu 15.04
Both use snmpd version 5.7.3
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 MIB search path: /opt/observium/mibs Cannot find module (LM-SENSORS-MIB): At line 1 in (none) SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65 #1 Thu Jul 16
11:50:24 EEST 2015 mips
SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62 days,
23:05:46.00
SNMPv2-MIB::sysContact.0 = STRING: networks SNMPv2-MIB::sysName.0 = STRING: domain.name.tld SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location SNMPv2-MIB::sysServices.0 = INTEGER: 79 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORIndex.1 = INTEGER: 1 SNMPv2-MIB::sysORIndex.2 = INTEGER: 2 SNMPv2-MIB::sysORIndex.3 = INTEGER: 3 SNMPv2-MIB::sysORIndex.4 = INTEGER: 4 SNMPv2-MIB::sysORIndex.5 = INTEGER: 5 SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036 SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB SNMPv2-MIB::sysORID.4 = OID: SNMPv2-SMI::enterprises.10002.1.1.1.31 SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112 SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP entities SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11
entities.
SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe generic
objects for network interface sub-layers
SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe system
resources
SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
- Server that is showing issues
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24
EEST 2015 mips"
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543981700) 62 days, 23:03:37.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "domain.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: "Some GPS Location" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11
entities."
iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic
objects for network interface sub-layers"
iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system
resources"
iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
On 25 Jul 2016, at 09:00, Lyndon Labuschagne wrote:
May not betheDB after all most of the SQL RUNTIME stats are
0.000xxxx so its probably not
found this in debug mode
It seems the poller / mib is freaking out about "Reason:
(noSuchName) There is no such dns.name.tld variable name in this MIB."
the device is reachable via SNMP as shown in a portion of the
snmpwalk below
this is where I am seeing the slowing down during the ifindex
scanning as previously posted
> if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to
be slow, even when the radio is directly connected to the hosting switch as a test.
> The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8
9 (that takes 40 to 50 seconds)
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated`
='1469428547' WHERE `error_id` = '6645']
SQL RUNTIME[0.00229788s] SNMP STATUS[FALSE] 6 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M
/opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
CMD EXITCODE[1] CMD RUNTIME[6.0817s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.6
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND
`error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';]
SQL RUNTIME[0.00058079s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated`
='1469428553' WHERE `error_id` = '6650']
SQL RUNTIME[0.00175095s] SNMP STATUS[FALSE] 7 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M
/opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
CMD EXITCODE[1] CMD RUNTIME[6.0846s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.7
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND
`error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';]
SQL RUNTIME[0.00056005s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated`
='1469428559' WHERE `error_id` = '6656']
SQL RUNTIME[0.00169611s] SNMP STATUS[FALSE] 8 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M
/opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
CMD EXITCODE[1] CMD RUNTIME[6.0792s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.8
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND
`error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';]
SQL RUNTIME[0.00071502s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated`
='1469428565' WHERE `error_id` = '6662']
SQL RUNTIME[0.00218582s] SNMP STATUS[FALSE] 9 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M
/opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
CMD EXITCODE[1] CMD RUNTIME[6.0803s] STDOUT[
This is a manual snmpwalk
root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16
11:50:24 EEST 2015 mips"
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11
entities."
iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe
generic objects for network interface sub-layers"
iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system
resources"
iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.2.1.0 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.2.2.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7 iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8 iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo" iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0" iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1" iso.3.6.1.2.1.2.2.1.2.4 = STRING: "wifi0" iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0" iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36" iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36" iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0" iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1" iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24 iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436 iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500 iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286 iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.6 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0
On 25 Jul 2016, at 07:24, Lyndon Labuschagne wrote:
Im going to fire up the server again today and test more, will let
you know,
if there anyway of running the poller and but not writing to the
DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
> On 25 Jul 2016, at 00:30, Adam Armstrong wrote: > > If do you have sufficient buffers and caching for innodb in mysql? > > We have some /very/ large installs which saw little impact from
the changes.
> > Adam. > > Sent from BlueMail > > On 24 Jul 2016, at 20:55, Lyndon Labuschagne wrote: > I have been performance tuning all day with little to no affect > > > its just very strange how a XEN VM out performs bare metal, it may
also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
> > > > On 24 Jul 2016, at 21:47, Adam Armstrong wrote: > > We've changed the way the ports table works, removing memory
tables.
> > It's likely that you now need more mysql resources, since were now
writing to an innodb table instead of a memory table.
> > Odd, really. It didn't seem to have a huge effect on my dev
install or some other very large installs.
> > Perhaps you could investigate mysql performance tuning? :) > > Adam. > > Sent from BlueMail > > > On 24 Jul 2016, at 19:58, Lyndon Labuschagne wrote: > i went back > to use the old database server on the VM and all was much better,
running pollers at 10 to 15 seconds per device,
> > until I upgraded the db by using discovery -u > > back in the same boat again > > On 24 Jul 2016, at 09:56, Lyndon Labuschagne wrote: > > Ok if I poll a router / switch / server etc it seems to be ok, > a full scan on a 7606 takes around 45 seconds, > > > if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to
be slow, even when the radio is directly connected to the hosting switch as a test.
> The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8
9 (that takes 40 to 50 seconds)
> > o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName
ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts
> ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts
ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts
> ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts
ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
> > > > The the other thing that had changed is that I was running a
slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
> > On 24 Jul 2016, at 09:16, Lyndon Labuschagne wrote: > > I know its bad form to reply to your own email but here is some
more information re drive performance
> > > root@observium:/opt/observium/rrd# hdparm -Tt
/dev/mapper/observium--vg-root
> > > /dev/mapper/observium--vg-root: > Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec > Timing buffered disk reads: 950 MB in 3.01 seconds > = 315.94 MB/sec > > root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output
bs=8k count=10k; rm -f /tmp/output
> 10240+0 records in > 10240+0 records out > 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s > > > On 24 Jul 2016, at 09:13, Lyndon Labuschagne wrote: > > Hi All > > I have moved from a VM server to a physical server. as the VM was
taking a bit of a pounding, graph dropouts etc
> The physical server is a dual quad core Xeon, 2.66Ghz with 128GB
of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
> > > Will all of this my average poller-wrapper time is around 60 to 70
seconds per device
> if I manually run an snmpwalk a single device takes around 6
seconds
> > I have also disabled the ramdisk, rrdcached etc and I am not
seeing much of a difference either way
> > I have around 900 devices and the VM server was way > faster, any idea where to start looking for a bottleneck? > > > Regards > Lyndon > > observium mailing list > observium@observium.org javascript:; > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > > observium mailing list > observium@observium.org javascript:; > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > > observium mailing list > observium@observium.org javascript:; > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > > observium mailing list > observium@observium.org javascript:; > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > observium mailing list > observium@observium.org javascript:; > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > > observium mailing list > observium@observium.org javascript:; > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > _______________________________________________ > observium mailing list > observium@observium.org javascript:; > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org javascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org javascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org javascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org javascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org javascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org javascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
I have been using it extensively and it has made some improvements, I will try disabling the query cache Thank you for the input
On 28 Jul 2016, at 20:04, Eduardo Schoedler <listas@esds.com.brmailto:listas@esds.com.br> wrote:
Have you tried to tune your mysql?
Try mysqltuner.plhttp://mysqltuner.pl/, it's on github.
Disable mysql query cache, it have a giant lock and generally slows down things instead of speeds up.
-- Eduardo Schoedler
Em quinta-feira, 28 de julho de 2016, Lyndon Labuschagne <lyndonl@mobiletorque.co.zamailto:lyndonl@mobiletorque.co.za> escreveu: Hi Yes all up to date
Is there any particular things I should be looking at / for, I have totally killed off the DB and it is currently building back up, discovering hosts etc a possible kick ass mysql config file or other hints
root@observium:/opt/observium# svn update Updating '.': At revision 8013.
root@observium:/opt/observium# ./discovery.php -u
___ _ _ / _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___ | | | || '_ \ / __| / _ | '__|\ \ / /| || | | || '_ ` _ \ | |_| || |_) |__ | __/| | \ V / | || |_| || | | | | | ___/ |_.__/ |___/ ___||_| _/ |_| __,_||_| |_| |_| Observium Professional 0.16.7.8003 http://www.observium.orghttp://www.observium.org/
-- Database is up to date.
On 28 Jul 2016, at 14:39, Adam Armstrong <adama@memetic.orgjavascript:;> wrote:
Hmm. I don't understand how they can be particularly slow. They're generally small tables.
Are you sure that's running the latest version? The ports table UPDATE seems fairly short.
adam.
On 28/07/2016 09:06:42, Lyndon Labuschagne <lyndonl@mobiletorque.co.zajavascript:;> wrote:
The continuing saga
It seem that the "processors-state” integration waits for a long time, most other queries / read / writes are being completed in under 5 seconds, as shown below these are “Waiting” for 40+ seconds, these seems to be linked to JOINS in most cases Does anyone have an idea on how to speed these up from a tuning perspective?
8603 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2740' 8604 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2554' 8609 observium localhost observium 40 Query query en UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2555' 8610 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2154' 8611 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1880' 8612 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1916' 8627 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1936' 8629 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2161' 8633 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2596' 8634 observium localhost observium 37 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '864' 8637 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '956' 8638 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1073' 8639 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '578' 8640 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '513' 8572 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '312' 8630 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '430' 8646 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1099' 8647 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '573' 8613 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='20824064',`mempool_perc` ='32.73',`mempool_free` ='42799104',`mempool_total` ='63623168' WHERE `mempool_id` = '2359' 8615 observium localhost observium 12 Query query en UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='27287552',`mempool_perc` ='42.8',`mempool_free` ='36466688',`mempool_total` ='63754240' WHERE `mempool_id` = '1740' 8628 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='22896640',`mempool_perc` ='35.93',`mempool_free` ='40824832',`mempool_total` ='63721472' WHERE `mempool_id` = '2273' 8649 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '161' 8650 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '847' 8605 observium localhost observium 9 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1191' 8625 observium localhost observium 7 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1057' 8597 observium localhost observium 6 Query query en UPDATE `sensors-state` set `sensor_value` ='23.8',`sensor_event` ='ok',`sensor_status` ='',`sensor_last_change` ='1469690416',`sensor_polled` ='1469691635' WHERE `sensor_id` = '5102' 8576 observium localhost observium 4 Query query en UPDATE `ports` set `poll_time` ='1469691627',`poll_period` ='703',`ifInOctets` ='88925',`ifInOctets_rate` ='0',`ifOutOctets` ='88925',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` 7746 observium localhost observium 2 Query query en INSERT INTO `vlans_fdb` (`device_id`,`vlan_id`,`port_id`,`mac_address`,`fdb_status`) VALUES ('701','205','13401','84b59cb47383','learned') 8428 observium localhost observium 2 Query query en UPDATE `ports` set `poll_time` ='1469691592',`poll_period` ='328',`ifInOctets` ='0',`ifInOctets_rate` ='0',`ifOutOctets` ='0',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` ='0',`i
411 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2708','0','1469692947') 421 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2404','0','1469692947') 423 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2119','0','1469692947') 425 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2004','0','1469692947') 426 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2287','0','1469692947') 429 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2888','0','1469692947') 431 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2875','0','1469692947') 432 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2661','0','1469692947') 433 observium localhost observium 24 Query query en INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2759','0','1469692947') 434 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2455','22','1469692947') 435 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2982','0','1469692947') 436 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2120','1','1469692947') 437 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2452','30','1469692947') 439 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1926','0','1469692947') 441 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2447','36','1469692947') 450 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2986','0','1469692947') 451 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2267','0','1469692947') 454 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2937','0','1469692947') 456 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('3006','0','1469692947') 457 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2916','0','1469692947') 458 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2358','0','1469692947') 459 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2825','0','1469692947') 460 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2332','0','1469692947') 471 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1866','0','1469692947') 478 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2121','0','1469692947') 482 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2549','0','1469692947')
On 25 Jul 2016, at 11:35, Lyndon Labuschagne wrote:
This is a walk from the Server that is running at a decent poller speed It complains about LM-SENSORS-MIB, and yet the MIB exists, and is identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
What I also find interesting although possibly nothing is that the returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04 The old server which is running fine, is Ubuntu 15.04
Both use snmpd version 5.7.3
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 MIB search path: /opt/observium/mibs Cannot find module (LM-SENSORS-MIB): At line 1 in (none) SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62 days, 23:05:46.00 SNMPv2-MIB::sysContact.0 = STRING: networks SNMPv2-MIB::sysName.0 = STRING: domain.name.tld SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location SNMPv2-MIB::sysServices.0 = INTEGER: 79 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORIndex.1 = INTEGER: 1 SNMPv2-MIB::sysORIndex.2 = INTEGER: 2 SNMPv2-MIB::sysORIndex.3 = INTEGER: 3 SNMPv2-MIB::sysORIndex.4 = INTEGER: 4 SNMPv2-MIB::sysORIndex.5 = INTEGER: 5 SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036 SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB SNMPv2-MIB::sysORID.4 = OID: SNMPv2-SMI::enterprises.10002.1.1.1.31 SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112 SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP entities SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11 entities. SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe generic objects for network interface sub-layers SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe system resources SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
- Server that is showing issues
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543981700) 62 days, 23:03:37.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "domain.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: "Some GPS Location" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
On 25 Jul 2016, at 09:00, Lyndon Labuschagne wrote:
May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not
found this in debug mode
It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.name.tld variable name in this MIB."
the device is reachable via SNMP as shown in a portion of the snmpwalk below
this is where I am seeing the slowing down during the ifindex scanning as previously posted
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645'] SQL RUNTIME[0.00229788s] SNMP STATUS[FALSE] 6 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
CMD EXITCODE[1] CMD RUNTIME[6.0817s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.6
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';] SQL RUNTIME[0.00058079s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650'] SQL RUNTIME[0.00175095s] SNMP STATUS[FALSE] 7 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
CMD EXITCODE[1] CMD RUNTIME[6.0846s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.7
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';] SQL RUNTIME[0.00056005s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656'] SQL RUNTIME[0.00169611s] SNMP STATUS[FALSE] 8 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
CMD EXITCODE[1] CMD RUNTIME[6.0792s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.8
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';] SQL RUNTIME[0.00071502s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662'] SQL RUNTIME[0.00218582s] SNMP STATUS[FALSE] 9 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
CMD EXITCODE[1] CMD RUNTIME[6.0803s] STDOUT[
This is a manual snmpwalk
root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.2.1.0 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.2.2.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7 iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8 iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo" iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0" iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1" iso.3.6.1.2.1.2.2.1.2.4 = STRING: "wifi0" iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0" iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36" iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36" iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0" iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1" iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24 iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436 iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500 iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286 iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.6 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0
On 25 Jul 2016, at 07:24, Lyndon Labuschagne wrote:
Im going to fire up the server again today and test more, will let you know,
if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
On 25 Jul 2016, at 00:30, Adam Armstrong wrote:
If do you have sufficient buffers and caching for innodb in mysql?
We have some /very/ large installs which saw little impact from the changes.
Adam.
Sent from BlueMail
On 24 Jul 2016, at 20:55, Lyndon Labuschagne wrote: I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne wrote: i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device,
until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec
root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Eduardo Schoedler
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
I disabled query cache and let it run over night, the average climbed to about 280 seconds with 6700 ports (this is my physical server, it does not have all 18000+ yet) I then reenabled query cache this morning, and the poller on the first 2 runs are back down to 180 seconds or so.
so im my case the query cache works for me, the server does have around 60GB if free memory at any given time
On 28 Jul 2016, at 20:04, Eduardo Schoedler <listas@esds.com.brmailto:listas@esds.com.br> wrote:
Have you tried to tune your mysql?
Try mysqltuner.plhttp://mysqltuner.pl/, it's on github.
Disable mysql query cache, it have a giant lock and generally slows down things instead of speeds up.
-- Eduardo Schoedler
Em quinta-feira, 28 de julho de 2016, Lyndon Labuschagne <lyndonl@mobiletorque.co.zamailto:lyndonl@mobiletorque.co.za> escreveu: Hi Yes all up to date
Is there any particular things I should be looking at / for, I have totally killed off the DB and it is currently building back up, discovering hosts etc a possible kick ass mysql config file or other hints
root@observium:/opt/observium# svn update Updating '.': At revision 8013.
root@observium:/opt/observium# ./discovery.php -u
___ _ _ / _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___ | | | || '_ \ / __| / _ | '__|\ \ / /| || | | || '_ ` _ \ | |_| || |_) |__ | __/| | \ V / | || |_| || | | | | | ___/ |_.__/ |___/ ___||_| _/ |_| __,_||_| |_| |_| Observium Professional 0.16.7.8003 http://www.observium.orghttp://www.observium.org/
-- Database is up to date.
On 28 Jul 2016, at 14:39, Adam Armstrong <adama@memetic.orgjavascript:;> wrote:
Hmm. I don't understand how they can be particularly slow. They're generally small tables.
Are you sure that's running the latest version? The ports table UPDATE seems fairly short.
adam.
On 28/07/2016 09:06:42, Lyndon Labuschagne <lyndonl@mobiletorque.co.zajavascript:;> wrote:
The continuing saga
It seem that the "processors-state” integration waits for a long time, most other queries / read / writes are being completed in under 5 seconds, as shown below these are “Waiting” for 40+ seconds, these seems to be linked to JOINS in most cases Does anyone have an idea on how to speed these up from a tuning perspective?
8603 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2740' 8604 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2554' 8609 observium localhost observium 40 Query query en UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2555' 8610 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2154' 8611 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1880' 8612 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1916' 8627 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '1936' 8629 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2161' 8633 observium localhost observium 40 Query Waiting UPDATE `processors-state` set `processor_usage` ='0',`processor_polled` ='1469691601' WHERE `processor_id` = '2596' 8634 observium localhost observium 37 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '864' 8637 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '956' 8638 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1073' 8639 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '578' 8640 observium localhost observium 35 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '513' 8572 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '312' 8630 observium localhost observium 14 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '430' 8646 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '1099' 8647 observium localhost observium 13 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '573' 8613 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='20824064',`mempool_perc` ='32.73',`mempool_free` ='42799104',`mempool_total` ='63623168' WHERE `mempool_id` = '2359' 8615 observium localhost observium 12 Query query en UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='27287552',`mempool_perc` ='42.8',`mempool_free` ='36466688',`mempool_total` ='63754240' WHERE `mempool_id` = '1740' 8628 observium localhost observium 12 Query Waiting UPDATE `mempools-state` set `mempool_polled` ='1469691629',`mempool_used` ='22896640',`mempool_perc` ='35.93',`mempool_free` ='40824832',`mempool_total` ='63721472' WHERE `mempool_id` = '2273' 8649 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '161' 8650 observium localhost observium 10 Query Waiting SELECT `processors`.*, `processors-state`.`processor_usage`, `processors-state`.`processor_polled` FROM `processors` LEFT JOIN `processors-state` USING(`processor_id`) WHERE `device_id` = '847' 8605 observium localhost observium 9 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1191' 8625 observium localhost observium 7 Query Waiting SELECT * FROM `mempools` LEFT JOIN `mempools-state` USING (`mempool_id`) WHERE `device_id` = '1057' 8597 observium localhost observium 6 Query query en UPDATE `sensors-state` set `sensor_value` ='23.8',`sensor_event` ='ok',`sensor_status` ='',`sensor_last_change` ='1469690416',`sensor_polled` ='1469691635' WHERE `sensor_id` = '5102' 8576 observium localhost observium 4 Query query en UPDATE `ports` set `poll_time` ='1469691627',`poll_period` ='703',`ifInOctets` ='88925',`ifInOctets_rate` ='0',`ifOutOctets` ='88925',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` 7746 observium localhost observium 2 Query query en INSERT INTO `vlans_fdb` (`device_id`,`vlan_id`,`port_id`,`mac_address`,`fdb_status`) VALUES ('701','205','13401','84b59cb47383','learned') 8428 observium localhost observium 2 Query query en UPDATE `ports` set `poll_time` ='1469691592',`poll_period` ='328',`ifInOctets` ='0',`ifInOctets_rate` ='0',`ifOutOctets` ='0',`ifOutOctets_rate` ='0',`ifInErrors` ='0',`ifInErrors_rate` ='0',`ifInErrors_delta` ='0',`ifOutErrors` ='0',`ifOutErrors_rate` ='0',`i
411 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2708','0','1469692947') 421 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2404','0','1469692947') 423 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2119','0','1469692947') 425 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2004','0','1469692947') 426 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2287','0','1469692947') 429 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2888','0','1469692947') 431 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2875','0','1469692947') 432 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2661','0','1469692947') 433 observium localhost observium 24 Query query en INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2759','0','1469692947') 434 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2455','22','1469692947') 435 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2982','0','1469692947') 436 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2120','1','1469692947') 437 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2452','30','1469692947') 439 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1926','0','1469692947') 441 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2447','36','1469692947') 450 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2986','0','1469692947') 451 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2267','0','1469692947') 454 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2937','0','1469692947') 456 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('3006','0','1469692947') 457 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2916','0','1469692947') 458 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2358','0','1469692947') 459 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2825','0','1469692947') 460 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2332','0','1469692947') 471 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('1866','0','1469692947') 478 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2121','0','1469692947') 482 observium localhost observium 24 Query Waiting INSERT INTO `processors-state` (`processor_id`,`processor_usage`,`processor_polled`) VALUES ('2549','0','1469692947')
On 25 Jul 2016, at 11:35, Lyndon Labuschagne wrote:
This is a walk from the Server that is running at a decent poller speed It complains about LM-SENSORS-MIB, and yet the MIB exists, and is identical to the other servers file, the other server seems fine and does not show any errors but is much slower on polling
What I also find interesting although possibly nothing is that the returned details are quoted in “ “ on the server that is can’t find the LM-SENSORS-MIB Ubuntu 16.04 The old server which is running fine, is Ubuntu 15.04
Both use snmpd version 5.7.3
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 MIB search path: /opt/observium/mibs Cannot find module (LM-SENSORS-MIB): At line 1 in (none) SNMPv2-MIB::sysDescr.0 = STRING: Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.41112.1.4 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (543994600) 62 days, 23:05:46.00 SNMPv2-MIB::sysContact.0 = STRING: networks SNMPv2-MIB::sysName.0 = STRING: domain.name.tld SNMPv2-MIB::sysLocation.0 = STRING: Some GPS Location SNMPv2-MIB::sysServices.0 = INTEGER: 79 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORIndex.1 = INTEGER: 1 SNMPv2-MIB::sysORIndex.2 = INTEGER: 2 SNMPv2-MIB::sysORIndex.3 = INTEGER: 3 SNMPv2-MIB::sysORIndex.4 = INTEGER: 4 SNMPv2-MIB::sysORIndex.5 = INTEGER: 5 SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB SNMPv2-MIB::sysORID.2 = OID: iso.2.840.10036 SNMPv2-MIB::sysORID.3 = OID: IF-MIB::ifMIB SNMPv2-MIB::sysORID.4 = OID: SNMPv2-SMI::enterprises.10002.1.1.1.31 SNMPv2-MIB::sysORID.5 = OID: SNMPv2-SMI::enterprises.41112 SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMP entities SNMPv2-MIB::sysORDescr.2 = STRING: The MIB module for IEEE 802.11 entities. SNMPv2-MIB::sysORDescr.3 = STRING: The MIB module to describe generic objects for network interface sub-layers SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module to describe system resources SNMPv2-MIB::sysORDescr.5 = STRING: Ubiquiti Networks MIB module SNMPv2-MIB::sysORUpTime.1 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.2 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.3 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.4 = Timeticks: (9956100) 1 day, 3:39:21.00 SNMPv2-MIB::sysORUpTime.5 = Timeticks: (9956100) 1 day, 3:39:21.00
- Server that is showing issues
snmpwalk -v1 -c SNMPKEY domain.name.tld .1.3.6.1.2.1.1 iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543981700) 62 days, 23:03:37.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "domain.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: "Some GPS Location" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00
On 25 Jul 2016, at 09:00, Lyndon Labuschagne wrote:
May not betheDB after all most of the SQL RUNTIME stats are 0.000xxxx so its probably not
found this in debug mode
It seems the poller / mib is freaking out about "Reason: (noSuchName) There is no such dns.name.tld variable name in this MIB."
the device is reachable via SNMP as shown in a portion of the snmpwalk below
this is where I am seeing the slowing down during the ifindex scanning as previously posted
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428547' WHERE `error_id` = '6645'] SQL RUNTIME[0.00229788s] SNMP STATUS[FALSE] 6 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6]
CMD EXITCODE[1] CMD RUNTIME[6.0817s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.6
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.6 ifOutMulticastPkts.6 ifInBroadcastPkts.6 ifOutBroadcastPkts.6 ifHCInOctets.6 ifHCOutOctets.6 ifHCInUcastPkts.6 ifHCOutUcastPkts.6 ifHCInMulticastPkts.6 ifHCOutMulticastPkts.6 ifHCInBroadcastPkts.6 ifHCOutBroadcastPkts.6 ifInOctets.6 ifOutOctets.6 ifInUcastPkts.6 ifOutUcastPkts.6 ifInNUcastPkts.6 ifOutNUcastPkts.6 ifInDiscards.6 ifOutDiscards.6 ifInErrors.6 ifOutErrors.6 ifInUnknownProtos.6 ifMtu.6 ifSpeed.6 ifPhysAddress.6 ifAdminStatus.6 ifLastChange.6 ifPromiscuousMode.6 ifConnectorPresent.6';] SQL RUNTIME[0.00058079s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428553' WHERE `error_id` = '6650'] SQL RUNTIME[0.00175095s] SNMP STATUS[FALSE] 7 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7]
CMD EXITCODE[1] CMD RUNTIME[6.0846s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.7
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.7 ifOutMulticastPkts.7 ifInBroadcastPkts.7 ifOutBroadcastPkts.7 ifHCInOctets.7 ifHCOutOctets.7 ifHCInUcastPkts.7 ifHCOutUcastPkts.7 ifHCInMulticastPkts.7 ifHCOutMulticastPkts.7 ifHCInBroadcastPkts.7 ifHCOutBroadcastPkts.7 ifInOctets.7 ifOutOctets.7 ifInUcastPkts.7 ifOutUcastPkts.7 ifInNUcastPkts.7 ifOutNUcastPkts.7 ifInDiscards.7 ifOutDiscards.7 ifInErrors.7 ifOutErrors.7 ifInUnknownProtos.7 ifMtu.7 ifSpeed.7 ifPhysAddress.7 ifAdminStatus.7 ifLastChange.7 ifPromiscuousMode.7 ifConnectorPresent.7';] SQL RUNTIME[0.00056005s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428559' WHERE `error_id` = '6656'] SQL RUNTIME[0.00169611s] SNMP STATUS[FALSE] 8 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8]
CMD EXITCODE[1] CMD RUNTIME[6.0792s] STDOUT[
] STDERR[ Error in packet Reason: (noSuchName) There is no such variable name in this MIB. Failed object: ifInMulticastPkts.8
Timeout: No Response from udp:dns.name.tld:161. ] SNMP ERROR[#1002 - Request timeout]
SQL[SELECT * FROM `snmp_errors` WHERE `device_id` = '27' AND `error_code` = '1002' AND `snmp_cmd` = 'snmpget' AND `snmp_options` = '-OQUs' AND `mib` = 'IF-MIB' AND `oid` = 'ifInMulticastPkts.8 ifOutMulticastPkts.8 ifInBroadcastPkts.8 ifOutBroadcastPkts.8 ifHCInOctets.8 ifHCOutOctets.8 ifHCInUcastPkts.8 ifHCOutUcastPkts.8 ifHCInMulticastPkts.8 ifHCOutMulticastPkts.8 ifHCInBroadcastPkts.8 ifHCOutBroadcastPkts.8 ifInOctets.8 ifOutOctets.8 ifInUcastPkts.8 ifOutUcastPkts.8 ifInNUcastPkts.8 ifOutNUcastPkts.8 ifInDiscards.8 ifOutDiscards.8 ifInErrors.8 ifOutErrors.8 ifInUnknownProtos.8 ifMtu.8 ifSpeed.8 ifPhysAddress.8 ifAdminStatus.8 ifLastChange.8 ifPromiscuousMode.8 ifConnectorPresent.8';] SQL RUNTIME[0.00071502s]
SQL[UPDATE `snmp_errors` set `error_count` ='148',`updated` ='1469428565' WHERE `error_id` = '6662'] SQL RUNTIME[0.00218582s] SNMP STATUS[FALSE] 9 CMD[/usr/bin/snmpget -v1 -c *** -Pu -OQUs -m IF-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'dns.name.tld':'161' ifInMulticastPkts.9 ifOutMulticastPkts.9 ifInBroadcastPkts.9 ifOutBroadcastPkts.9 ifHCInOctets.9 ifHCOutOctets.9 ifHCInUcastPkts.9 ifHCOutUcastPkts.9 ifHCInMulticastPkts.9 ifHCOutMulticastPkts.9 ifHCInBroadcastPkts.9 ifHCOutBroadcastPkts.9 ifInOctets.9 ifOutOctets.9 ifInUcastPkts.9 ifOutUcastPkts.9 ifInNUcastPkts.9 ifOutNUcastPkts.9 ifInDiscards.9 ifOutDiscards.9 ifInErrors.9 ifOutErrors.9 ifInUnknownProtos.9 ifMtu.9 ifSpeed.9 ifPhysAddress.9 ifAdminStatus.9 ifLastChange.9 ifPromiscuousMode.9 ifConnectorPresent.9]
CMD EXITCODE[1] CMD RUNTIME[6.0803s] STDOUT[
This is a manual snmpwalk
root@observium:~# snmpwalk -v1 -c SNMP-KEY dns.name.tld iso.3.6.1.2.1.1.1.0 = STRING: "Linux 2.6.32.65 #1 Thu Jul 16 11:50:24 EEST 2015 mips" iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.41112.1.4 iso.3.6.1.2.1.1.3.0 = Timeticks: (543077500) 62 days, 20:32:55.00 iso.3.6.1.2.1.1.4.0 = STRING: "networks" iso.3.6.1.2.1.1.5.0 = STRING: "vdns.name.tld" iso.3.6.1.2.1.1.6.0 = STRING: “Some GPS Co Ords" iso.3.6.1.2.1.1.7.0 = INTEGER: 79 iso.3.6.1.2.1.1.8.0 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.1.9.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.1.9.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.1.9.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.1.9.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1 iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.2.840.10036 iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.2.1.31 iso.3.6.1.2.1.1.9.1.2.4 = OID: iso.3.6.1.4.1.10002.1.1.1.31 iso.3.6.1.2.1.1.9.1.2.5 = OID: iso.3.6.1.4.1.41112 iso.3.6.1.2.1.1.9.1.3.1 = STRING: "The MIB module for SNMP entities" iso.3.6.1.2.1.1.9.1.3.2 = STRING: "The MIB module for IEEE 802.11 entities." iso.3.6.1.2.1.1.9.1.3.3 = STRING: "The MIB module to describe generic objects for network interface sub-layers" iso.3.6.1.2.1.1.9.1.3.4 = STRING: "The MIB module to describe system resources" iso.3.6.1.2.1.1.9.1.3.5 = STRING: "Ubiquiti Networks MIB module " iso.3.6.1.2.1.1.9.1.4.1 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.2 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.3 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.4 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.1.9.1.4.5 = Timeticks: (9956100) 1 day, 3:39:21.00 iso.3.6.1.2.1.2.1.0 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1 iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2 iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3 iso.3.6.1.2.1.2.2.1.1.4 = INTEGER: 4 iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5 iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7 iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8 iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9 iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lo" iso.3.6.1.2.1.2.2.1.2.2 = STRING: "eth0" iso.3.6.1.2.1.2.2.1.2.3 = STRING: "eth1" iso.3.6.1.2.1.2.2.1.2.4 = STRING: "wifi0" iso.3.6.1.2.1.2.2.1.2.5 = STRING: "ath0" iso.3.6.1.2.1.2.2.1.2.6 = STRING: "eth0.36" iso.3.6.1.2.1.2.2.1.2.7 = STRING: "ath0.36" iso.3.6.1.2.1.2.2.1.2.8 = STRING: "br0" iso.3.6.1.2.1.2.2.1.2.9 = STRING: "br1" iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 24 iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.6 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.7 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.8 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.3.9 = INTEGER: 6 iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 16436 iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500 iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 2286 iso.3.6.1.2.1.2.2.1.4.5 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.6 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.7 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.8 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.4.9 = INTEGER: 2020 iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.2 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.3 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.4 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.5 = Gauge32: 0 iso.3.6.1.2.1.2.2.1.5.6 = Gauge32: 100000000 iso.3.6.1.2.1.2.2.1.5.7 = Gauge32: 0
On 25 Jul 2016, at 07:24, Lyndon Labuschagne wrote:
Im going to fire up the server again today and test more, will let you know,
if there anyway of running the poller and but not writing to the DB, I see there is a flag to disable the writing of RRDs, I want to see if the poller performance is the issue, the disk I/O or the DB
On 25 Jul 2016, at 00:30, Adam Armstrong wrote:
If do you have sufficient buffers and caching for innodb in mysql?
We have some /very/ large installs which saw little impact from the changes.
Adam.
Sent from BlueMail
On 24 Jul 2016, at 20:55, Lyndon Labuschagne wrote: I have been performance tuning all day with little to no affect
its just very strange how a XEN VM out performs bare metal, it may also be Ubuntu 16.04 as that is on the bare metal server and 15.04 is on the VM
On 24 Jul 2016, at 21:47, Adam Armstrong wrote:
We've changed the way the ports table works, removing memory tables.
It's likely that you now need more mysql resources, since were now writing to an innodb table instead of a memory table.
Odd, really. It didn't seem to have a huge effect on my dev install or some other very large installs.
Perhaps you could investigate mysql performance tuning? :)
Adam.
Sent from BlueMail
On 24 Jul 2016, at 19:58, Lyndon Labuschagne wrote: i went back to use the old database server on the VM and all was much better, running pollers at 10 to 15 seconds per device,
until I upgraded the db by using discovery -u
back in the same boat again
On 24 Jul 2016, at 09:56, Lyndon Labuschagne wrote:
Ok if I poll a router / switch / server etc it seems to be ok, a full scan on a 7606 takes around 45 seconds,
if I poll a radio, Ubiquiti Airfibre/ M5 . Rocket etc it seems to be slow, even when the radio is directly connected to the hosting switch as a test. The issue pops up when doing the ifindex - ifIndex: 1 2 3 5 6 7 8 9 (that takes 40 to 50 seconds)
o Caching Oids ifDescr ifType ifOperStatus ifAlias ifName ifHighSpeed ifInMulticastPkts ifOutMulticastPkts ifInBroadcastPkts ifOutBroadcastPkts ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts ifHCInMulticastPkts ifHCOutMulticastPkts ifHCInBroadcastPkts ifHCOutBroadcastPkts ifInOctets ifOutOctets ifInUcastPkts ifOutUcastPkts ifInNUcastPkts ifOutNUcastPkts ifInDiscards ifOutDiscards ifInErrors ifOutErrors ifInUnknownProtos ifMtu ifSpeed ifPhysAddress ifAdminStatus ifLastChange ifPromiscuousMode ifConnectorPresent, ifIndex: 1 2 3 5 6 7 8 9 ADSL-LINE-MIB EtherLike-MIB dot3StatsDuplexStatus
The the other thing that had changed is that I was running a slightly older version of Observium on the VM server, still in the 79xx range not the 8003 version
On 24 Jul 2016, at 09:16, Lyndon Labuschagne wrote:
I know its bad form to reply to your own email but here is some more information re drive performance
root@observium:/opt/observium/rrd# hdparm -Tt /dev/mapper/observium--vg-root
/dev/mapper/observium--vg-root: Timing cached reads: 14428 MB in 2.00 seconds = 7218.63 MB/sec Timing buffered disk reads: 950 MB in 3.01 seconds = 315.94 MB/sec
root@observium:/opt/observium/rrd# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 0.125735 s, 667 MB/s
On 24 Jul 2016, at 09:13, Lyndon Labuschagne wrote:
Hi All
I have moved from a VM server to a physical server. as the VM was taking a bit of a pounding, graph dropouts etc The physical server is a dual quad core Xeon, 2.66Ghz with 128GB of RAM and a 32GB ramdisk, and rrdcache I also have 2.5TB or SAS storage in a raid6 config
Will all of this my average poller-wrapper time is around 60 to 70 seconds per device if I manually run an snmpwalk a single device takes around 6 seconds
I have also disabled the ramdisk, rrdcached etc and I am not seeing much of a difference either way
I have around 900 devices and the VM server was way faster, any idea where to start looking for a bottleneck?
Regards Lyndon
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.orgjavascript:; http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Eduardo Schoedler
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (3)
-
Adam Armstrong
-
Eduardo Schoedler
-
Lyndon Labuschagne