renamed block devices
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Observium CE 0.13.10.4585 doesn't seem to handle renamed block devices very gracefully.
I added some disks to one of my boxes, which caused the old sda and sdb to come up as sdb and sdd. Now on the disk i/o page some devices are missing, some appear twice, most are displaying incorrect info.
I've attached the contents of the ucd_diskio table. sdc and it's partitions are missing and a number of the others are duplicated.
How do I clear up this mess? Ideally I'd like to keep the old data so I could see what effect the changes I'm making has.
- -- Tony Lill, OCT, ajlill@AJLC.Waterloo.ON.CA President, A. J. Lill Consultants (519) 650 0660 539 Grand Valley Dr., Cambridge, Ont. N3H 2S2 (519) 241 2461 - --------------- http://www.ajlc.waterloo.on.ca/ ----------------
Hi,
You need to re-run discovery after changes. Some modules don't pick up changes during the poller, as it'd be too slow (it wouldn't really be slower for this module, but that's just how it was written).
./discovery.php -h all -m ucd-diskio
adam.
On 2014-03-11 18:41, Tony Lill wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Observium CE 0.13.10.4585 doesn't seem to handle renamed block devices very gracefully.
I added some disks to one of my boxes, which caused the old sda and sdb to come up as sdb and sdd. Now on the disk i/o page some devices are missing, some appear twice, most are displaying incorrect info.
I've attached the contents of the ucd_diskio table. sdc and it's partitions are missing and a number of the others are duplicated.
How do I clear up this mess? Ideally I'd like to keep the old data so I could see what effect the changes I'm making has.
Tony Lill, OCT, ajlill@AJLC.Waterloo.ON.CA President, A. J. Lill Consultants (519) 650 0660 539 Grand Valley Dr., Cambridge, Ont. N3H 2S2 (519) 241 2461
- --------------- http://www.ajlc.waterloo.on.ca/ ----------------
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlMfrVAACgkQGS8yZq1uvxDMvwCff66JvTYJMebz+J6sdq7M5DSy 1DcAnRNL7haZlrK7KF0psYodjVCcmQEA =bMYY -----END PGP SIGNATURE-----
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Running that caused one minor change, but didn't fix anything. The problem is block device names such as returned by the ucd diskio mib are essentially meaningless and and should be assumed to be randomly assigned on reboot, if not more often.
When I was doing my own thing with mrtg, I ran a script that mapped the snmp index to a string that I grubbed out of /dev/disk/by-id
I'm assuming that observium uses that table to map the snmp index to a diskio_id and uses that to store the data. If I go in and fix the index and description, will observium overwrite it. I'm assuming no, since it hasn't fixed itself.
On 03/11/2014 08:55 PM, Adam Armstrong wrote:
Hi,
You need to re-run discovery after changes. Some modules don't pick up changes during the poller, as it'd be too slow (it wouldn't really be slower for this module, but that's just how it was written).
./discovery.php -h all -m ucd-diskio
adam.
On 2014-03-11 18:41, Tony Lill wrote: Observium CE 0.13.10.4585 doesn't seem to handle renamed block devices very gracefully.
I added some disks to one of my boxes, which caused the old sda and sdb to come up as sdb and sdd. Now on the disk i/o page some devices are missing, some appear twice, most are displaying incorrect info.
I've attached the contents of the ucd_diskio table. sdc and it's partitions are missing and a number of the others are duplicated.
How do I clear up this mess? Ideally I'd like to keep the old data so I could see what effect the changes I'm making has.
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
- -- Tony Lill, OCT, ajlill@AJLC.Waterloo.ON.CA President, A. J. Lill Consultants (519) 650 0660 539 Grand Valley Dr., Cambridge, Ont. N3H 2S2 (519) 241 2461 - --------------- http://www.ajlc.waterloo.on.ca/ ----------------
On 2014-03-11 19:56, Tony Lill wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Running that caused one minor change, but didn't fix anything. The problem is block device names such as returned by the ucd diskio mib are essentially meaningless and and should be assumed to be randomly assigned on reboot, if not more often.
When I was doing my own thing with mrtg, I ran a script that mapped the snmp index to a string that I grubbed out of /dev/disk/by-id
I'm assuming that observium uses that table to map the snmp index to a diskio_id and uses that to store the data. If I go in and fix the index and description, will observium overwrite it. I'm assuming no, since it hasn't fixed itself.
It seems the discovery code is still unfinished, it won't update existing entries, only add or remove them.
From includes/discovery/ucd-diskio.inc.php :
else { echo("."); // FIXME Need update code here! }
This is what happens when you code on the move and forget what you're doing!
I'll fix it in SVN, but it won't make it into CE for a while.
In the mean time, you could truncate the table and rediscover, and it'll be fine.
Thanks, adam.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
If you're going to mess around with that bit of code anyway...
I'm running a bunch of xen servers. On the dom0, I have dozens of logical volumes which all show up as dm-* which presents me with a number of problems
1) The names are totally meaningless 2) The dm-* name is a random string assigned at boot/creation time and has no connection with an actual device. I'm not even sure that the names aren't recycled if you delete a bunch of logical volumes and create new ones without rebooting.
These points actually applies to all block devices, not just the dm-*s, but at least the sd* devices usually require someone to open up the box and change hardware.
It would be very nice if there was an option to have the unix agent maintain this table, since it has access to all the info to map the snmp index number to something tied to the actual hardware or function of the block device. If that was stable, then it would also be feasible to allow the user to associate a description or label to the data.
The only alternative to this is to try and get the snmp folk to use something more stable than these names..... Or, extend snmpd to return the mapping in a user defined oid.... Hmmmmmm
3) The number of these devices make the disk i/o page unwieldy and horribly slow 4) The summary rrd charts are ridiculously large and provide no value 5) Most of this data is duplicated since I monitor the vms as well, which is a performance hit.
I could solve most of these issues if there was a way to tell observium to ignore any device named dm-[0-9]+ (and vif[0-9]+ on the network interface side). Is there some way to filter the devices that observium enumerates, and if not, could you add that pretty please??
On 03/11/2014 10:30 PM, Adam Armstrong wrote:
On 2014-03-11 19:56, Tony Lill wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Running that caused one minor change, but didn't fix anything. The problem is block device names such as returned by the ucd diskio mib are essentially meaningless and and should be assumed to be randomly assigned on reboot, if not more often.
When I was doing my own thing with mrtg, I ran a script that mapped the snmp index to a string that I grubbed out of /dev/disk/by-id
I'm assuming that observium uses that table to map the snmp index to a diskio_id and uses that to store the data. If I go in and fix the index and description, will observium overwrite it. I'm assuming no, since it hasn't fixed itself.
It seems the discovery code is still unfinished, it won't update existing entries, only add or remove them.
From includes/discovery/ucd-diskio.inc.php :
else { echo("."); // FIXME Need update code here! }
This is what happens when you code on the move and forget what you're doing!
I'll fix it in SVN, but it won't make it into CE for a while.
In the mean time, you could truncate the table and rediscover, and it'll be fine.
Thanks, adam. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
- -- Tony Lill, OCT, ajlill@AJLC.Waterloo.ON.CA President, A. J. Lill Consultants (519) 650 0660 539 Grand Valley Dr., Cambridge, Ont. N3H 2S2 (519) 241 2461 - --------------- http://www.ajlc.waterloo.on.ca/ ----------------
Hi,
It would save many people a lot of time if one would check the documentation before writing lengthy e-mails. Just sayin'. ;-)
(Because, yes, that is possible)
Tom
On 03/12/2014 04:16 PM, Tony Lill wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
If you're going to mess around with that bit of code anyway...
I'm running a bunch of xen servers. On the dom0, I have dozens of logical volumes which all show up as dm-* which presents me with a number of problems
- The names are totally meaningless
- The dm-* name is a random string assigned at boot/creation time and
has no connection with an actual device. I'm not even sure that the names aren't recycled if you delete a bunch of logical volumes and create new ones without rebooting.
These points actually applies to all block devices, not just the dm-*s, but at least the sd* devices usually require someone to open up the box and change hardware.
It would be very nice if there was an option to have the unix agent maintain this table, since it has access to all the info to map the snmp index number to something tied to the actual hardware or function of the block device. If that was stable, then it would also be feasible to allow the user to associate a description or label to the data.
The only alternative to this is to try and get the snmp folk to use something more stable than these names..... Or, extend snmpd to return the mapping in a user defined oid.... Hmmmmmm
- The number of these devices make the disk i/o page unwieldy and
horribly slow 4) The summary rrd charts are ridiculously large and provide no value 5) Most of this data is duplicated since I monitor the vms as well, which is a performance hit.
I could solve most of these issues if there was a way to tell observium to ignore any device named dm-[0-9]+ (and vif[0-9]+ on the network interface side). Is there some way to filter the devices that observium enumerates, and if not, could you add that pretty please??
On 03/11/2014 10:30 PM, Adam Armstrong wrote:
On 2014-03-11 19:56, Tony Lill wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Running that caused one minor change, but didn't fix anything. The problem is block device names such as returned by the ucd diskio mib are essentially meaningless and and should be assumed to be randomly assigned on reboot, if not more often.
When I was doing my own thing with mrtg, I ran a script that mapped the snmp index to a string that I grubbed out of /dev/disk/by-id
I'm assuming that observium uses that table to map the snmp index to a diskio_id and uses that to store the data. If I go in and fix the index and description, will observium overwrite it. I'm assuming no, since it hasn't fixed itself.
It seems the discovery code is still unfinished, it won't update existing entries, only add or remove them.
From includes/discovery/ucd-diskio.inc.php :
else { echo("."); // FIXME Need update code here! }
This is what happens when you code on the move and forget what you're doing!
I'll fix it in SVN, but it won't make it into CE for a while.
In the mean time, you could truncate the table and rediscover, and it'll be fine.
Thanks, adam. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Tony Lill, OCT, ajlill@AJLC.Waterloo.ON.CA President, A. J. Lill Consultants (519) 650 0660 539 Grand Valley Dr., Cambridge, Ont. N3H 2S2 (519) 241 2461
- --------------- http://www.ajlc.waterloo.on.ca/ ----------------
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlMgel4ACgkQGS8yZq1uvxARQACfZmMbWL/5rq4zr9Gc8TQLteQM OyAAn22qehK9b0Y5udCpfpEksmtDwFwH =07rw -----END PGP SIGNATURE----- _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/12/2014 01:36 PM, Tom Laermans wrote:
Hi,
It would save many people a lot of time if one would check the documentation before writing lengthy e-mails. Just sayin'. ;-)
(Because, yes, that is possible)
Tom
Would you care to elucidate, or was this just a more polite RTFM, which I have BTW?
- -- Tony Lill, OCT, ajlill@AJLC.Waterloo.ON.CA President, A. J. Lill Consultants (519) 650 0660 539 Grand Valley Dr., Cambridge, Ont. N3H 2S2 (519) 241 2461 - --------------- http://www.ajlc.waterloo.on.ca/ ----------------
http://www.observium.org/wiki/Configuration_Options#Filters_2
On 2014-03-12 16:35, Tony Lill wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/12/2014 01:36 PM, Tom Laermans wrote: Hi,
It would save many people a lot of time if one would check the documentation before writing lengthy e-mails. Just sayin'. ;-)
(Because, yes, that is possible)
Tom
Would you care to elucidate, or was this just a more polite RTFM, which I have BTW?
Tony Lill, OCT, ajlill@AJLC.Waterloo.ON.CA President, A. J. Lill Consultants (519) 650 0660 539 Grand Valley Dr., Cambridge, Ont. N3H 2S2 (519) 241 2461
- --------------- http://www.ajlc.waterloo.on.ca/ ----------------
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEUEARECAAYFAlMg4S0ACgkQGS8yZq1uvxAbSQCffjYzwwkfNTP4BmwRfnFa0dxL CSgAmPa1py2sSo3Boey0ag7E1OPg16E= =FeO+ -----END PGP SIGNATURE----- _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On 12/03/2014 23:35, Tony Lill wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/12/2014 01:36 PM, Tom Laermans wrote:
Hi,
It would save many people a lot of time if one would check the documentation before writing lengthy e-mails. Just sayin'. ;-)
(Because, yes, that is possible)
Tom
Would you care to elucidate, or was this just a more polite RTFM, which I have BTW?
It was. But sure: http://www.observium.org/wiki/Configuration_Options
Ctrl-F ignore Enter
Tom
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I see where I can ignore interfaces and I see where I can ignore mount points in the storage pages, but I don't see where I can ignore block devices, so a way to filter them out would still be appreciated.
And none of that addresses the other issue of the e-mail, which is that the snmp indexes and devices names stored in usd_diskio table are essentially random and cannot be assumed to have any continuity over a reboot.
I've been playing around with it, and I now have an snmp extension that will give a mapping from the name in the ucd diskio mib and a more persistent identifier. I'm starting to hack at the discovery/ucd-diskio.inc.php file.
I just want to confirm that the data that observium stores is tied to the diskio_descr column. I played about in mysql and if I change that value I seem to get a new graph started.
On 03/12/2014 07:25 PM, Tom Laermans wrote:
On 12/03/2014 23:35, Tony Lill wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/12/2014 01:36 PM, Tom Laermans wrote:
Hi,
It would save many people a lot of time if one would check the documentation before writing lengthy e-mails. Just sayin'. ;-)
(Because, yes, that is possible)
Tom
Would you care to elucidate, or was this just a more polite RTFM, which I have BTW?
It was. But sure: http://www.observium.org/wiki/Configuration_Options
Ctrl-F ignore Enter
Tom _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
- -- Tony Lill, OCT, ajlill@AJLC.Waterloo.ON.CA President, A. J. Lill Consultants (519) 650 0660 539 Grand Valley Dr., Cambridge, Ont. N3H 2S2 (519) 241 2461 - --------------- http://www.ajlc.waterloo.on.ca/ ----------------
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I have the first run at dealing with these problems available at:
ftp://ftp.ajlc.waterloo.on.ca/pub/observium-persistent-blocks-0.1.tar.gz
It contains an snmp extension that maps the old block device names to better/persistent names. It also contains a replacement ucd-diskio.inc.php that
1) uses the snmp extension, if present, to store the diskio data 2) updates the table properly 3) now filters block device names a la the bad_if* filtering config options for interfaces.
Enjoy.
On 03/12/2014 07:59 PM, Tony Lill wrote:
I see where I can ignore interfaces and I see where I can ignore mount points in the storage pages, but I don't see where I can ignore block devices, so a way to filter them out would still be appreciated.
And none of that addresses the other issue of the e-mail, which is that the snmp indexes and devices names stored in usd_diskio table are essentially random and cannot be assumed to have any continuity over a reboot.
I've been playing around with it, and I now have an snmp extension that will give a mapping from the name in the ucd diskio mib and a more persistent identifier. I'm starting to hack at the discovery/ucd-diskio.inc.php file.
I just want to confirm that the data that observium stores is tied to the diskio_descr column. I played about in mysql and if I change that value I seem to get a new graph started.
On 03/12/2014 07:25 PM, Tom Laermans wrote:
On 12/03/2014 23:35, Tony Lill wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/12/2014 01:36 PM, Tom Laermans wrote:
Hi,
It would save many people a lot of time if one would check the documentation before writing lengthy e-mails. Just sayin'. ;-)
(Because, yes, that is possible)
Tom
Would you care to elucidate, or was this just a more polite RTFM, which I have BTW?
It was. But sure: http://www.observium.org/wiki/Configuration_Options
Ctrl-F ignore Enter
Tom _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
- -- Tony Lill, OCT, ajlill@AJLC.Waterloo.ON.CA President, A. J. Lill Consultants (519) 650 0660 539 Grand Valley Dr., Cambridge, Ont. N3H 2S2 (519) 241 2461 - --------------- http://www.ajlc.waterloo.on.ca/ ----------------
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I have a new version of my code to fix several problems observium has with the disk I/O charts available at
ftp://ftp.ajlc.waterloo.on.ca/pub/observium-persistent-blocks-0.2.tar.gz
It has a better implementation of the snmp extension and fixes a bug updating existing table entries
On 03/15/2014 12:58 PM, Tony Lill wrote:
I have the first run at dealing with these problems available at:
ftp://ftp.ajlc.waterloo.on.ca/pub/observium-persistent-blocks-0.1.tar.gz
It contains an snmp extension that maps the old block device names to better/persistent names. It also contains a replacement ucd-diskio.inc.php that
- uses the snmp extension, if present, to store the diskio data 2)
updates the table properly 3) now filters block device names a la the bad_if* filtering config options for interfaces.
Enjoy.
- -- Tony Lill, OCT, ajlill@AJLC.Waterloo.ON.CA President, A. J. Lill Consultants (519) 650 0660 539 Grand Valley Dr., Cambridge, Ont. N3H 2S2 (519) 241 2461 - --------------- http://www.ajlc.waterloo.on.ca/ ----------------
Hi,
We have no interest in SNMP extensions.
Any script-based extraction of information from hosts should be done via the agent.
Thanks, adam.
On 2014-04-28 10:43, Tony Lill wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I have a new version of my code to fix several problems observium has with the disk I/O charts available at
ftp://ftp.ajlc.waterloo.on.ca/pub/observium-persistent-blocks-0.2.tar.gz
It has a better implementation of the snmp extension and fixes a bug updating existing table entries
On 03/15/2014 12:58 PM, Tony Lill wrote: I have the first run at dealing with these problems available at:
ftp://ftp.ajlc.waterloo.on.ca/pub/observium-persistent-blocks-0.1.tar.gz
It contains an snmp extension that maps the old block device names to better/persistent names. It also contains a replacement ucd-diskio.inc.php that
- uses the snmp extension, if present, to store the diskio data 2)
updates the table properly 3) now filters block device names a la the bad_if* filtering config options for interfaces.
Enjoy.
Tony Lill, OCT, ajlill@AJLC.Waterloo.ON.CA President, A. J. Lill Consultants (519) 650 0660 539 Grand Valley Dr., Cambridge, Ont. N3H 2S2 (519) 241 2461
- --------------- http://www.ajlc.waterloo.on.ca/ ----------------
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlNedy8ACgkQGS8yZq1uvxCYUgCfUARXHlJZSUcK/7wQ0CZFrCgl 120An0u2HBhh8kBRS5C5jKxuK6YIetIC =KDXG -----END PGP SIGNATURE----- _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On 2014-03-11 18:41, Tony Lill wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Observium CE 0.13.10.4585 doesn't seem to handle renamed block devices very gracefully.
I added some disks to one of my boxes, which caused the old sda and sdb to come up as sdb and sdd. Now on the disk i/o page some devices are missing, some appear twice, most are displaying incorrect info.
I've attached the contents of the ucd_diskio table. sdc and it's partitions are missing and a number of the others are duplicated.
This is now fixed in SVN.
adam.
participants (3)
-
Adam Armstrong -
Tom Laermans -
Tony Lill