ZFS File systems issue and incorrect storage reporting
I have a system where the storage sizes are being reported incorrectly and there appears to be an issue with reading ZFS file systems being reported at all.
I have 4 NAS (OmniOS) servers that are ZFS based, but none of the ZFS disk space is being reported at all. For example:
Filesystem 1K-blocks Used Available Use% Mounted on /usr/lib/libc/libc_hwcap1.so.1 118167920 2484288 115683632 3% /lib/libc.so.1 swap 8399672 336 8399336 1% /etc/svc/volatile swap 8399428 92 8399336 1% /tmp swap 8399396 60 8399336 1% /var/run rpool/export 115683728 96 115683632 1% /export rpool/export/home 115683728 96 115683632 1% /export/home nas01c 4392986368 96 4392986272 1% /nas01c nas01c/vm 5658664448 1437476772 4221187676 26% /nas01c/vm rpool 115683736 104 115683632 1% /rpool nas01c/vm-data 5031018412 809830736 4221187676 17% /nas01c/vm-data nas01c/backup_appliance 4221268004 80328 4221187676 1% /nas01c/backup_appliance
Yet Observium is only showing: Mountpoint Size Used Free Usage % /etc/svc/volatile 9.29GB 336kB 9.29GB 100% 0% /lib/libc.so.1 112GB 2.37GB 110GB 98% 2% /tmp 9.29GB 92kB 9.29GB 100% 0% /var/run 9.29GB 60kB 9.29GB 100% 0%
I have another system that is mixed ZFS, XFS storage and on that one, on the "Alerts" page for the system Observium is seeing the filesystems as:
Mountpoint Size Used Free Usage % / 281GB 138GB 142GB 51% 49% /backup0 757GB 7.17TB -6.4TB -868% 968% /backup1 757GB 10.1TB -9.4TB -1276% 1376% /scratch 3.64TB 903GB 2.75TB 76% 24% /zfsbkp 1.64TB 128kB 1.64TB 100% 0% /zfsbkp/code 7.11TB 5.47TB 1.64TB 23% 77%
The actual sizes are:
/ 281GB /backup0 34.33TB /backup1 34.33TB /scratch 3.64TB /zfsbkp 1.64TB /zfsbkp/code 7.11TB
I've tried deleting it and re-adding with the same result. The system is a Debian 7.6 using XFS and ZFS file systems.
This is causing us to get alerts for this system when in fact it is fine. Does anyone have any suggestions on how to fix this?
-- [exegy_119x45_xparent]https://www.exegy.com/ S. Kent Hamilton Senior Systems Administrator 349 Marshall Ave, Saint Louis, MO US 63119 Work: +1 314-218-3558 Mobile: +1 636-634-1549 Email: KHamilton@Exegy.COMmailto:KHamilton@Exegy.COM
________________________________
This e-mail and any documents accompanying it may contain legally privileged and/or confidential information belonging to Exegy, Inc. Such information may be protected from disclosure by law. The information is intended for use by only the addressee. If you are not the intended recipient, you are hereby notified that any disclosure or use of the information is strictly prohibited. If you have received this e-mail in error, please immediately contact the sender by e-mail or phone regarding instructions for return or destruction and do not use or disclose the content to others.
This is probably the result of the device not reporting the correct values. Very common with large filesystems.
Have you checked what data the device is reporting?
adam. On 03/08/2017 18:08:51, Hamilton, Kent khamilton@exegy.com wrote: I have a system where the storage sizes are being reported incorrectly and there appears to be an issue with reading ZFS file systems being reported at all. I have 4 NAS (OmniOS) servers that are ZFS based, but none of the ZFS disk space is being reported at all. For example: Filesystem 1K-blocks Used Available Use% Mounted on /usr/lib/libc/libc_hwcap1.so.1 118167920 2484288 115683632 3% /lib/libc.so.1 swap 8399672 336 8399336 1% /etc/svc/volatile swap 8399428 92 8399336 1% /tmp swap 8399396 60 8399336 1% /var/run rpool/export 115683728 96 115683632 1% /export rpool/export/home 115683728 96 115683632 1% /export/home nas01c 4392986368 96 4392986272 1% /nas01c nas01c/vm 5658664448 1437476772 4221187676 26% /nas01c/vm rpool 115683736 104 115683632 1% /rpool nas01c/vm-data 5031018412 809830736 4221187676 17% /nas01c/vm-data nas01c/backup_appliance 4221268004 80328 4221187676 1% /nas01c/backup_appliance Yet Observium is only showing: Mountpoint Size Used Free Usage % /etc/svc/volatile 9.29GB 336kB 9.29GB 100% 0% /lib/libc.so.1 112GB 2.37GB 110GB 98% 2% /tmp 9.29GB 92kB 9.29GB 100% 0% /var/run 9.29GB 60kB 9.29GB 100% 0% I have another system that is mixed ZFS, XFS storage and on that one, on the “Alerts” page for the system Observium is seeing the filesystems as: Mountpoint Size Used Free Usage % / 281GB 138GB 142GB 51% 49% /backup0 757GB 7.17TB -6.4TB -868% 968% /backup1 757GB 10.1TB -9.4TB -1276% 1376% /scratch 3.64TB 903GB 2.75TB 76% 24% /zfsbkp 1.64TB 128kB 1.64TB 100% 0% /zfsbkp/code 7.11TB 5.47TB 1.64TB 23% 77% The actual sizes are: / 281GB /backup0 34.33TB /backup1 34.33TB /scratch 3.64TB /zfsbkp 1.64TB /zfsbkp/code 7.11TB I’ve tried deleting it and re-adding with the same result. The system is a Debian 7.6 using XFS and ZFS file systems. This is causing us to get alerts for this system when in fact it is fine. Does anyone have any suggestions on how to fix this? -- [exegy_119x45_xparent] [https://www.exegy.com/] S. Kent Hamilton Senior Systems Administrator 349 Marshall Ave, Saint Louis, MO US 63119 Work: +1 314-218-3558 Mobile: +1 636-634-1549 Email: KHamilton@Exegy.COM [mailto:KHamilton@Exegy.COM]
This e-mail and any documents accompanying it may contain legally privileged and/or confidential information belonging to Exegy, Inc. Such information may be protected from disclosure by law. The information is intended for use by only the addressee. If you are not the intended recipient, you are hereby notified that any disclosure or use of the information is strictly prohibited. If you have received this e-mail in error, please immediately contact the sender by e-mail or phone regarding instructions for return or destruction and do not use or disclose the content to others.
Duh, you are correct sir. NetSNMP on the OmniOS systems is not reporting any of the ZFS file systems. NetSNMP on the Debian is reporting half the storage. <sigh>
Should have checked that before posting.
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: Thursday, August 3, 2017 12:11 PM To: observium@observium.org Subject: Re: [Observium] ZFS File systems issue and incorrect storage reporting
This is probably the result of the device not reporting the correct values. Very common with large filesystems.
Have you checked what data the device is reporting?
adam.
On 03/08/2017 18:08:51, Hamilton, Kent <khamilton@exegy.commailto:khamilton@exegy.com> wrote: I have a system where the storage sizes are being reported incorrectly and there appears to be an issue with reading ZFS file systems being reported at all.
I have 4 NAS (OmniOS) servers that are ZFS based, but none of the ZFS disk space is being reported at all. For example:
Filesystem 1K-blocks Used Available Use% Mounted on /usr/lib/libc/libc_hwcap1.so.1 118167920 2484288 115683632 3% /lib/libc.so.1 swap 8399672 336 8399336 1% /etc/svc/volatile swap 8399428 92 8399336 1% /tmp swap 8399396 60 8399336 1% /var/run rpool/export 115683728 96 115683632 1% /export rpool/export/home 115683728 96 115683632 1% /export/home nas01c 4392986368 96 4392986272 1% /nas01c nas01c/vm 5658664448 1437476772 4221187676 26% /nas01c/vm rpool 115683736 104 115683632 1% /rpool nas01c/vm-data 5031018412 809830736 4221187676 17% /nas01c/vm-data nas01c/backup_appliance 4221268004 80328 4221187676 1% /nas01c/backup_appliance
Yet Observium is only showing: Mountpoint Size Used Free Usage % /etc/svc/volatile 9.29GB 336kB 9.29GB 100% 0% /lib/libc.so.1 112GB 2.37GB 110GB 98% 2% /tmp 9.29GB 92kB 9.29GB 100% 0% /var/run 9.29GB 60kB 9.29GB 100% 0%
I have another system that is mixed ZFS, XFS storage and on that one, on the “Alerts” page for the system Observium is seeing the filesystems as:
Mountpoint Size Used Free Usage % / 281GB 138GB 142GB 51% 49% /backup0 757GB 7.17TB -6.4TB -868% 968% /backup1 757GB 10.1TB -9.4TB -1276% 1376% /scratch 3.64TB 903GB 2.75TB 76% 24% /zfsbkp 1.64TB 128kB 1.64TB 100% 0% /zfsbkp/code 7.11TB 5.47TB 1.64TB 23% 77%
The actual sizes are:
/ 281GB /backup0 34.33TB /backup1 34.33TB /scratch 3.64TB /zfsbkp 1.64TB /zfsbkp/code 7.11TB
I’ve tried deleting it and re-adding with the same result. The system is a Debian 7.6 using XFS and ZFS file systems.
This is causing us to get alerts for this system when in fact it is fine. Does anyone have any suggestions on how to fix this?
-- [exegy_119x45_xparent]https://www.exegy.com/ S. Kent Hamilton Senior Systems Administrator 349 Marshall Ave, Saint Louis, MO US 63119 Work: +1 314-218-3558 Mobile: +1 636-634-1549 Email: KHamilton@Exegy.COMmailto:KHamilton@Exegy.COM
________________________________
This e-mail and any documents accompanying it may contain legally privileged and/or confidential information belonging to Exegy, Inc. Such information may be protected from disclosure by law. The information is intended for use by only the addressee. If you are not the intended recipient, you are hereby notified that any disclosure or use of the information is strictly prohibited. If you have received this e-mail in error, please immediately contact the sender by e-mail or phone regarding instructions for return or destruction and do not use or disclose the content to others.
________________________________
This e-mail and any documents accompanying it may contain legally privileged and/or confidential information belonging to Exegy, Inc. Such information may be protected from disclosure by law. The information is intended for use by only the addressee. If you are not the intended recipient, you are hereby notified that any disclosure or use of the information is strictly prohibited. If you have received this e-mail in error, please immediately contact the sender by e-mail or phone regarding instructions for return or destruction and do not use or disclose the content to others.
On 03/08/2017 20:19, Hamilton, Kent wrote:
Duh, you are correct sir.
NetSNMP on the OmniOS systems is not reporting any of the ZFS file systems.
NetSNMP on the Debian is reporting half the storage. <sigh>
Should have checked that before posting.
I'd start by upgrading Debian to 8 (or 9!), that'll get you Net-SNMP 5.7.x which has plenty of fixes for large filesystems.
Tom
participants (3)
-
Adam Armstrong -
Hamilton, Kent -
Tom Laermans