Re: [Observium] RRD Perms Issue - Error Drawing Graph
It seems that someone has modified your cron jobs to no longer be run as root, since your rrds were created by root and svn has no ability to change them.
You need to work out where the poller is run from. It should be /etc/cron.d/observium
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 15 December 2015 18:08:08 Robbie Wright robbie@siuslawbroadband.com wrote:
Not sure who the poller is running as. What's the easy way to check that?
Yes, svn update has always been run as su.
/opt/observium/rrd is owned by www-data, not root unlike the other folders in the main directory. However, digging into the directory, all of the folders inside for each host are all owned by root.
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 9:24 AM, Adam Armstrong adama@memetic.org wrote:
Are you running your poller observium poller as a non-root user?
Did you svn up as root?
The contents of the rrd directory, and anything else written to by the poller, should be owned by the user who runs the poller.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 15 December 2015 17:06:24 Robbie Wright robbie@siuslawbroadband.com wrote:
Recently upgraded from 69xx to 72xx after we renewed our license and it broke all of our RRD's, resulting in the "Error Drawing Graph" issue.
Running this "./poller.php -h <hostname> -m ports -d" gives me two noticeable, but unobjectionable to me things:
SNMP_STATUS[TRUE] o Device status Device is reachable by PING (8.28ms) and SNMP (81.63ms)
RRD CMD[update /opt/observium/rrd/hostname/status.rrd N:1] RRD RUNTIME[0.0012s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/hostname/status.rrd': Permission denied] RRD_STATUS[FALSE]
and this at the end of the results:
Log file '/opt/observium/logs/observium.log' is not writable, check file permissions.
So clearly some type of permission issue but not sure what happened during the upgrade that would have caused this. Any thoughts on what chmod's need to be set on the directories or what else could have went haywire?
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101 _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
33 */6 * * * root /opt/observium/discovery.php -h all >> /dev/null 2>&1 */5 * * * * root /opt/observium/discovery.php -h new >> /dev/null 2>&1 */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Looks like root to me...
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 10:18 AM, Adam Armstrong adama@memetic.org wrote:
It seems that someone has modified your cron jobs to no longer be run as root, since your rrds were created by root and svn has no ability to change them.
You need to work out where the poller is run from. It should be /etc/cron.d/observium
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 15 December 2015 18:08:08 Robbie Wright robbie@siuslawbroadband.com wrote:
Not sure who the poller is running as. What's the easy way to check that?
Yes, svn update has always been run as su.
/opt/observium/rrd is owned by www-data, not root unlike the other folders in the main directory. However, digging into the directory, all of the folders inside for each host are all owned by root.
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 9:24 AM, Adam Armstrong adama@memetic.org wrote:
Are you running your poller observium poller as a non-root user?
Did you svn up as root?
The contents of the rrd directory, and anything else written to by the poller, should be owned by the user who runs the poller.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 15 December 2015 17:06:24 Robbie Wright robbie@siuslawbroadband.com wrote:
Recently upgraded from 69xx to 72xx after we renewed our license and it broke all of our RRD's, resulting in the "Error Drawing Graph" issue.
Running this "./poller.php -h <hostname> -m ports -d" gives me two noticeable, but unobjectionable to me things:
SNMP_STATUS[TRUE] o Device status Device is reachable by PING (8.28ms) and SNMP (81.63ms)
RRD CMD[update /opt/observium/rrd/hostname/status.rrd N:1] RRD RUNTIME[0.0012s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/hostname/status.rrd': Permission denied] RRD_STATUS[FALSE]
and this at the end of the results:
Log file '/opt/observium/logs/observium.log' is not writable, check file permissions.
So clearly some type of permission issue but not sure what happened during the upgrade that would have caused this. Any thoughts on what chmod's need to be set on the directories or what else could have went haywire?
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101 _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Well what makes you think they can't write to the rrds? :>
adam. On 15/12/2015 18:58:31, Robbie Wright robbie@siuslawbroadband.com wrote: 33 */6 * * * root /opt/observium/discovery.php -h all >> /dev/null 2>&1 */5 * * * * root /opt/observium/discovery.php -h new >> /dev/null 2>&1 */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Looks like root to me...
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101
On Tue, Dec 15, 2015 at 10:18 AM, Adam Armstrong <adama@memetic.org [mailto:adama@memetic.org]> wrote:
It seems that someone has modified your cron jobs to no longer be run as root, since your rrds were created by root and svn has no ability to change them. You need to work out where the poller is run from. It should be /etc/cron.d/observium Adam. Sent with AquaMail for Android http://www.aqua-mail.com [http://www.aqua-mail.com] On 15 December 2015 18:08:08 Robbie Wright <robbie@siuslawbroadband.com [mailto:robbie@siuslawbroadband.com]> wrote: Not sure who the poller is running as. What's the easy way to check that?
Yes, svn update has always been run as su.
/opt/observium/rrd is owned by www-data, not root unlike the other folders in the main directory. However, digging into the directory, all of the folders inside for each host are all owned by root.
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101 [tel:541-902-5101]
On Tue, Dec 15, 2015 at 9:24 AM, Adam Armstrong <adama@memetic.org [mailto:adama@memetic.org]> wrote:
Are you running your poller observium poller as a non-root user? Did you svn up as root? The contents of the rrd directory, and anything else written to by the poller, should be owned by the user who runs the poller. Adam. Sent with AquaMail for Android http://www.aqua-mail.com [http://www.aqua-mail.com] On 15 December 2015 17:06:24 Robbie Wright <robbie@siuslawbroadband.com [mailto:robbie@siuslawbroadband.com]> wrote: Recently upgraded from 69xx to 72xx after we renewed our license and it broke all of our RRD's, resulting in the "Error Drawing Graph" issue.
Running this "./poller.php -h <hostname> -m ports -d" gives me two noticeable, but unobjectionable to me things:
SNMP_STATUS[TRUE] o Device status Device is reachable by PING (8.28ms) and SNMP (81.63ms)
RRD CMD[update /opt/observium/rrd/hostname/status.rrd N:1] RRD RUNTIME[0.0012s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/hostname/status.rrd': Permission denied] RRD_STATUS[FALSE]
and this at the end of the results:
Log file '/opt/observium/logs/observium.log' is not writable, check file permissions.
So clearly some type of permission issue but not sure what happened during the upgrade that would have caused this. Any thoughts on what chmod's need to be set on the directories or what else could have went haywire?
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101 [tel:541-902-5101] _______________________________________________ observium mailing list observium@observium.org [mailto:observium%40observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium%40observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
The debug log...
File system isn't mounted read only per chance?
On 15/12/2015 20:19, Adam Armstrong wrote:
Well what makes you think they can't write to the rrds? :>
adam.
On 15/12/2015 18:58:31, Robbie Wright robbie@siuslawbroadband.com wrote:
33 */6 * * * root /opt/observium/discovery.php -h all >> /dev/null 2>&1 */5 * * * * root /opt/observium/discovery.php -h new >> /dev/null 2>&1 */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Looks like root to me...
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 10:18 AM, Adam Armstrong <adama@memetic.org mailto:adama@memetic.org> wrote:
It seems that someone has modified your cron jobs to no longer be run as root, since your rrds were created by root and svn has no ability to change them. You need to work out where the poller is run from. It should be /etc/cron.d/observium Adam. Sent with AquaMail for Android http://www.aqua-mail.com On 15 December 2015 18:08:08 Robbie Wright <robbie@siuslawbroadband.com <mailto:robbie@siuslawbroadband.com>> wrote:Not sure who the poller is running as. What's the easy way to check that? Yes, svn update has always been run as su. /opt/observium/rrd is owned by www-data, not root unlike the other folders in the main directory. However, digging into the directory, all of the folders inside for each host are all owned by root. Robbie Wright Siuslaw Broadband <https://siuslawbroadband.com> 541-902-5101 <tel:541-902-5101> On Tue, Dec 15, 2015 at 9:24 AM, Adam Armstrong <adama@memetic.org <mailto:adama@memetic.org>> wrote: Are you running your poller observium poller as a non-root user? Did you svn up as root? The contents of the rrd directory, and anything else written to by the poller, should be owned by the user who runs the poller. Adam. Sent with AquaMail for Android http://www.aqua-mail.com On 15 December 2015 17:06:24 Robbie Wright <robbie@siuslawbroadband.com <mailto:robbie@siuslawbroadband.com>> wrote:Recently upgraded from 69xx to 72xx after we renewed our license and it broke all of our RRD's, resulting in the "Error Drawing Graph" issue. Running this "./poller.php -h <hostname> -m ports -d" gives me two noticeable, but unobjectionable to me things: SNMP_STATUS[TRUE] o Device status Device is reachable by PING (8.28ms) and SNMP (81.63ms) RRD CMD[update /opt/observium/rrd/hostname/status.rrd N:1] RRD RUNTIME[0.0012s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/hostname/status.rrd': Permission denied] RRD_STATUS[FALSE] and this at the end of the results: Log file '/opt/observium/logs/observium.log' is not writable, check file permissions. So clearly some type of permission issue but not sure what happened during the upgrade that would have caused this. Any thoughts on what chmod's need to be set on the directories or what else could have went haywire? Robbie Wright Siuslaw Broadband <https://siuslawbroadband.com> 541-902-5101 <tel:541-902-5101> _______________________________________________ observium mailing list observium@observium.org <mailto:observium%40observium.org> http:// postman.memetic.org/ cgi-bin/mailman/ listinfo/ observium <http://postman.memetic.org/cgi-bin/mailman/listinfo/observium>_______________________________________________ observium mailing list observium@observium.org <mailto:observium@observium.org> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org <mailto:observium%40observium.org> http://postman.memetic. org/cgi-bin/mailman/listinfo/ observium <http://postman.memetic.org/cgi-bin/mailman/listinfo/observium>_______________________________________________ observium mailing list observium@observium.org <mailto:observium@observium.org> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Adam, I'm not sure it can't write the rrd's, I think it can't read them, but that' just based on the two log outputs I provided before.
Tom, I don't think the FS is mounted read only but I'm not sure how to check that. I can sudo nano a new file into the /opt/observium folder without issue.
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 11:24 AM, Tom Laermans tom.laermans@powersource.cx wrote:
The debug log...
File system isn't mounted read only per chance?
On 15/12/2015 20:19, Adam Armstrong wrote:
Well what makes you think they can't write to the rrds? :>
adam.
On 15/12/2015 18:58:31, Robbie Wright robbie@siuslawbroadband.com robbie@siuslawbroadband.com wrote: 33 */6 * * * root /opt/observium/discovery.php -h all >> /dev/null 2>&1 */5 * * * * root /opt/observium/discovery.php -h new >> /dev/null 2>&1 */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Looks like root to me...
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 10:18 AM, Adam Armstrong < adama@memetic.org adama@memetic.org> wrote:
It seems that someone has modified your cron jobs to no longer be run as root, since your rrds were created by root and svn has no ability to change them.
You need to work out where the poller is run from. It should be /etc/cron.d/observium
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 15 December 2015 18:08:08 Robbie Wright < robbie@siuslawbroadband.comrobbie@siuslawbroadband.com> wrote:
Not sure who the poller is running as. What's the easy way to check that?
Yes, svn update has always been run as su.
/opt/observium/rrd is owned by www-data, not root unlike the other folders in the main directory. However, digging into the directory, all of the folders inside for each host are all owned by root.
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 9:24 AM, Adam Armstrong < adama@memetic.org adama@memetic.org> wrote:
Are you running your poller observium poller as a non-root user?
Did you svn up as root?
The contents of the rrd directory, and anything else written to by the poller, should be owned by the user who runs the poller.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 15 December 2015 17:06:24 Robbie Wright < robbie@siuslawbroadband.comrobbie@siuslawbroadband.com> wrote:
Recently upgraded from 69xx to 72xx after we renewed our license and it broke all of our RRD's, resulting in the "Error Drawing Graph" issue.
Running this "./poller.php -h <hostname> -m ports -d" gives me two noticeable, but unobjectionable to me things:
SNMP_STATUS[TRUE] o Device status Device is reachable by PING (8.28ms) and SNMP (81.63ms) RRD CMD[update /opt/observium/rrd/hostname/status.rrd N:1] RRD RUNTIME[0.0012s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/hostname/status.rrd': Permission denied] RRD_STATUS[FALSE]
and this at the end of the results:
Log file '/opt/observium/logs/observium.log' is not writable, check file permissions.
So clearly some type of permission issue but not sure what happened during the upgrade that would have caused this. Any thoughts on what chmod's need to be set on the directories or what else could have went haywire?
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101 _______________________________________________ observium mailing list observium@observium.org http:// postman.memetic.org/ cgi-bin/mailman/listinfo/ observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic. org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing listobservium@observium.orghttp://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Did you generate the output from the first email as not-root?
http://wildgoosefestival.org/wp-content/uploads/2014/06/wild-goose-in-action...
I think it wants someone to chase it! :D
adam. On 15/12/2015 19:43:35, Robbie Wright robbie@siuslawbroadband.com wrote: Adam, I'm not sure it can't write the rrd's, I think it can't read them, but that' just based on the two log outputs I provided before.
Tom, I don't think the FS is mounted read only but I'm not sure how to check that. I can sudo nano a new file into the /opt/observium folder without issue.
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101
On Tue, Dec 15, 2015 at 11:24 AM, Tom Laermans <tom.laermans@powersource.cx [mailto:tom.laermans@powersource.cx]> wrote:
The debug log...
File system isn't mounted read only per chance?
On 15/12/2015 20:19, Adam Armstrong wrote:
Well what makes you think they can't write to the rrds? :>
adam. On 15/12/2015 18:58:31, Robbie Wright robbie@siuslawbroadband.com [mailto:robbie@siuslawbroadband.com] wrote: 33 */6 * * * root /opt/observium/discovery.php -h all >> /dev/null 2>&1 */5 * * * * root /opt/observium/discovery.php -h new >> /dev/null 2>&1 */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Looks like root to me...
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101 [tel:541-902-5101]
On Tue, Dec 15, 2015 at 10:18 AM, Adam Armstrong < [mailto:adama@memetic.org]adama@memetic.org [mailto:adama@memetic.org]> wrote:
It seems that someone has modified your cron jobs to no longer be run as root, since your rrds were created by root and svn has no ability to change them. You need to work out where the poller is run from. It should be /etc/cron.d/observium Adam. Sent with AquaMail for Android http://www.aqua-mail.com [http://www.aqua-mail.com] On 15 December 2015 18:08:08 Robbie Wright < [mailto:robbie@siuslawbroadband.com]robbie@siuslawbroadband.com [mailto:robbie@siuslawbroadband.com]> wrote: Not sure who the poller is running as. What's the easy way to check that?
Yes, svn update has always been run as su.
/opt/observium/rrd is owned by www-data, not root unlike the other folders in the main directory. However, digging into the directory, all of the folders inside for each host are all owned by root.
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101 [tel:541-902-5101]
On Tue, Dec 15, 2015 at 9:24 AM, Adam Armstrong < [mailto:adama@memetic.org]adama@memetic.org [mailto:adama@memetic.org]> wrote:
Are you running your poller observium poller as a non-root user? Did you svn up as root? The contents of the rrd directory, and anything else written to by the poller, should be owned by the user who runs the poller. Adam. Sent with AquaMail for Android http://www.aqua-mail.com [http://www.aqua-mail.com] On 15 December 2015 17:06:24 Robbie Wright < [mailto:robbie@siuslawbroadband.com]robbie@siuslawbroadband.com [mailto:robbie@siuslawbroadband.com]> wrote: Recently upgraded from 69xx to 72xx after we renewed our license and it broke all of our RRD's, resulting in the "Error Drawing Graph" issue.
Running this "./poller.php -h <hostname> -m ports -d" gives me two noticeable, but unobjectionable to me things: SNMP_STATUS[TRUE] o Device status Device is reachable by PING (8.28ms) and SNMP (81.63ms) RRD CMD[update /opt/observium/rrd/hostname/status.rrd N:1] RRD RUNTIME[0.0012s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/hostname/status.rrd': Permission denied] RRD_STATUS[FALSE] and this at the end of the results: Log file '/opt/observium/logs/observium.log' is not writable, check file permissions.
So clearly some type of permission issue but not sure what happened during the upgrade that would have caused this. Any thoughts on what chmod's need to be set on the directories or what else could have went haywire?
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101 [tel:541-902-5101] _______________________________________________ observium mailing list observium@observium.org [mailto:observium%40observium.org] http:// postman.memetic.org/ cgi-bin/mailman/listinfo/ observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium%40observium.org] http://postman.memetic. org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Thanks Adam, yes, that was a wild goose chase. I did not run "./poller.php -h <hostname> -m ports -d" that command with su. I ran it again with sudo and the test seemed to work fine. Still have the root issue of error drawing graph and none of the ports being visible at all. On all devices. Device details like ping response, process, etc work, but none of the port info on switches, firewalls, routers, or servers works.
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 11:45 AM, Adam Armstrong adama@memetic.org wrote:
Did you generate the output from the first email as not-root?
http://wildgoosefestival.org/wp-content/uploads/2014/06/wild-goose-in-action...
I think it wants someone to chase it! :D
adam.
On 15/12/2015 19:43:35, Robbie Wright robbie@siuslawbroadband.com wrote: Adam, I'm not sure it can't write the rrd's, I think it can't read them, but that' just based on the two log outputs I provided before.
Tom, I don't think the FS is mounted read only but I'm not sure how to check that. I can sudo nano a new file into the /opt/observium folder without issue.
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 11:24 AM, Tom Laermans < tom.laermans@powersource.cx> wrote:
The debug log...
File system isn't mounted read only per chance?
On 15/12/2015 20:19, Adam Armstrong wrote:
Well what makes you think they can't write to the rrds? :>
adam.
On 15/12/2015 18:58:31, Robbie Wright robbie@siuslawbroadband.com robbie@siuslawbroadband.com wrote: 33 */6 * * * root /opt/observium/discovery.php -h all >> /dev/null 2>&1 */5 * * * * root /opt/observium/discovery.php -h new >> /dev/null 2>&1 */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Looks like root to me...
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 10:18 AM, Adam Armstrong < adama@memetic.org adama@memetic.org> wrote:
It seems that someone has modified your cron jobs to no longer be run as root, since your rrds were created by root and svn has no ability to change them.
You need to work out where the poller is run from. It should be /etc/cron.d/observium
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 15 December 2015 18:08:08 Robbie Wright < robbie@siuslawbroadband.comrobbie@siuslawbroadband.com> wrote:
Not sure who the poller is running as. What's the easy way to check that?
Yes, svn update has always been run as su.
/opt/observium/rrd is owned by www-data, not root unlike the other folders in the main directory. However, digging into the directory, all of the folders inside for each host are all owned by root.
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 9:24 AM, Adam Armstrong < adama@memetic.org adama@memetic.org> wrote:
Are you running your poller observium poller as a non-root user?
Did you svn up as root?
The contents of the rrd directory, and anything else written to by the poller, should be owned by the user who runs the poller.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 15 December 2015 17:06:24 Robbie Wright < robbie@siuslawbroadband.comrobbie@siuslawbroadband.com> wrote:
Recently upgraded from 69xx to 72xx after we renewed our license and it broke all of our RRD's, resulting in the "Error Drawing Graph" issue.
Running this "./poller.php -h <hostname> -m ports -d" gives me two noticeable, but unobjectionable to me things:
SNMP_STATUS[TRUE] o Device status Device is reachable by PING (8.28ms) and SNMP (81.63ms) RRD CMD[update /opt/observium/rrd/hostname/status.rrd N:1] RRD RUNTIME[0.0012s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/hostname/status.rrd': Permission denied] RRD_STATUS[FALSE]
and this at the end of the results:
Log file '/opt/observium/logs/observium.log' is not writable, check file permissions.
So clearly some type of permission issue but not sure what happened during the upgrade that would have caused this. Any thoughts on what chmod's need to be set on the directories or what else could have went haywire?
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101 _______________________________________________ observium mailing list observium@observium.org http:// postman.memetic.org/ cgi-bin/mailman/listinfo/ observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic. org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing listobservium@observium.orghttp://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
rm -rf mibs svn up
adam. On 15/12/2015 20:02:31, Robbie Wright robbie@siuslawbroadband.com wrote: Thanks Adam, yes, that was a wild goose chase. I did not run "./poller.php -h <hostname> -m ports -d" that command with su. I ran it again with sudo and the test seemed to work fine. Still have the root issue of error drawing graph and none of the ports being visible at all. On all devices. Device details like ping response, process, etc work, but none of the port info on switches, firewalls, routers, or servers works.
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101
On Tue, Dec 15, 2015 at 11:45 AM, Adam Armstrong <adama@memetic.org [mailto:adama@memetic.org]> wrote:
Did you generate the output from the first email as not-root?
http://wildgoosefestival.org/wp-content/uploads/2014/06/wild-goose-in-action... [http://wildgoosefestival.org/wp-content/uploads/2014/06/wild-goose-in-action...]
I think it wants someone to chase it! :D
adam. On 15/12/2015 19:43:35, Robbie Wright <robbie@siuslawbroadband.com [mailto:robbie@siuslawbroadband.com]> wrote: Adam, I'm not sure it can't write the rrd's, I think it can't read them, but that' just based on the two log outputs I provided before.
Tom, I don't think the FS is mounted read only but I'm not sure how to check that. I can sudo nano a new file into the /opt/observium folder without issue.
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101 [tel:541-902-5101]
On Tue, Dec 15, 2015 at 11:24 AM, Tom Laermans <tom.laermans@powersource.cx [mailto:tom.laermans@powersource.cx]> wrote:
The debug log...
File system isn't mounted read only per chance?
On 15/12/2015 20:19, Adam Armstrong wrote:
Well what makes you think they can't write to the rrds? :>
adam. On 15/12/2015 18:58:31, Robbie Wright robbie@siuslawbroadband.com [mailto:robbie@siuslawbroadband.com] wrote: 33 */6 * * * root /opt/observium/discovery.php -h all >> /dev/null 2>&1 */5 * * * * root /opt/observium/discovery.php -h new >> /dev/null 2>&1 */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Looks like root to me...
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101 [tel:541-902-5101]
On Tue, Dec 15, 2015 at 10:18 AM, Adam Armstrong < [mailto:adama@memetic.org]adama@memetic.org [mailto:adama@memetic.org]> wrote:
It seems that someone has modified your cron jobs to no longer be run as root, since your rrds were created by root and svn has no ability to change them. You need to work out where the poller is run from. It should be /etc/cron.d/observium Adam. Sent with AquaMail for Android http://www.aqua-mail.com [http://www.aqua-mail.com] On 15 December 2015 18:08:08 Robbie Wright < [mailto:robbie@siuslawbroadband.com]robbie@siuslawbroadband.com [mailto:robbie@siuslawbroadband.com]> wrote: Not sure who the poller is running as. What's the easy way to check that?
Yes, svn update has always been run as su.
/opt/observium/rrd is owned by www-data, not root unlike the other folders in the main directory. However, digging into the directory, all of the folders inside for each host are all owned by root.
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101 [tel:541-902-5101]
On Tue, Dec 15, 2015 at 9:24 AM, Adam Armstrong < [mailto:adama@memetic.org]adama@memetic.org [mailto:adama@memetic.org]> wrote:
Are you running your poller observium poller as a non-root user? Did you svn up as root? The contents of the rrd directory, and anything else written to by the poller, should be owned by the user who runs the poller. Adam. Sent with AquaMail for Android http://www.aqua-mail.com [http://www.aqua-mail.com] On 15 December 2015 17:06:24 Robbie Wright < [mailto:robbie@siuslawbroadband.com]robbie@siuslawbroadband.com [mailto:robbie@siuslawbroadband.com]> wrote: Recently upgraded from 69xx to 72xx after we renewed our license and it broke all of our RRD's, resulting in the "Error Drawing Graph" issue.
Running this "./poller.php -h <hostname> -m ports -d" gives me two noticeable, but unobjectionable to me things: SNMP_STATUS[TRUE] o Device status Device is reachable by PING (8.28ms) and SNMP (81.63ms) RRD CMD[update /opt/observium/rrd/hostname/status.rrd N:1] RRD RUNTIME[0.0012s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/hostname/status.rrd': Permission denied] RRD_STATUS[FALSE] and this at the end of the results: Log file '/opt/observium/logs/observium.log' is not writable, check file permissions.
So clearly some type of permission issue but not sure what happened during the upgrade that would have caused this. Any thoughts on what chmod's need to be set on the directories or what else could have went haywire?
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101 [tel:541-902-5101] _______________________________________________ observium mailing list observium@observium.org [mailto:observium%40observium.org] http:// postman.memetic.org/ cgi-bin/mailman/listinfo/ observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium%40observium.org] http://postman.memetic. org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Thanks Adam, that mostly did the trick. Dropped all the mibs, ran svn, and it didn't work out of the gate. Ran discovery on all devices again and we're back in business. Thanks for your help!
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 1:14 PM, Adam Armstrong adama@memetic.org wrote:
rm -rf mibs svn up
adam.
On 15/12/2015 20:02:31, Robbie Wright robbie@siuslawbroadband.com wrote: Thanks Adam, yes, that was a wild goose chase. I did not run "./poller.php -h <hostname> -m ports -d" that command with su. I ran it again with sudo and the test seemed to work fine. Still have the root issue of error drawing graph and none of the ports being visible at all. On all devices. Device details like ping response, process, etc work, but none of the port info on switches, firewalls, routers, or servers works.
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 11:45 AM, Adam Armstrong adama@memetic.org wrote:
Did you generate the output from the first email as not-root?
http://wildgoosefestival.org/wp-content/uploads/2014/06/wild-goose-in-action...
I think it wants someone to chase it! :D
adam.
On 15/12/2015 19:43:35, Robbie Wright robbie@siuslawbroadband.com wrote: Adam, I'm not sure it can't write the rrd's, I think it can't read them, but that' just based on the two log outputs I provided before.
Tom, I don't think the FS is mounted read only but I'm not sure how to check that. I can sudo nano a new file into the /opt/observium folder without issue.
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 11:24 AM, Tom Laermans < tom.laermans@powersource.cx> wrote:
The debug log...
File system isn't mounted read only per chance?
On 15/12/2015 20:19, Adam Armstrong wrote:
Well what makes you think they can't write to the rrds? :>
adam.
On 15/12/2015 18:58:31, Robbie Wright robbie@siuslawbroadband.com robbie@siuslawbroadband.com wrote: 33 */6 * * * root /opt/observium/discovery.php -h all >> /dev/null 2>&1 */5 * * * * root /opt/observium/discovery.php -h new >> /dev/null 2>&1 */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Looks like root to me...
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 10:18 AM, Adam Armstrong < adama@memetic.org adama@memetic.org> wrote:
It seems that someone has modified your cron jobs to no longer be run as root, since your rrds were created by root and svn has no ability to change them.
You need to work out where the poller is run from. It should be /etc/cron.d/observium
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 15 December 2015 18:08:08 Robbie Wright < robbie@siuslawbroadband.comrobbie@siuslawbroadband.com> wrote:
Not sure who the poller is running as. What's the easy way to check that?
Yes, svn update has always been run as su.
/opt/observium/rrd is owned by www-data, not root unlike the other folders in the main directory. However, digging into the directory, all of the folders inside for each host are all owned by root.
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 9:24 AM, Adam Armstrong < adama@memetic.org adama@memetic.org> wrote:
Are you running your poller observium poller as a non-root user?
Did you svn up as root?
The contents of the rrd directory, and anything else written to by the poller, should be owned by the user who runs the poller.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 15 December 2015 17:06:24 Robbie Wright < robbie@siuslawbroadband.comrobbie@siuslawbroadband.com> wrote:
> Recently upgraded from 69xx to 72xx after we renewed our license and > it broke all of our RRD's, resulting in the "Error Drawing Graph" issue. > > Running this "./poller.php -h <hostname> -m ports -d" gives me two > noticeable, but unobjectionable to me things: > > SNMP_STATUS[TRUE] > o Device status Device is reachable by PING (8.28ms) and > SNMP (81.63ms) > RRD CMD[update /opt/observium/rrd/hostname/status.rrd N:1] > RRD RUNTIME[0.0012s] > RRD STDOUT[ERROR: opening '/opt/observium/rrd/hostname/status.rrd': > Permission denied] > RRD_STATUS[FALSE] > > and this at the end of the results: > > Log file '/opt/observium/logs/observium.log' is not writable, check > file permissions. > > So clearly some type of permission issue but not sure what happened > during the upgrade that would have caused this. Any thoughts on what > chmod's need to be set on the directories or what else could have went > haywire? > > Robbie Wright > Siuslaw Broadband https://siuslawbroadband.com > 541-902-5101 > _______________________________________________ > observium mailing list > observium@observium.org > http:// postman.memetic.org/ cgi-bin/mailman/listinfo/ > observium > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic. org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing listobservium@observium.orghttp://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
It's the horrible net-snmp .index bug.
If things ever spontaneously vanish, it's usually that net-snmp has corrupted its .index file. Tres annoyeaux.
adam. On 15/12/2015 21:50:43, Robbie Wright robbie@siuslawbroadband.com wrote: Thanks Adam, that mostly did the trick. Dropped all the mibs, ran svn, and it didn't work out of the gate. Ran discovery on all devices again and we're back in business. Thanks for your help!
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101
On Tue, Dec 15, 2015 at 1:14 PM, Adam Armstrong <adama@memetic.org [mailto:adama@memetic.org]> wrote:
rm -rf mibs svn up
adam. On 15/12/2015 20:02:31, Robbie Wright <robbie@siuslawbroadband.com [mailto:robbie@siuslawbroadband.com]> wrote: Thanks Adam, yes, that was a wild goose chase. I did not run "./poller.php -h <hostname> -m ports -d" that command with su. I ran it again with sudo and the test seemed to work fine. Still have the root issue of error drawing graph and none of the ports being visible at all. On all devices. Device details like ping response, process, etc work, but none of the port info on switches, firewalls, routers, or servers works.
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101 [tel:541-902-5101]
On Tue, Dec 15, 2015 at 11:45 AM, Adam Armstrong <adama@memetic.org [mailto:adama@memetic.org]> wrote:
Did you generate the output from the first email as not-root?
http://wildgoosefestival.org/wp-content/uploads/2014/06/wild-goose-in-action... [http://wildgoosefestival.org/wp-content/uploads/2014/06/wild-goose-in-action...]
I think it wants someone to chase it! :D
adam. On 15/12/2015 19:43:35, Robbie Wright <robbie@siuslawbroadband.com [mailto:robbie@siuslawbroadband.com]> wrote: Adam, I'm not sure it can't write the rrd's, I think it can't read them, but that' just based on the two log outputs I provided before.
Tom, I don't think the FS is mounted read only but I'm not sure how to check that. I can sudo nano a new file into the /opt/observium folder without issue.
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101 [tel:541-902-5101]
On Tue, Dec 15, 2015 at 11:24 AM, Tom Laermans <tom.laermans@powersource.cx [mailto:tom.laermans@powersource.cx]> wrote:
The debug log...
File system isn't mounted read only per chance?
On 15/12/2015 20:19, Adam Armstrong wrote:
Well what makes you think they can't write to the rrds? :>
adam. On 15/12/2015 18:58:31, Robbie Wright robbie@siuslawbroadband.com [mailto:robbie@siuslawbroadband.com] wrote: 33 */6 * * * root /opt/observium/discovery.php -h all >> /dev/null 2>&1 */5 * * * * root /opt/observium/discovery.php -h new >> /dev/null 2>&1 */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Looks like root to me...
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101 [tel:541-902-5101]
On Tue, Dec 15, 2015 at 10:18 AM, Adam Armstrong < [mailto:adama@memetic.org]adama@memetic.org [mailto:adama@memetic.org]> wrote:
It seems that someone has modified your cron jobs to no longer be run as root, since your rrds were created by root and svn has no ability to change them. You need to work out where the poller is run from. It should be /etc/cron.d/observium Adam. Sent with AquaMail for Android http://www.aqua-mail.com [http://www.aqua-mail.com] On 15 December 2015 18:08:08 Robbie Wright < [mailto:robbie@siuslawbroadband.com]robbie@siuslawbroadband.com [mailto:robbie@siuslawbroadband.com]> wrote: Not sure who the poller is running as. What's the easy way to check that?
Yes, svn update has always been run as su.
/opt/observium/rrd is owned by www-data, not root unlike the other folders in the main directory. However, digging into the directory, all of the folders inside for each host are all owned by root.
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101 [tel:541-902-5101]
On Tue, Dec 15, 2015 at 9:24 AM, Adam Armstrong < [mailto:adama@memetic.org]adama@memetic.org [mailto:adama@memetic.org]> wrote:
Are you running your poller observium poller as a non-root user? Did you svn up as root? The contents of the rrd directory, and anything else written to by the poller, should be owned by the user who runs the poller. Adam. Sent with AquaMail for Android http://www.aqua-mail.com [http://www.aqua-mail.com] On 15 December 2015 17:06:24 Robbie Wright < [mailto:robbie@siuslawbroadband.com]robbie@siuslawbroadband.com [mailto:robbie@siuslawbroadband.com]> wrote: Recently upgraded from 69xx to 72xx after we renewed our license and it broke all of our RRD's, resulting in the "Error Drawing Graph" issue.
Running this "./poller.php -h <hostname> -m ports -d" gives me two noticeable, but unobjectionable to me things: SNMP_STATUS[TRUE] o Device status Device is reachable by PING (8.28ms) and SNMP (81.63ms) RRD CMD[update /opt/observium/rrd/hostname/status.rrd N:1] RRD RUNTIME[0.0012s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/hostname/status.rrd': Permission denied] RRD_STATUS[FALSE] and this at the end of the results: Log file '/opt/observium/logs/observium.log' is not writable, check file permissions.
So clearly some type of permission issue but not sure what happened during the upgrade that would have caused this. Any thoughts on what chmod's need to be set on the directories or what else could have went haywire?
Robbie Wright Siuslaw Broadband [https://siuslawbroadband.com] 541-902-5101 [tel:541-902-5101] _______________________________________________ observium mailing list observium@observium.org [mailto:observium%40observium.org] http:// postman.memetic.org/ cgi-bin/mailman/listinfo/ observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium%40observium.org] http://postman.memetic. org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Good to know, thanks Adam.
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 1:59 PM, Adam Armstrong adama@memetic.org wrote:
It's the horrible net-snmp .index bug.
If things ever spontaneously vanish, it's usually that net-snmp has corrupted its .index file. Tres annoyeaux.
adam.
On 15/12/2015 21:50:43, Robbie Wright robbie@siuslawbroadband.com wrote: Thanks Adam, that mostly did the trick. Dropped all the mibs, ran svn, and it didn't work out of the gate. Ran discovery on all devices again and we're back in business. Thanks for your help!
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 1:14 PM, Adam Armstrong adama@memetic.org wrote:
rm -rf mibs svn up
adam.
On 15/12/2015 20:02:31, Robbie Wright robbie@siuslawbroadband.com wrote: Thanks Adam, yes, that was a wild goose chase. I did not run "./poller.php -h <hostname> -m ports -d" that command with su. I ran it again with sudo and the test seemed to work fine. Still have the root issue of error drawing graph and none of the ports being visible at all. On all devices. Device details like ping response, process, etc work, but none of the port info on switches, firewalls, routers, or servers works.
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 11:45 AM, Adam Armstrong adama@memetic.org wrote:
Did you generate the output from the first email as not-root?
http://wildgoosefestival.org/wp-content/uploads/2014/06/wild-goose-in-action...
I think it wants someone to chase it! :D
adam.
On 15/12/2015 19:43:35, Robbie Wright robbie@siuslawbroadband.com wrote: Adam, I'm not sure it can't write the rrd's, I think it can't read them, but that' just based on the two log outputs I provided before.
Tom, I don't think the FS is mounted read only but I'm not sure how to check that. I can sudo nano a new file into the /opt/observium folder without issue.
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 11:24 AM, Tom Laermans < tom.laermans@powersource.cx> wrote:
The debug log...
File system isn't mounted read only per chance?
On 15/12/2015 20:19, Adam Armstrong wrote:
Well what makes you think they can't write to the rrds? :>
adam.
On 15/12/2015 18:58:31, Robbie Wright robbie@siuslawbroadband.com robbie@siuslawbroadband.com wrote: 33 */6 * * * root /opt/observium/discovery.php -h all >> /dev/null 2>&1 */5 * * * * root /opt/observium/discovery.php -h new >> /dev/null 2>&1 */5 * * * * root /opt/observium/poller-wrapper.py 5 >> /dev/null 2>&1
Looks like root to me...
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 10:18 AM, Adam Armstrong < adama@memetic.org adama@memetic.org> wrote:
It seems that someone has modified your cron jobs to no longer be run as root, since your rrds were created by root and svn has no ability to change them.
You need to work out where the poller is run from. It should be /etc/cron.d/observium
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 15 December 2015 18:08:08 Robbie Wright < robbie@siuslawbroadband.comrobbie@siuslawbroadband.com> wrote:
Not sure who the poller is running as. What's the easy way to check that?
Yes, svn update has always been run as su.
/opt/observium/rrd is owned by www-data, not root unlike the other folders in the main directory. However, digging into the directory, all of the folders inside for each host are all owned by root.
Robbie Wright Siuslaw Broadband https://siuslawbroadband.com 541-902-5101
On Tue, Dec 15, 2015 at 9:24 AM, Adam Armstrong < adama@memetic.org adama@memetic.org> wrote:
> Are you running your poller observium poller as a non-root user? > > Did you svn up as root? > > The contents of the rrd directory, and anything else written to by > the poller, should be owned by the user who runs the poller. > > Adam. > > Sent with AquaMail for Android > http://www.aqua-mail.com > > On 15 December 2015 17:06:24 Robbie Wright < > robbie@siuslawbroadband.comrobbie@siuslawbroadband.com> wrote: > >> Recently upgraded from 69xx to 72xx after we renewed our license >> and it broke all of our RRD's, resulting in the "Error Drawing Graph" >> issue. >> >> Running this "./poller.php -h <hostname> -m ports -d" gives me two >> noticeable, but unobjectionable to me things: >> >> SNMP_STATUS[TRUE] >> o Device status Device is reachable by PING (8.28ms) and >> SNMP (81.63ms) >> RRD CMD[update /opt/observium/rrd/hostname/status.rrd N:1] >> RRD RUNTIME[0.0012s] >> RRD STDOUT[ERROR: opening '/opt/observium/rrd/hostname/status.rrd': >> Permission denied] >> RRD_STATUS[FALSE] >> >> and this at the end of the results: >> >> Log file '/opt/observium/logs/observium.log' is not writable, check >> file permissions. >> >> So clearly some type of permission issue but not sure what happened >> during the upgrade that would have caused this. Any thoughts on what >> chmod's need to be set on the directories or what else could have went >> haywire? >> >> Robbie Wright >> Siuslaw Broadband https://siuslawbroadband.com >> 541-902-5101 >> _______________________________________________ >> observium mailing list >> observium@observium.org >> http:// postman.memetic.org/ cgi-bin/mailman/listinfo/ >> observium >> http://postman.memetic.org/cgi-bin/mailman/listinfo/observium >> >> > _______________________________________________ > observium mailing list > observium@observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > _______________________________________________ observium mailing list observium@observium.org http://postman.memetic. org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing listobservium@observium.orghttp://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (3)
-
Adam Armstrong -
Robbie Wright -
Tom Laermans