Hi,
Overnight the update script brought us to 9304 and the schema up to 378. Immediately after, all of our devices when down and alerts went out. The problem was that the poller script SQL call was failing because it was looking for status = 1 and all of our boxes were in status = 0. poller couldn’t find the host even though the hostname matched hostname in devices. I did an “ update devices set status=1 where `disabled`=0; “ and poller started again but I’m sure something else it at play here.
Joey
as a follow on, poller now runs (as does discovery), but there is another process which I haven’t isolated yet which immediately takes the device from online back to offline (in the UI) while leaving status =1.
On Jun 30, 2018, at 08:28 , Joey Stanford nv0n@rmham.org wrote:
Hi,
Overnight the update script brought us to 9304 and the schema up to 378. Immediately after, all of our devices when down and alerts went out. The problem was that the poller script SQL call was failing because it was looking for status = 1 and all of our boxes were in status = 0. poller couldn’t find the host even though the hostname matched hostname in devices. I did an “ update devices set status=1 where `disabled`=0; “ and poller started again but I’m sure something else it at play here.
Joey
it seems I was still running poller-wrapper and not the new observium-wrapper …when I did that I got everything back
On Jun 30, 2018, at 08:51 , Joey Stanford nv0n@rmham.org wrote:
as a follow on, poller now runs (as does discovery), but there is another process which I haven’t isolated yet which immediately takes the device from online back to offline (in the UI) while leaving status =1.
On Jun 30, 2018, at 08:28 , Joey Stanford nv0n@rmham.org wrote:
Hi,
Overnight the update script brought us to 9304 and the schema up to 378. Immediately after, all of our devices when down and alerts went out. The problem was that the poller script SQL call was failing because it was looking for status = 1 and all of our boxes were in status = 0. poller couldn’t find the host even though the hostname matched hostname in devices. I did an “ update devices set status=1 where `disabled`=0; “ and poller started again but I’m sure something else it at play here.
Joey
Umm...
observium-wrapper is a symlink to poller-wrapper.py
lrwxrwxrwx 1 root root 17 Jun 22 2017 observium-wrapper -> poller-wrapper.py
adam.
------ Original Message ------ From: "Joey Stanford" nv0n@rmham.org To: "Observium" observium@observium.org Sent: 2018-06-30 16:06:30 Subject: Re: [Observium] 9304 issues
it seems I was still running poller-wrapper and not the new observium-wrapper …when I did that I got everything back
On Jun 30, 2018, at 08:51 , Joey Stanford nv0n@rmham.org wrote:
as a follow on, poller now runs (as does discovery), but there is another process which I haven’t isolated yet which immediately takes the device from online back to offline (in the UI) while leaving status =1.
On Jun 30, 2018, at 08:28 , Joey Stanford nv0n@rmham.org wrote:
Hi,
Overnight the update script brought us to 9304 and the schema up to 378. Immediately after, all of our devices when down and alerts went out. The problem was that the poller script SQL call was failing because it was looking for status = 1 and all of our boxes were in status = 0. poller couldn’t find the host even though the hostname matched hostname in devices. I did an “ update devices set status=1 where `disabled`=0; “ and poller started again but I’m sure something else it at play here.
Joey
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On Jun 30, 2018, at 09:08 , Adam Armstrong adama@memetic.org wrote:
Umm...
observium-wrapper is a symlink to poller-wrapper.py
lrwxrwxrwx 1 root root 17 Jun 22 2017 observium-wrapper -> poller-wrapper.py
Yeah I don’t get it yet either. Everything broke in our Obs instance immediately after the svn update at midnight. I’m still running everything by hand with -dd to figure out what’s going on.
So the good news is that the poller is running and we have recovery emails and alerts for everything, status is back to 0, but the UI still shows everything is down.
I’ve traced the error down to SNMP
add_mibdir: strings scanned in from /opt/observium/mibs/rfc/.index are too large. count = 131 MIB search path: /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp Cannot find module (SNMPv2-MIB): At line 0 in (none) sysObjectID.0: Unknown Object Identifier (Sub-id not found: (top) -> sysObjectID) sysUpTime.0: Unknown Object Identifier (Sub-id not found: (top) -> sysUpTime)
and that let me to http://jira.observium.org/browse/OBS-1702 http://jira.observium.org/browse/OBS-1702 so I’m on the right path. Sorry for the noise… I have a lot of folks worried since the system is being used for the fires in Colorado
Get those damn fires out, I'm flying to DEN then driving to Grand Junction tomorrow :)
adam.
On 2018-06-30 17:46, Joey Stanford wrote:
and that let me to http://jira.observium.org/browse/OBS-1702 so I’m on the right path. Sorry for the noise… I have a lot of folks worried since the system is being used for the fires in Colorado _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
This is not our issue,
update your system, this old-old net-snmp issue.. probably you use Centos 6 or something same old..
FAQ: http://docs.observium.org/faq/#all-my-hosts-seem-down-to-observium-snmp-does...
Joey Stanford mailto:nv0n@rmham.org 30 June 2018 at 19:46 and that let me to http://jira.observium.org/browse/OBS-1702 so I’m on the right path. Sorry for the noise… I have a lot of folks worried since the system is being used for the fires in Colorado
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium Joey Stanford mailto:nv0n@rmham.org 30 June 2018 at 19:40 I’ve traced the error down to SNMP
add_mibdir: strings scanned in from /opt/observium/mibs/rfc/.index are too large. count = 131 MIB search path: /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp Cannot find module (SNMPv2-MIB): At line 0 in (none) sysObjectID.0: Unknown Object Identifier (Sub-id not found: (top) -> sysObjectID) sysUpTime.0: Unknown Object Identifier (Sub-id not found: (top) -> sysUpTime) _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium Joey Stanford mailto:nv0n@rmham.org 30 June 2018 at 18:38 So the good news is that the poller is running and we have recovery emails and alerts for everything, status is back to 0, but the UI still shows everything is down. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium Joey Stanford mailto:nv0n@rmham.org 30 June 2018 at 18:16
Yeah I don’t get it yet either. Everything broke in our Obs instance immediately after the svn update at midnight. I’m still running everything by hand with -dd to figure out what’s going on. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium Adam Armstrong mailto:adama@memetic.org 30 June 2018 at 18:08 Umm...
observium-wrapper is a symlink to poller-wrapper.py
lrwxrwxrwx 1 root root 17 Jun 22 2017 observium-wrapper -> poller-wrapper.py
adam.
On Jun 30, 2018, at 13:25 , Mike Stupalov mike@observium.org wrote:
update your system, this old-old net-snmp issue.. probably you use Centos 6 or something same old..
I already had the discovery.php -u as part of the update script.
And I’m on the latest version of Ubuntu 16.04
Why it manifested now I don’t know but once I found it, I fixed it with
sudo rm -rf /var/lib/snmp/mib_indexes/* sudo rm -f /opt/observium/mibs/*/.index sudo rm -f /opt/observium/*/.index
Interestingly enough, there were no .index files under /opt/observium but there were some in /var/lib/snmp/mib_indexes
Yeah. This is super annoying. I have no idea why they changed the location. It makes it hard to tell people what to delete!
This is something we used to see a *lot*, but these days it's thankfully rarer as a problem. I'm not sure why, but I'm glad.
Adam.
Sent from BlueMail
On 30 Jun 2018, 22:26, at 22:26, Joey Stanford nv0n@rmham.org wrote:
On Jun 30, 2018, at 13:25 , Mike Stupalov mike@observium.org wrote:
update your system, this old-old net-snmp issue.. probably you use
Centos 6 or something same old..
I already had the discovery.php -u as part of the update script.
And I’m on the latest version of Ubuntu 16.04
Why it manifested now I don’t know but once I found it, I fixed it with
sudo rm -rf /var/lib/snmp/mib_indexes/* sudo rm -f /opt/observium/mibs/*/.index sudo rm -f /opt/observium/*/.index
Interestingly enough, there were no .index files under /opt/observium but there were some in /var/lib/snmp/mib_indexes
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi,
immediately after 'svn up' you must immediately do ./discovery.php -u
:P
Joey Stanford mailto:nv0n@rmham.org 30 June 2018 at 18:16
Yeah I don’t get it yet either. Everything broke in our Obs instance immediately after the svn update at midnight. I’m still running everything by hand with -dd to figure out what’s going on. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium Adam Armstrong mailto:adama@memetic.org 30 June 2018 at 18:08 Umm...
observium-wrapper is a symlink to poller-wrapper.py
lrwxrwxrwx 1 root root 17 Jun 22 2017 observium-wrapper -> poller-wrapper.py
adam.
Joey Stanford mailto:nv0n@rmham.org 30 June 2018 at 18:06 it seems I was still running poller-wrapper and not the new observium-wrapper …when I did that I got everything back
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium Joey Stanford mailto:nv0n@rmham.org 30 June 2018 at 17:51 as a follow on, poller now runs (as does discovery), but there is another process which I haven’t isolated yet which immediately takes the device from online back to offline (in the UI) while leaving status =1.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium Joey Stanford mailto:nv0n@rmham.org 30 June 2018 at 17:28 Hi,
Overnight the update script brought us to 9304 and the schema up to 378. Immediately after, all of our devices when down and alerts went out. The problem was that the poller script SQL call was failing because it was looking for status = 1 and all of our boxes were in status = 0. poller couldn’t find the host even though the hostname matched hostname in devices. I did an “ update devices set status=1 where `disabled`=0; “ and poller started again but I’m sure something else it at play here.
Joey
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (3)
-
Adam Armstrong -
Joey Stanford -
Mike Stupalov