Hello, we're trying to implement observium and it is about 400 devices. At the moment there are 226 but 2 of them suddenly went offline and cannot be brought back up with polling and I'm not sure if the snmp gets blocked on out end or on the devices end. There are also Palo Alto firewalls present at the location and I'm thinking they're causing the issue but just in case im posting this here also. Here is the ./poller with debug if it helps: DEBUG! DEFINITIONS Time : 0.056319 ms
DEFINITIONS Memory: 18.6MB
SQL[SELECT * FROM `observium_attribs`] SQL RUNTIME[0.00029492s]
SQL[SELECT @@SESSION.sql_mode;] SQL RUNTIME[0.00012803s]
SQL[SET SESSION `sql_mode` = 'NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'] SQL RUNTIME[6.7E-5s] DB mode(s) disabled: STRICT_TRANS_TABLES, ONLY_FULL_GROUP_BY, NO_ZERO_DATE, NO_ZERO_IN_DATE, ERROR_FOR_DIVISION_BY_ZERO
SQL[SELECT TIMEDIFF(NOW(), UTC_TIMESTAMP);] SQL RUNTIME[0.000103s]
SQL[SELECT * FROM `config`] SQL RUNTIME[0.0001359s]
CMD[/bin/hostname -f]
CMD EXITCODE[0] CMD RUNTIME[0.0056s] CMD EXITDELAY[5ms] STDOUT[ observium ]
___ _ _ / _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___ | | | || '_ \ / __| / _ | '__|\ \ / /| || | | || '_ ` _ \ | |_| || |_) |__ | __/| | \ V / | || |_| || | | | | | ___/ |_.__/ |___/ ___||_| _/ |_| __,_||_| |_| |_| Observium Community Edition 24.12.13800 https://www.observium.org
CMD[/opt/observium/scripts/distro]
CMD EXITCODE[0] CMD RUNTIME[0.0125s] STDOUT[ Linux|4.15.0-213-generic|amd64|Ubuntu|18.04|vmware| ]
CMD[/usr/bin/env python --version 2>&1]
CMD EXITCODE[0] CMD RUNTIME[0.0028s] STDOUT[ Python 2.7.17 ]
CMD[/usr/bin/env python3 --version 2>&1]
CMD EXITCODE[0] CMD RUNTIME[0.0025s] STDOUT[ Python 3.6.9 ]
SQL[SELECT VERSION();] SQL RUNTIME[0.00018501s]
CMD[/usr/bin/snmpget --version 2>&1]
CMD EXITCODE[0] CMD RUNTIME[0.0024s] STDOUT[ NET-SNMP version: 5.7.3 ]
CMD[/usr/bin/rrdtool --version | head -n1]
CMD EXITCODE[0] CMD RUNTIME[0.0075s] STDOUT[ RRDtool 1.7.0 Copyright by Tobias Oetiker tobi@oetiker.ch ]
CMD[/usr/bin/fping -v 2>&1]
CMD EXITCODE[0] CMD RUNTIME[0.0022s] STDOUT[ /usr/bin/fping: Version 4.0 /usr/bin/fping: comments to david@schweikert.ch ]
CMD[/usr/sbin/apache2 -v | awk '/Server version:/ {print $3}']
CMD EXITCODE[0] CMD RUNTIME[0.044s] STDOUT[ Apache/2.4.29 ]
SQL[SELECT @@SESSION.sql_mode;] SQL RUNTIME[0.00016904s]
SQL[SHOW VARIABLES LIKE 'log_bin'] SQL RUNTIME[0.01490617s]
SQL[SHOW VARIABLES LIKE 'character_set_connection'] SQL RUNTIME[0.00202799s]
##### Observium #####
o Version 24.12.13800 o Released 24th December 2024
##### Software versions #####
o OS Linux 4.15.0-213-generic [amd64] (Ubuntu 18.04) o Apache 2.4.29 o PHP 7.2.24-0ubuntu0.18.04.17 (OPcache: ENABLED) o Python 3.6.9 (python3 not used as default python) o MySQL 5.7.42-0ubuntu0.18.04.1 (extension: mysqli 5.0.12-dev) o SNMP NET-SNMP 5.7.3 o RRDtool 1.7.0 o Fping 4.0 (IPv4 and IPv6) o Fetch cURL 7.58.0 (OpenSSL/1.1.1, LibZ 1.2.11)
##### Memory Limit #####
o PHP Unlimited
##### DB info #####
o DB schema 495 o MySQL binlog OFF o MySQL mode NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Friday, 13-Jun-25 13:01:23 CEST o PHP +02:00 o MySQL +02:00
SQL[SELECT * FROM `alerts_maint` WHERE `maint_start` < '1749812483' AND `maint_end` > '1749812483'] SQL RUNTIME[0.00025201s] ##### Starting polling run at 2025-06-13 13:01:23 #####
SQL[SELECT `device_id` FROM `devices` WHERE `disabled` = 0 AND `hostname` LIKE 'upssrv2.komunala.local' AND `poller_id` = '0' ORDER BY `device_id` ASC] SQL RUNTIME[0.0003252s]
SQL[SELECT * FROM `devices` WHERE `device_id` = '1052'] SQL RUNTIME[0.00029206s]
SQL[SELECT * FROM `entity_attribs` WHERE `entity_type` = 'device' AND `entity_id` = '1052'] SQL RUNTIME[0.00020504s] Array ( [device_id] => 1052 [poller_id] => 0 [hostname] => upssrv2.komunala.local [sysName] => upssrv2.komunala.local [label] => [ip] => 213.250.36.26 [snmp_community] => Public95Hidden [snmp_authlevel] => [snmp_authname] => [snmp_authpass] => [snmp_authalgo] => [snmp_cryptopass] => [snmp_cryptoalgo] => [snmp_context] => [snmpable] => [snmp_version] => v2c [snmp_port] => 10181 [snmp_timeout] => 15 [snmp_retries] => 5 [snmp_maxrep] => 12 [ssh_port] => 22 [agent_version] => [snmp_transport] => udp [bgpLocalAs] => [snmpEngineID] => 800000000300C0B7C71A77 [sysObjectID] => [sysDescr] => APC Web/SNMP Management Card (MB:v3.8.6 PF:v5.0.3 PN:apc_hw05_aos_503.bin AF1:v5.0.3 AN1:apc_hw05_sumx_503.bin MN:AP9630 HR:05 SN: ZA1023016503 MD:06/04/2010) (Embedded PowerNet SNMP Agent SW v2.2 compatible) [sysContact] => sos@minicom.si [version] => 5.0.3 [hardware] => Smart-UPS 3000 UPS 09.3 / MCU 19.0 / UBL 08.5 / MBL 19.0 [vendor] => APC [features] => Smart-UPS DSP [location] => Kopaliska Ul., Murska Sobota, SI [os] => generic [status] => 0 [status_type] => ping [ignore] => 0 [ignore_until] => [asset_tag] => [disabled] => 0 [uptime] => 13256450 [last_rebooted] => 1736481732 [force_discovery] => 0 [last_polled] => 2025-06-12 16:27:21 [last_discovered] => 2025-06-13 08:26:30 [last_alerter] => 2025-06-13 12:45:59 [last_polled_timetaken] => 3.06 [last_discovered_timetaken] => 387.35 [purpose] => [type] => power [serial] => ZA1023016503 [icon] => [device_state] => a:5:{s:15:"poller_mod_perf";a:20:{s:6:"system";d:0.2138;s:2:"os";d:0.3433;s:7:"sensors";d:0.9053;s:6:"status";d:0.3725;s:7:"counter";d:0.0006;s:10:"processors";d:0.0006;s:8:"mempools";d:0.0003;s:7:"storage";d:0.0006;s:8:"netstats";d:0.532;s:7:"ucd-mib";d:0.0002;s:13:"ipSystemStats";d:0.1151;s:5:"ports";d:0.4809;s:15:"printersupplies";d:0.0007;s:10:"ucd-diskio";d:0.0005;s:4:"ospf";d:0.0002;s:3:"sla";d:0.0009;s:14:"mac-accounting";d:0.0013;s:15:"entity-physical";d:0.0005;s:9:"fdb-table";d:0.0033;s:6:"graphs";d:0.0004;}s:17:"poller_ports_perf";a:3:{s:6:"IF-MIB";d:0.2936229705810547;s:13:"EtherLike-MIB";d:0.07470893859863281;s:6:"IP-MIB";d:0.09396791458129883;}s:14:"poller_history";a:288:{i:1749738438;d:3.0618;i:1749738208;d:4.2238;i:1749738140;d:3.4869;i:1749737855;d:3.0784;i:1749737626;d:4.0923;i:1749737519;d:3.1383;i:1749737192;d:3.0538;i:1749737035;d:3.4784;i:1749736931;d:3.5268;i:1749736570;d:2.9656;i:1749736442;d:3.5027;i:1749736281;d:2.8993;i:1749736004;d:2.9603;i:1749735809;d:3.0529;i:1749735638;d:3.0026;i:1749735360;d:3.3181;i:1749735226;d:4.4443;i:1749734801;d:3.0956;i:1749734624;d:2.9595;i:1749734423;d:3.0414;i:1749734143;d:3.1799;i:1749734007;d:3.0043;i:1749733916;d:3.361;i:1749733595;d:3.2164;i:1749733385;d:3.2873;i:1749733296;d:3.0735;i:1749732966;d:4.0385;i:1749732756;d:3.0887;i:1749732714;d:4.0922;i:1749732378;d:4.7093;i:1749732222;d:3.0779;i:1749732131;d:3.3076;i:1749731835;d:3.193;i:1749731733;d:3.642;i:1749731502;d:3.2965;i:1749731170;d:3.295;i:1749731060;d:4.6041;i:1749730932;d:3.3409;i:1749730635;d:3.3308;i:1749730393;d:6.0173;i:1749730049;d:3.5515;i:1749729843;d:4.8746;i:1749729810;d:3.4133;i:1749729456;d:3.4072;i:1749729163;d:3.3064;i:1749729129;d:3.3968;i:1749728947;d:5.2722;i:1749728634;d:2.9416;i:1749728467;d:3.1248;i:1749728282;d:3.0656;i:1749728011;d:4.1433;i:1749727901;d:4.0402;i:1749727613;d:3.2587;i:1749727400;d:4.3347;i:1749727320;d:3.6798;i:1749726995;d:4.1341;i:1749726804;d:3.2313;i:1749726718;d:4.3125;i:1749726441;d:3.121;i:1749726304;d:4.0524;i:1749726249;d:3.652;i:1749725828;d:3.4258;i:1749725636;d:3.0245;i:1749725538;d:3.6954;i:1749725216;d:3.167;i:1749725025;d:4.1817;i:1749724946;d:3.0821;i:1749724605;d:3.1564;i:1749724440;d:4.4093;i:1749724041;d:3.2009;i:1749723769;d:3.3049;i:1749723763;d:3.2371;i:1749723437;d:4.7006;i:1749723173;d:4.5375;i:1749722603;d:4.8981;i:1749722599;d:5.6263;i:1749722024;d:5.1299;i:1749721720;d:3.1521;i:1749721418;d:3.3172;i:1749720829;d:4.2989;i:1749720692;d:5.8752;i:1749720341;d:3.233;i:1749720207;d:3.4445;i:1749720085;d:3.1147;i:1749719790;d:3.1086;i:1749719609;d:3.075;i:1749719493;d:2.8119;i:1749719036;d:4.6502;i:1749718894;d:3.0848;i:1749718640;d:2.9379;i:1749718416;d:3.2494;i:1749718343;d:3.4622;i:1749718006;d:3.0574;i:1749717811;d:3.1954;i:1749717729;d:3.1686;i:1749717407;d:3.0071;i:1749717280;d:3.5784;i:1749717187;d:3.513;i:1749716812;d:3.0417;i:1749716581;d:2.983;i:1749716523;d:3.2777;i:1749716222;d:4.5187;i:1749715957;d:3.5873;i:1749715884;d:3.0163;i:1749715596;d:4.3571;i:1749715429;d:3.1659;i:1749715310;d:4.928;i:1749715008;d:3.0166;i:1749714794;d:3.7413;i:1749714740;d:4.2167;i:1749714394;d:3.0739;i:1749714197;d:3.105;i:1749713599;d:2.8258;i:1749713475;d:3.0262;i:1749713202;d:3.0628;i:1749713010;d:3.0971;i:1749712893;d:3.0806;i:1749712606;d:3.1635;i:1749712287;d:3.2386;i:1749712001;d:3.4897;i:1749711661;d:3.1639;i:1749711393;d:2.9135;i:1749711073;d:3.0417;i:1749710130;d:4.1916;i:1749709570;d:3.1502;i:1749709342;d:3.0585;i:1749709261;d:2.94;i:1749708181;d:5.0987;i:1749707811;d:4.0774;i:1749707573;d:3.3582;i:1749707200;d:4.116;i:1749706882;d:4.8638;i:1749706865;d:3.4298;i:1749706791;d:5.9448;i:1749706778;d:3.743;i:1749705947;d:3.0158;i:1749705814;d:2.9233;i:1749705697;d:3.0277;i:1749705445;d:3.1456;i:1749705262;d:3.1111;i:1749705136;d:3.1943;i:1749704824;d:3.2105;i:1749704709;d:4.3898;i:1749704571;d:3.1006;i:1749704184;d:3.074;i:1749703963;d:3.1956;i:1749703944;d:3.4926;i:1749703594;d:4.093;i:1749703434;d:3.1553;i:1749703295;d:3.0769;i:1749702978;d:2.9163;i:1749702802;d:2.9788;i:1749702695;d:3.247;i:1749702432;d:3.2204;i:1749702418;d:3.0743;i:1749702231;d:2.9114;i:1749702119;d:2.9206;i:1749702039;d:7.9277;i:1749702037;d:7.3172;i:1749701918;d:2.9738;i:1749701589;d:3.9971;i:1749701194;d:3.5617;i:1749701041;d:2.9469;i:1749700846;d:3.1473;i:1749700586;d:3.0539;i:1749700402;d:2.8453;i:1749700231;d:3.0317;i:1749699987;d:3.0497;i:1749699685;d:2.7944;i:1749699394;d:3.0414;i:1749699217;d:3.0233;i:1749698812;d:3.0028;i:1749698452;d:2.7561;i:1749698210;d:3.0167;i:1749696335;d:2.9227;i:1749695252;d:2.95;i:1749694583;d:3.1199;i:1749693985;d:2.8845;i:1749693624;d:4.8865;i:1749693356;d:2.98;i:1749693106;d:2.8904;i:1749692820;d:2.8787;i:1749692585;d:3.0068;i:1749692456;d:2.9868;i:1749692191;d:3.028;i:1749691993;d:3.0677;i:1749691902;d:3.5402;i:1749691576;d:3.0194;i:1749691394;d:2.9704;i:1749691296;d:3.179;i:1749691008;d:2.9235;i:1749690785;d:2.9993;i:1749690705;d:3.389;i:1749690411;d:2.9714;i:1749690214;d:3.0422;i:1749690088;d:3.0665;i:1749689805;d:2.8724;i:1749689441;d:2.9037;i:1749689177;d:2.9579;i:1749689001;d:2.8615;i:1749688892;d:2.9507;i:1749688602;d:3.9019;i:1749688408;d:2.8358;i:1749687969;d:2.8371;i:1749687777;d:2.9357;i:1749687387;d:3.0137;i:1749687154;d:2.8007;i:1749687054;d:3.4541;i:1749686782;d:3.1283;i:1749686524;d:3.3633;i:1749686229;d:2.9402;i:1749685620;d:2.9339;i:1749685284;d:3.0794;i:1749685231;d:5.6467;i:1749685035;d:3.1549;i:1749681394;d:3.0682;i:1749681218;d:3.6287;i:1749681121;d:3.2408;i:1749680916;d:2.8448;i:1749680601;d:4.0062;i:1749680498;d:3.1405;i:1749680194;d:3.0757;i:1749679991;d:2.9302;i:1749679915;d:3.1371;i:1749679592;d:3.3427;i:1749679451;d:3.2951;i:1749679318;d:2.9649;i:1749678998;d:2.921;i:1749678821;d:2.7459;i:1749678689;d:2.9294;i:1749678392;d:2.8998;i:1749678203;d:2.879;i:1749678055;d:3.0618;i:1749677779;d:2.97;i:1749677619;d:2.9318;i:1749677430;d:2.9083;i:1749677222;d:3.1604;i:1749676980;d:3.0918;i:1749676849;d:2.9225;i:1749676605;d:2.9696;i:1749676386;d:2.771;i:1749676297;d:3.045;i:1749675981;d:3.4345;i:1749675859;d:3.1831;i:1749675395;d:2.9036;i:1749674790;d:3.0451;i:1749674622;d:4.0518;i:1749674518;d:2.8897;i:1749674193;d:2.8084;i:1749673848;d:3.0124;i:1749673619;d:2.925;i:1749672919;d:2.8596;i:1749672655;d:2.8645;i:1749672324;d:5.3573;i:1749669225;d:2.9646;i:1749669097;d:2.9559;i:1749668804;d:2.8976;i:1749668705;d:2.9129;i:1749668476;d:3.0896;i:1749668183;d:2.9723;i:1749667977;d:2.9534;i:1749667895;d:3.1282;i:1749667604;d:3.8656;i:1749667436;d:2.9998;i:1749667340;d:3.5311;i:1749667011;d:2.9199;i:1749666790;d:2.742;i:1749666679;d:3.0625;i:1749666390;d:3.1936;i:1749666295;d:3.1553;i:1749666119;d:3.3232;i:1749665803;d:4.0631;i:1749665551;d:3.2739;i:1749665519;d:3.0565;i:1749665236;d:4.921;i:1749665079;d:2.9497;i:1749664906;d:3.1913;i:1749664601;d:3.059;i:1749664458;d:3.079;i:1749664363;d:3.177;i:1749664063;d:3.0919;i:1749663851;d:3.6324;i:1749663734;d:3.1656;i:1749663368;d:3.0413;i:1749663226;d:2.9456;i:1749663130;d:3.2162;i:1749662800;d:3.0637;i:1749662643;d:3.0931;}s:17:"discovery_history";a:100:{i:1749795603;d:387.3494;i:1749725319;d:12.3666;i:1749725218;d:12.6031;i:1749703571;d:11.9745;i:1749681701;d:12.1248;i:1749660292;d:12.3958;i:1749660264;d:14.2735;i:1749638887;d:12.9033;i:1749574050;d:11.9178;i:1749573969;d:11.9894;i:1749552575;d:12.0187;i:1749552439;d:12.2768;i:1749530690;d:11.9339;i:1749509518;d:12.8574;i:1749509460;d:12.7049;i:1749487391;d:17.9025;i:1749487314;d:12.6653;i:1749466076;d:12.5715;i:1749466055;d:12.7853;i:1749444091;d:11.8692;i:1749444018;d:11.996;i:1749422433;d:11.7268;i:1749401116;d:11.8017;i:1749379241;d:11.5649;i:1749357629;d:11.5663;i:1749335988;d:11.3901;i:1749314718;d:12.4409;i:1749314675;d:11.2592;i:1749293096;d:11.7146;i:1749293083;d:11.396;i:1749271307;d:11.3211;i:1749249862;d:12.4283;i:1749228221;d:11.6094;i:1749228188;d:12.6402;i:1749206534;d:11.7667;i:1749206521;d:11.5338;i:1749184914;d:11.684;i:1749184895;d:11.7512;i:1749163523;d:11.6353;i:1749163499;d:11.6857;i:1749141874;d:11.3587;i:1749141843;d:11.4611;i:1749120278;d:11.8961;i:1749098359;d:11.719;i:1749076996;d:11.3718;i:1749055373;d:11.5598;i:1749055360;d:11.5001;i:1749033678;d:13.8816;i:1749028573;d:11.6227;i:1749012340;d:11.4993;i:1749012312;d:11.1807;i:1748990486;d:11.5018;i:1748969039;d:11.7778;i:1748969002;d:11.5313;i:1748947453;d:11.1967;i:1748947378;d:12.2244;i:1748861222;d:12.5562;i:1748861130;d:11.8993;i:1748839619;d:12.0593;i:1748839566;d:11.6181;i:1748817808;d:11.6571;i:1748796250;d:11.5268;i:1748796179;d:11.6209;i:1748774850;d:11.6775;i:1748774793;d:11.4434;i:1748753154;d:11.7982;i:1748753122;d:11.9909;i:1748731303;d:11.6366;i:1748710004;d:11.9226;i:1748709949;d:11.6838;i:1748688188;d:11.8362;i:1748666710;d:11.9111;i:1748666639;d:11.5911;i:1748645082;d:11.7618;i:1748623649;d:11.5698;i:1748623632;d:12.6731;i:1748602034;d:13.3614;i:1748601963;d:12.2858;i:1748580312;d:12.1121;i:1748580229;d:12.2093;i:1748558805;d:12.0051;i:1748558786;d:11.9119;i:1748536953;d:13.0996;i:1748536875;d:12.1898;i:1748515292;d:12.4839;i:1748493937;d:11.836;i:1748472382;d:11.481;i:1748472353;d:12.0042;i:1748450781;d:11.4484;i:1748450761;d:11.5744;i:1748428936;d:11.6852;i:1748428923;d:11.7349;i:1748407534;d:11.4526;i:1748385739;d:12.4064;i:1748364369;d:11.8435;i:1748364348;d:11.7349;i:1748342693;d:12.1723;i:1748321155;d:11.5715;i:1748321139;d:11.433;i:1748299389;d:11.5605;}s:18:"discovery_mod_perf";a:26:{s:5:"vlans";d:6.0366;s:11:"ports_vlans";N;s:2:"os";d:0.0084;s:4:"mibs";d:111.9102;s:3:"vrf";d:0.0148;s:5:"ports";d:30.218;s:11:"ports-stack";d:6.0712;s:4:"oids";d:0.1588;s:12:"ip-addresses";d:24.4418;s:10:"processors";d:12.2824;s:8:"mempools";d:12.1522;s:9:"inventory";d:18.2929;s:15:"printersupplies";d:0.0281;s:7:"sensors";d:6.9108;s:7:"storage";d:12.1076;s:10:"neighbours";d:12.1819;s:9:"arp-table";d:18.423;s:13:"junose-atm-vp";d:0.0124;s:9:"bgp-peers";d:6.0425;s:14:"mac-accounting";d:0.0074;s:3:"sla";d:6.0576;s:11:"pseudowires";d:6.1172;s:11:"cisco-cbqos";d:6.0897;s:10:"ucd-diskio";d:6.0609;s:10:"p2p-radios";d:0.0001;s:6:"graphs";d:0.0205;}} [distro] => [distro_ver] => [kernel] => [arch] => [attribs] => Array ( )
)
SQL[SELECT * FROM `observium_processes` WHERE `process_name` = 'poller.php' AND `device_id` = '1052' AND `poller_id` = '0'] SQL RUNTIME[0.00021911s]
SQL[DELETE FROM `observium_processes` WHERE `process_id` = '1215133'] SQL RUNTIME[0.00111103s]
SQL[INSERT INTO `observium_processes` (`process_pid`,`process_name`,`process_ppid`,`process_uid`,`process_command`,`process_start`,`device_id`) VALUES ('1574','poller.php','14950','1000','php ./poller.php -h upssrv2.komunala.local -d','1749812483','1052')] SQL RUNTIME[0.00731087s]
SQL[SELECT * FROM `alert_tests`] SQL RUNTIME[0.00016594s] Cached 0 alert rules.
SQL[SELECT * FROM `alert_table` WHERE `device_id` = '1052'] SQL RUNTIME[0.00018501s] ##### upssrv2.komunala.local [1052] #####
o OS generic o OS Group unix o Last poll duration 3.06 seconds o Last Polled 2025-06-12 16:27:21 o SNMP Version v2c Host 'upssrv2.komunala.local' found in /etc/hosts: 213.250.36.26
CMD[/usr/bin/fping -t 500 -c 1 -q 213.250.36.26 2>&1]
CMD EXITCODE[0] CMD RUNTIME[0.0117s] STDOUT[ 213.250.36.26 : xmt/rcv/%loss = 1/1/0%, min/avg/max = 9.11/9.11/9.11 ] Return cached 'upssrv2.komunala.local' = '213.250.36.26'
CMD[/usr/bin/snmpget -t '15' -r '5' -v2c -c *** -Pud -OQUsn -m SNMPv2-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'213.250.36.26':'10181' .1.3.6.1.2.1.1.2.0 .1.3.6.1.2.1.1.3.0]
CMD EXITCODE[1] CMD RUNTIME[90.0902s] STDOUT[
] STDERR[ Timeout: No Response from udp:213.250.36.26:10181. ] SNMP STATUS[FALSE] o Device status Device is not responding to SNMP requests Return cached 'upssrv2.komunala.local' = '213.250.36.26'
SQL[UPDATE `devices` SET `status_type` ='snmp' WHERE device_id = '1052'] SQL RUNTIME[0.00130391s]
SQL[SELECT * FROM `devices` WHERE `devices`.`device_id` = '1052'] SQL RUNTIME[0.00032711s]
SQL[INSERT INTO `eventlog` (`device_id`,`entity_id`,`entity_type`,`timestamp`,`severity`,`message`) VALUES ('1052','1052','device',NOW(),'3','Device status changed to Down (ping -> snmp)')] SQL RUNTIME[0.00673699s] RRD /opt/observium/rrd/upssrv2.komunala.local/status.rrd already exists - no need to create.
RRD CMD[update /opt/observium/rrd/upssrv2.komunala.local/status.rrd N:0] RRD RUNTIME[0.0011s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/upssrv2.komunala.local/status.rrd': Permission denied] RRD_STATUS[FALSE]
RRD /opt/observium/rrd/upssrv2.komunala.local/ping.rrd already exists - no need to create.
RRD CMD[update /opt/observium/rrd/upssrv2.komunala.local/ping.rrd N:9.11] RRD RUNTIME[0.0011s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/upssrv2.komunala.local/ping.rrd': Permission denied] RRD_STATUS[FALSE]
RRD /opt/observium/rrd/upssrv2.komunala.local/ping_snmp.rrd already exists - no need to create.
RRD CMD[update /opt/observium/rrd/upssrv2.komunala.local/ping_snmp.rrd N:U] RRD RUNTIME[0.0011s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/upssrv2.komunala.local/ping_snmp.rrd': Permission denied] RRD_STATUS[FALSE]
o Poller time 90.1356 seconds RRD /opt/observium/rrd/upssrv2.komunala.local/perf-poller.rrd already exists - no need to create.
RRD CMD[update /opt/observium/rrd/upssrv2.komunala.local/perf-poller.rrd N:90.1356] RRD RUNTIME[0.0011s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/upssrv2.komunala.local/perf-poller.rrd': Permission denied] RRD_STATUS[FALSE]
SQL[DELETE FROM `observium_processes` WHERE `process_pid` = '1574' AND `process_name` = 'poller.php' AND `device_id` = '1052' AND `poller_id` = '0'] SQL RUNTIME[0.00112414s] poller.php: upssrv2.komunala.local - 1 devices polled in 90.2475 secs ##### Completed polling run at 2025-06-13 13:02:53 #####
NOTE, $config['snmp']['hide_auth'] is set to TRUE, snmp community and snmp v3 auth hidden from debug output. o Devices Polled 1 o Poller Time 90.2475 secs o Definitions 0.056318998336792 secs o Memory usage 26MB (peak: 27.11MB) o MySQL Usage Insert[2/0.015s] Update[1/0.003s] Delete[2/0.004s] Cell[4/0.004s] Row[2/0.002s] Rows[7/0.085s] Col[1/0.001s] Connect[1/0s]Func[2/0.018s] (0.132s 0.146%) o RRDTool Usage update[4/0.004s] (0.004s 0.005%) o SNMP Usage errors[/0s] snmpget[1/90.09s] (90.09s 99.826%) Log file '/opt/observium/observium.log' is not writeable, check file permissions.
Hi,
You should probably first check the firewalls. They might be blocking SNMP/UDP between the Observium host and the end devices.
You should always test using snmpget on the commandline, rather than using the Observium poller.
snmpget -v2c -c <community> <device_ip> sysDescr.0
Thanks, adam.
zank--- via observium wrote on 13/06/2025 18:42:
Hello, we're trying to implement observium and it is about 400 devices. At the moment there are 226 but 2 of them suddenly went offline and cannot be brought back up with polling and I'm not sure if the snmp gets blocked on out end or on the devices end. There are also Palo Alto firewalls present at the location and I'm thinking they're causing the issue but just in case im posting this here also. Here is the ./poller with debug if it helps: DEBUG! DEFINITIONS Time : 0.056319 ms
DEFINITIONS Memory: 18.6MB
SQL[SELECT * FROM `observium_attribs`] SQL RUNTIME[0.00029492s]
SQL[SELECT @@SESSION.sql_mode;] SQL RUNTIME[0.00012803s]
SQL[SET SESSION `sql_mode` = 'NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'] SQL RUNTIME[6.7E-5s] DB mode(s) disabled: STRICT_TRANS_TABLES, ONLY_FULL_GROUP_BY, NO_ZERO_DATE, NO_ZERO_IN_DATE, ERROR_FOR_DIVISION_BY_ZERO
SQL[SELECT TIMEDIFF(NOW(), UTC_TIMESTAMP);] SQL RUNTIME[0.000103s]
SQL[SELECT * FROM `config`] SQL RUNTIME[0.0001359s]
CMD[/bin/hostname -f]
CMD EXITCODE[0] CMD RUNTIME[0.0056s] CMD EXITDELAY[5ms] STDOUT[ observium ]
/ _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___ | | | || '_ \ / __| / _ | '__|\ \ / /| || | | || '_ ` _ \ | |_| || |_) |__ | __/| | \ V / | || |_| || | | | | | ___/ |_.__/ |___/ ___||_| _/ |_| __,_||_| |_| |_| Observium Community Edition 24.12.13800 https://www.observium.org
CMD[/opt/observium/scripts/distro]
CMD EXITCODE[0] CMD RUNTIME[0.0125s] STDOUT[ Linux|4.15.0-213-generic|amd64|Ubuntu|18.04|vmware| ]
CMD[/usr/bin/env python --version 2>&1]
CMD EXITCODE[0] CMD RUNTIME[0.0028s] STDOUT[ Python 2.7.17 ]
CMD[/usr/bin/env python3 --version 2>&1]
CMD EXITCODE[0] CMD RUNTIME[0.0025s] STDOUT[ Python 3.6.9 ]
SQL[SELECT VERSION();] SQL RUNTIME[0.00018501s]
CMD[/usr/bin/snmpget --version 2>&1]
CMD EXITCODE[0] CMD RUNTIME[0.0024s] STDOUT[ NET-SNMP version: 5.7.3 ]
CMD[/usr/bin/rrdtool --version | head -n1]
CMD EXITCODE[0] CMD RUNTIME[0.0075s] STDOUT[ RRDtool 1.7.0 Copyright by Tobias Oetiker tobi@oetiker.ch ]
CMD[/usr/bin/fping -v 2>&1]
CMD EXITCODE[0] CMD RUNTIME[0.0022s] STDOUT[ /usr/bin/fping: Version 4.0 /usr/bin/fping: comments to david@schweikert.ch ]
CMD[/usr/sbin/apache2 -v | awk '/Server version:/ {print $3}']
CMD EXITCODE[0] CMD RUNTIME[0.044s] STDOUT[ Apache/2.4.29 ]
SQL[SELECT @@SESSION.sql_mode;] SQL RUNTIME[0.00016904s]
SQL[SHOW VARIABLES LIKE 'log_bin'] SQL RUNTIME[0.01490617s]
SQL[SHOW VARIABLES LIKE 'character_set_connection'] SQL RUNTIME[0.00202799s]
##### Observium #####
o Version 24.12.13800 o Released 24th December 2024
##### Software versions #####
o OS Linux 4.15.0-213-generic [amd64] (Ubuntu 18.04) o Apache 2.4.29 o PHP 7.2.24-0ubuntu0.18.04.17 (OPcache: ENABLED) o Python 3.6.9 (python3 not used as default python) o MySQL 5.7.42-0ubuntu0.18.04.1 (extension: mysqli 5.0.12-dev) o SNMP NET-SNMP 5.7.3 o RRDtool 1.7.0 o Fping 4.0 (IPv4 and IPv6) o Fetch cURL 7.58.0 (OpenSSL/1.1.1, LibZ 1.2.11)
##### Memory Limit #####
o PHP Unlimited
##### DB info #####
o DB schema 495 o MySQL binlog OFF o MySQL mode NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Friday, 13-Jun-25 13:01:23 CEST o PHP +02:00 o MySQL +02:00
SQL[SELECT * FROM `alerts_maint` WHERE `maint_start` < '1749812483' AND `maint_end` > '1749812483'] SQL RUNTIME[0.00025201s] ##### Starting polling run at 2025-06-13 13:01:23 #####
SQL[SELECT `device_id` FROM `devices` WHERE `disabled` = 0 AND `hostname` LIKE 'upssrv2.komunala.local' AND `poller_id` = '0' ORDER BY `device_id` ASC] SQL RUNTIME[0.0003252s]
SQL[SELECT * FROM `devices` WHERE `device_id` = '1052'] SQL RUNTIME[0.00029206s]
SQL[SELECT * FROM `entity_attribs` WHERE `entity_type` = 'device' AND `entity_id` = '1052'] SQL RUNTIME[0.00020504s] Array ( [device_id] => 1052 [poller_id] => 0 [hostname] => upssrv2.komunala.local [sysName] => upssrv2.komunala.local [label] => [ip] => 213.250.36.26 [snmp_community] => Public95Hidden [snmp_authlevel] => [snmp_authname] => [snmp_authpass] => [snmp_authalgo] => [snmp_cryptopass] => [snmp_cryptoalgo] => [snmp_context] => [snmpable] => [snmp_version] => v2c [snmp_port] => 10181 [snmp_timeout] => 15 [snmp_retries] => 5 [snmp_maxrep] => 12 [ssh_port] => 22 [agent_version] => [snmp_transport] => udp [bgpLocalAs] => [snmpEngineID] => 800000000300C0B7C71A77 [sysObjectID] => [sysDescr] => APC Web/SNMP Management Card (MB:v3.8.6 PF:v5.0.3 PN:apc_hw05_aos_503.bin AF1:v5.0.3 AN1:apc_hw05_sumx_503.bin MN:AP9630 HR:05 SN: ZA1023016503 MD:06/04/2010) (Embedded PowerNet SNMP Agent SW v2.2 compatible) [sysContact] => sos@minicom.si [version] => 5.0.3 [hardware] => Smart-UPS 3000 UPS 09.3 / MCU 19.0 / UBL 08.5 / MBL 19.0 [vendor] => APC [features] => Smart-UPS DSP [location] => Kopaliska Ul., Murska Sobota, SI [os] => generic [status] => 0 [status_type] => ping [ignore] => 0 [ignore_until] => [asset_tag] => [disabled] => 0 [uptime] => 13256450 [last_rebooted] => 1736481732 [force_discovery] => 0 [last_polled] => 2025-06-12 16:27:21 [last_discovered] => 2025-06-13 08:26:30 [last_alerter] => 2025-06-13 12:45:59 [last_polled_timetaken] => 3.06 [last_discovered_timetaken] => 387.35 [purpose] => [type] => power [serial] => ZA1023016503 [icon] => [device_state] => a:5:{s:15:"poller_mod_perf";a:20:{s:6:"system";d:0.2138;s:2:"os";d:0.3433;s:7:"sensors";d:0.9053;s:6:"status";d:0.3725;s:7:"counter";d:0.0006;s:10:"processors";d:0.0006;s:8:"mempools";d:0.0003;s:7:"storage";d:0.0006;s:8:"netstats";d:0.532;s:7:"ucd-mib";d:0.0002;s:13:"ipSystemStats";d:0.1151;s:5:"ports";d:0.4809;s:15:"printersupplies";d:0.0007;s:10:"ucd-diskio";d:0.0005;s:4:"ospf";d:0.0002;s:3:"sla";d:0.0009;s:14:"mac-accounting";d:0.0013;s:15:"entity-physical";d:0.0005;s:9:"fdb-table";d:0.0033;s:6:"graphs";d:0.0004;}s:17:"poller_ports_perf";a:3:{s:6:"IF-MIB";d:0.2936229705810547;s:13:"EtherLike-MIB";d:0.07470893859863281;s:6:"IP-MIB";d:0.09396791458129883;}s:14:"poller_history";a:288:{i:1749738438;d:3.0618;i:1749738208;d:4.2238;i:1749738140;d:3.4869;i:1749737855;d:3.0784;i:1749737626;d:4.0923;i:1749737519;d:3.1383;i:1749737192;d:3.0538;i:1749737035;d:3.4784;i:1749736931;d:3.5268;i:1749736570;d:2.9656;i:1749736442;d:3.5027;i:1749736281;d:2.8993;i:1749736004;d:2.9603;i:1749735809;d:3.0529;i:1749735638;d:3.0026;i:1749735360;d:3.3181;i:1749735226;d:4.4443;i:1749734801;d:3.0956;i:1749734624;d:2.9595;i:1749734423;d:3.0414;i:1749734143;d:3.1799;i:1749734007;d:3.0043;i:1749733916;d:3.361;i:1749733595;d:3.2164;i:1749733385;d:3.2873;i:1749733296;d:3.0735;i:1749732966;d:4.0385;i:1749732756;d:3.0887;i:1749732714;d:4.0922;i:1749732378;d:4.7093;i:1749732222;d:3.0779;i:1749732131;d:3.3076;i:1749731835;d:3.193;i:1749731733;d:3.642;i:1749731502;d:3.2965;i:1749731170;d:3.295;i:1749731060;d:4.6041;i:1749730932;d:3.3409;i:1749730635;d:3.3308;i:1749730393;d:6.0173;i:1749730049;d:3.5515;i:1749729843;d:4.8746;i:1749729810;d:3.4133;i:1749729456;d:3.4072;i:1749729163;d:3.3064;i:1749729129;d:3.3968;i:1749728947;d:5.2722;i:1749728634;d:2.9416;i:1749728467;d:3.1248;i:1749728282;d:3.0656;i:1749728011;d:4.1433;i:1749727901;d:4.0402;i:1749727613;d:3.2587;i:1749727400;d:4.3347;i:1749727320;d:3.6798;i:1749726995;d:4.1341;i:1749726804;d:3.2313;i:1749726718;d:4.3125;i:1749726441;d:3.121;i:1749726304;d:4.0524;i:1749726249;d:3.652;i:1749725828;d:3.4258;i:1749725636;d:3.0245;i:1749725538;d:3.6954;i:1749725216;d:3.167;i:1749725025;d:4.1817;i:1749724946;d:3.0821;i:1749724605;d:3.1564;i:1749724440;d:4.4093;i:1749724041;d:3.2009;i:1749723769;d:3.3049;i:1749723763;d:3.2371;i:1749723437;d:4.7006;i:1749723173;d:4.5375;i:1749722603;d:4.8981;i:1749722599;d:5.6263;i:1749722024;d:5.1299;i:1749721720;d:3.1521;i:1749721418;d:3.3172;i:1749720829;d:4.2989;i:1749720692;d:5.8752;i:1749720341;d:3.233;i:1749720207;d:3.4445;i:1749720085;d:3.1147;i:1749719790;d:3.1086;i:1749719609;d:3.075;i:1749719493;d:2.8119;i:1749719036;d:4.6502;i:1749718894;d:3.0848;i:1749718640;d:2.9379;i:1749718416;d:3.2494;i:1749718343;d:3.4622;i:1749718006;d:3.0574;i:1749717811;d:3.1954;i:1749717729;d:3.1686;i:1749717407;d:3.0071;i:1749717280;d:3.5784;i:1749717187;d:3.513;i:1749716812;d:3.0417;i:1749716581;d:2.983;i:1749716523;d:3.2777;i:1749716222;d:4.5187;i:1749715957;d:3.5873;i:1749715884;d:3.0163;i:1749715596;d:4.3571;i:1749715429;d:3.1659;i:1749715310;d:4.928;i:1749715008;d:3.0166;i:1749714794;d:3.7413;i:1749714740;d:4.2167;i:1749714394;d:3.0739;i:1749714197;d:3.105;i:1749713599;d:2.8258;i:1749713475;d:3.0262;i:1749713202;d:3.0628;i:1749713010;d:3.0971;i:1749712893;d:3.0806;i:1749712606;d:3.1635;i:1749712287;d:3.2386;i:1749712001;d:3.4897;i:1749711661;d:3.1639;i:1749711393;d:2.9135;i:1749711073;d:3.0417;i:1749710130;d:4.1916;i:1749709570;d:3.1502;i:1749709342;d:3.0585;i:1749709261;d:2.94;i:1749708181;d:5.0987;i:1749707811;d:4.0774;i:1749707573;d:3.3582;i:1749707200;d:4.116;i:1749706882;d:4.8638;i:1749706865;d:3.4298;i:1749706791;d:5.9448;i:1749706778;d:3.743;i:1749705947;d:3.0158;i:1749705814;d:2.9233;i:1749705697;d:3.0277;i:1749705445;d:3.1456;i:1749705262;d:3.1111;i:1749705136;d:3.1943;i:1749704824;d:3.2105;i:1749704709;d:4.3898;i:1749704571;d:3.1006;i:1749704184;d:3.074;i:1749703963;d:3.1956;i:1749703944;d:3.4926;i:1749703594;d:4.093;i:1749703434;d:3.1553;i:1749703295;d:3.0769;i:1749702978;d:2.9163;i:1749702802;d:2.9788;i:1749702695;d:3.247;i:1749702432;d:3.2204;i:1749702418;d:3.0743;i:1749702231;d:2.9114;i:1749702119;d:2.9206;i:1749702039;d:7.9277;i:1749702037;d:7.3172;i:1749701918;d:2.9738;i:1749701589;d:3.9971;i:1749701194;d:3.5617;i:1749701041;d:2.9469;i:1749700846;d:3.1473;i:1749700586;d:3.0539;i:1749700402;d:2.8453;i:1749700231;d:3.0317;i:1749699987;d:3.0497;i:1749699685;d:2.7944;i:1749699394;d:3.0414;i:1749699217;d:3.0233;i:1749698812;d:3.0028;i:1749698452;d:2.7561;i:1749698210;d:3.0167;i:1749696335;d:2.9227;i:1749695252;d:2.95;i:1749694583;d:3.1199;i:1749693985;d:2.8845;i:1749693624;d:4.8865;i:1749693356;d:2.98;i:1749693106;d:2.8904;i:1749692820;d:2.8787;i:1749692585;d:3.0068;i:1749692456;d:2.9868;i:1749692191;d:3.028;i:1749691993;d:3.0677;i:1749691902;d:3.5402;i:1749691576;d:3.0194;i:1749691394;d:2.9704;i:1749691296;d:3.179;i:1749691008;d:2.9235;i:1749690785;d:2.9993;i:1749690705;d:3.389;i:1749690411;d:2.9714;i:1749690214;d:3.0422;i:1749690088;d:3.0665;i:1749689805;d:2.8724;i:1749689441;d:2.9037;i:1749689177;d:2.9579;i:1749689001;d:2.8615;i:1749688892;d:2.9507;i:1749688602;d:3.9019;i:1749688408;d:2.8358;i:1749687969;d:2.8371;i:1749687777;d:2.9357;i:1749687387;d:3.0137;i:1749687154;d:2.8007;i:1749687054;d:3.4541;i:1749686782;d:3.1283;i:1749686524;d:3.3633;i:1749686229;d:2.9402;i:1749685620;d:2.9339;i:1749685284;d:3.0794;i:1749685231;d:5.6467;i:1749685035;d:3.1549;i:1749681394;d:3.0682;i:1749681218;d:3.6287;i:1749681121;d:3.2408;i:1749680916;d:2.8448;i:1749680601;d:4.0062;i:1749680498;d:3.1405;i:1749680194;d:3.0757;i:1749679991;d:2.9302;i:1749679915;d:3.1371;i:1749679592;d:3.3427;i:1749679451;d:3.2951;i:1749679318;d:2.9649;i:1749678998;d:2.921;i:1749678821;d:2.7459;i:1749678689;d:2.9294;i:1749678392;d:2.8998;i:1749678203;d:2.879;i:1749678055;d:3.0618;i:1749677779;d:2.97;i:1749677619;d:2.9318;i:1749677430;d:2.9083;i:1749677222;d:3.1604;i:1749676980;d:3.0918;i:1749676849;d:2.9225;i:1749676605;d:2.9696;i:1749676386;d:2.771;i:1749676297;d:3.045;i:1749675981;d:3.4345;i:1749675859;d:3.1831;i:1749675395;d:2.9036;i:1749674790;d:3.0451;i:1749674622;d:4.0518;i:1749674518;d:2.8897;i:1749674193;d:2.8084;i:1749673848;d:3.0124;i:1749673619;d:2.925;i:1749672919;d:2.8596;i:1749672655;d:2.8645;i:1749672324;d:5.3573;i:1749669225;d:2.9646;i:1749669097;d:2.9559;i:1749668804;d:2.8976;i:1749668705;d:2.9129;i:1749668476;d:3.0896;i:1749668183;d:2.9723;i:1749667977;d:2.9534;i:1749667895;d:3.1282;i:1749667604;d:3.8656;i:1749667436;d:2.9998;i:1749667340;d:3.5311;i:1749667011;d:2.9199;i:1749666790;d:2.742;i:1749666679;d:3.0625;i:1749666390;d:3.1936;i:1749666295;d:3.1553;i:1749666119;d:3.3232;i:1749665803;d:4.0631;i:1749665551;d:3.2739;i:1749665519;d:3.0565;i:1749665236;d:4.921;i:1749665079;d:2.9497;i:1749664906;d:3.1913;i:1749664601;d:3.059;i:1749664458;d:3.079;i:1749664363;d:3.177;i:1749664063;d:3.0919;i:1749663851;d:3.6324;i:1749663734;d:3.1656;i:1749663368;d:3.0413;i:1749663226;d:2.9456;i:1749663130;d:3.2162;i:1749662800;d:3.0637;i:1749662643;d:3.0931;}s:17:"discovery_history";a:100:{i:1749795603;d:387.3494;i:1749725319;d:12.3666;i:1749725218;d:12.6031;i:1749703571;d:11.9745;i:1749681701;d:12.1248;i:1749660292;d:12.3958;i:1749660264;d:14.2735;i:1749638887;d:12.9033;i:1749574050;d:11.9178;i:1749573969;d:11.9894;i:1749552575;d:12.0187;i:1749552439;d:12.2768;i:1749530690;d:11.9339;i:1749509518;d:12.8574;i:1749509460;d:12.7049;i:1749487391;d:17.9025;i:1749487314;d:12.6653;i:1749466076;d:12.5715;i:1749466055;d:12.7853;i:1749444091;d:11.8692;i:1749444018;d:11.996;i:1749422433;d:11.7268;i:1749401116;d:11.8017;i:1749379241;d:11.5649;i:1749357629;d:11.5663;i:1749335988;d:11.3901;i:1749314718;d:12.4409;i:1749314675;d:11.2592;i:1749293096;d:11.7146;i:1749293083;d:11.396;i:1749271307;d:11.3211;i:1749249862;d:12.4283;i:1749228221;d:11.6094;i:1749228188;d:12.6402;i:1749206534;d:11.7667;i:1749206521;d:11.5338;i:1749184914;d:11.684;i:1749184895;d:11.7512;i:1749163523;d:11.6353;i:1749163499;d:11.6857;i:1749141874;d:11.3587;i:1749141843;d:11.4611;i:1749120278;d:11.8961;i:1749098359;d:11.719;i:1749076996;d:11.3718;i:1749055373;d:11.5598;i:1749055360;d:11.5001;i:1749033678;d:13.8816;i:1749028573;d:11.6227;i:1749012340;d:11.4993;i:1749012312;d:11.1807;i:1748990486;d:11.5018;i:1748969039;d:11.7778;i:1748969002;d:11.5313;i:1748947453;d:11.1967;i:1748947378;d:12.2244;i:1748861222;d:12.5562;i:1748861130;d:11.8993;i:1748839619;d:12.0593;i:1748839566;d:11.6181;i:1748817808;d:11.6571;i:1748796250;d:11.5268;i:1748796179;d:11.6209;i:1748774850;d:11.6775;i:1748774793;d:11.4434;i:1748753154;d:11.7982;i:1748753122;d:11.9909;i:1748731303;d:11.6366;i:1748710004;d:11.9226;i:1748709949;d:11.6838;i:1748688188;d:11.8362;i:1748666710;d:11.9111;i:1748666639;d:11.5911;i:1748645082;d:11.7618;i:1748623649;d:11.5698;i:1748623632;d:12.6731;i:1748602034;d:13.3614;i:1748601963;d:12.2858;i:1748580312;d:12.1121;i:1748580229;d:12.2093;i:1748558805;d:12.0051;i:1748558786;d:11.9119;i:1748536953;d:13.0996;i:1748536875;d:12.1898;i:1748515292;d:12.4839;i:1748493937;d:11.836;i:1748472382;d:11.481;i:1748472353;d:12.0042;i:1748450781;d:11.4484;i:1748450761;d:11.5744;i:1748428936;d:11.6852;i:1748428923;d:11.7349;i:1748407534;d:11.4526;i:1748385739;d:12.4064;i:1748364369;d:11.8435;i:1748364348;d:11.7349;i:1748342693;d:12.1723;i:1748321155;d:11.5715;i:1748321139;d:11.433;i:1748299389;d:11.5605;}s:18:"discovery_mod_perf";a:26:{s:5:"vlans";d:6.0366;s:11:"ports_vlans";N;s:2:"os";d:0.0084;s:4:"mibs";d:111.9102;s:3:"vrf";d:0.0148;s:5:"ports";d:30.218;s:11:"ports-stack";d:6.0712;s:4:"oids";d:0.1588;s:12:"ip-addresses";d:24.4418;s:10:"processors";d:12.2824;s:8:"mempools";d:12.1522;s:9:"inventory";d:18.2929;s:15:"printersupplies";d:0.0281;s:7:"sensors";d:6.9108;s:7:"storage";d:12.1076;s:10:"neighbours";d:12.1819;s:9:"arp-table";d:18.423;s:13:"junose-atm-vp";d:0.0124;s:9:"bgp-peers";d:6.0425;s:14:"mac-accounting";d:0.0074;s:3:"sla";d:6.0576;s:11:"pseudowires";d:6.1172;s:11:"cisco-cbqos";d:6.0897;s:10:"ucd-diskio";d:6.0609;s:10:"p2p-radios";d:0.0001;s:6:"graphs";d:0.0205;}} [distro] => [distro_ver] => [kernel] => [arch] => [attribs] => Array ( )
)
SQL[SELECT * FROM `observium_processes` WHERE `process_name` = 'poller.php' AND `device_id` = '1052' AND `poller_id` = '0'] SQL RUNTIME[0.00021911s]
SQL[DELETE FROM `observium_processes` WHERE `process_id` = '1215133'] SQL RUNTIME[0.00111103s]
SQL[INSERT INTO `observium_processes` (`process_pid`,`process_name`,`process_ppid`,`process_uid`,`process_command`,`process_start`,`device_id`) VALUES ('1574','poller.php','14950','1000','php ./poller.php -h upssrv2.komunala.local -d','1749812483','1052')] SQL RUNTIME[0.00731087s]
SQL[SELECT * FROM `alert_tests`] SQL RUNTIME[0.00016594s] Cached 0 alert rules.
SQL[SELECT * FROM `alert_table` WHERE `device_id` = '1052'] SQL RUNTIME[0.00018501s] ##### upssrv2.komunala.local [1052] #####
o OS generic o OS Group unix o Last poll duration 3.06 seconds o Last Polled 2025-06-12 16:27:21 o SNMP Version v2c Host 'upssrv2.komunala.local' found in /etc/hosts: 213.250.36.26
CMD[/usr/bin/fping -t 500 -c 1 -q 213.250.36.26 2>&1]
CMD EXITCODE[0] CMD RUNTIME[0.0117s] STDOUT[ 213.250.36.26 : xmt/rcv/%loss = 1/1/0%, min/avg/max = 9.11/9.11/9.11 ] Return cached 'upssrv2.komunala.local' = '213.250.36.26'
CMD[/usr/bin/snmpget -t '15' -r '5' -v2c -c *** -Pud -OQUsn -m SNMPv2-MIB -M /opt/observium/mibs/rfc:/opt/observium/mibs/net-snmp 'udp':'213.250.36.26':'10181' .1.3.6.1.2.1.1.2.0 .1.3.6.1.2.1.1.3.0]
CMD EXITCODE[1] CMD RUNTIME[90.0902s] STDOUT[
] STDERR[ Timeout: No Response from udp:213.250.36.26:10181. ] SNMP STATUS[FALSE] o Device status Device is not responding to SNMP requests Return cached 'upssrv2.komunala.local' = '213.250.36.26'
SQL[UPDATE `devices` SET `status_type` ='snmp' WHERE device_id = '1052'] SQL RUNTIME[0.00130391s]
SQL[SELECT * FROM `devices` WHERE `devices`.`device_id` = '1052'] SQL RUNTIME[0.00032711s]
SQL[INSERT INTO `eventlog` (`device_id`,`entity_id`,`entity_type`,`timestamp`,`severity`,`message`) VALUES ('1052','1052','device',NOW(),'3','Device status changed to Down (ping -> snmp)')] SQL RUNTIME[0.00673699s] RRD /opt/observium/rrd/upssrv2.komunala.local/status.rrd already exists - no need to create.
RRD CMD[update /opt/observium/rrd/upssrv2.komunala.local/status.rrd N:0] RRD RUNTIME[0.0011s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/upssrv2.komunala.local/status.rrd': Permission denied] RRD_STATUS[FALSE]
RRD /opt/observium/rrd/upssrv2.komunala.local/ping.rrd already exists - no need to create.
RRD CMD[update /opt/observium/rrd/upssrv2.komunala.local/ping.rrd N:9.11] RRD RUNTIME[0.0011s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/upssrv2.komunala.local/ping.rrd': Permission denied] RRD_STATUS[FALSE]
RRD /opt/observium/rrd/upssrv2.komunala.local/ping_snmp.rrd already exists - no need to create.
RRD CMD[update /opt/observium/rrd/upssrv2.komunala.local/ping_snmp.rrd N:U] RRD RUNTIME[0.0011s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/upssrv2.komunala.local/ping_snmp.rrd': Permission denied] RRD_STATUS[FALSE]
o Poller time 90.1356 seconds RRD /opt/observium/rrd/upssrv2.komunala.local/perf-poller.rrd already exists - no need to create.
RRD CMD[update /opt/observium/rrd/upssrv2.komunala.local/perf-poller.rrd N:90.1356] RRD RUNTIME[0.0011s] RRD STDOUT[ERROR: opening '/opt/observium/rrd/upssrv2.komunala.local/perf-poller.rrd': Permission denied] RRD_STATUS[FALSE]
SQL[DELETE FROM `observium_processes` WHERE `process_pid` = '1574' AND `process_name` = 'poller.php' AND `device_id` = '1052' AND `poller_id` = '0'] SQL RUNTIME[0.00112414s] poller.php: upssrv2.komunala.local - 1 devices polled in 90.2475 secs ##### Completed polling run at 2025-06-13 13:02:53 #####
NOTE, $config['snmp']['hide_auth'] is set to TRUE, snmp community and snmp v3 auth hidden from debug output. o Devices Polled 1 o Poller Time 90.2475 secs o Definitions 0.056318998336792 secs o Memory usage 26MB (peak: 27.11MB) o MySQL Usage Insert[2/0.015s] Update[1/0.003s] Delete[2/0.004s] Cell[4/0.004s] Row[2/0.002s] Rows[7/0.085s] Col[1/0.001s] Connect[1/0s]Func[2/0.018s] (0.132s 0.146%) o RRDTool Usage update[4/0.004s] (0.004s 0.005%) o SNMP Usage errors[/0s] snmpget[1/90.09s] (90.09s 99.826%) Log file '/opt/observium/observium.log' is not writeable, check file permissions. _______________________________________________ observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
Hello, thank you for the reply. So for the firewalls it's kind of weird because we have multiple devices on that gateway that show up normal (UP) only this one UPS is shown as down and it can't do snmpget from the observium host, but when I try from a different virtual machine snmpget goes through. Is it possible the firewall is only blocking this one poll? I am verry sorry if it seem as a dumb question I am new to this and they just launched me into the deep end since at the start 180 out of 226 devices showed down, and I got it down to only showing one down
I usually look at the packets themselves to see where they are being blocked.
In our case, we use mostly cisco; so I'll set up an RSPAN on the far side of the firewall between it and the target and filter for SNMP from/to the server's IP. Doesn't matter what the content is, you just want to confirm that you see both a query packet and a reply packet. If you see both; then move the RSPAN collection point to somewhere between the firewall and the server and again confirm that you see both the query and reply (your firewall might only be blocking the replies). And, of course, you can run tcpdump right on your server (and potentially the remote device too).
Juniper's let you do a tcpdump right on the switch so it's quite a bit easier. Palo's will collect a dump file you can download to your desk. I don't have a lot of experience with other brands; but I'm sure they all let you collect a dump one way or another.
On 6/16/25 07:39, zank--- via observium wrote:
This email originated outside of WHOI. Please use caution if clicking on links or opening attachments.
Hello, thank you for the reply. So for the firewalls it's kind of weird because we have multiple devices on that gateway that show up normal (UP) only this one UPS is shown as down and it can't do snmpget from the observium host, but when I try from a different virtual machine snmpget goes through. Is it possible the firewall is only blocking this one poll? I am verry sorry if it seem as a dumb question I am new to this and they just launched me into the deep end since at the start 180 out of 226 devices showed down, and I got it down to only showing one down _______________________________________________ observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
Yes, the firewalls are probably blocking a specific source as part of some flood protection.
I’m not really familiar with these devices though, so I’m not sure what the option or rule will be called to turn it off.
Thanks, Adam.
Sent from my iPhone
On 16 Jun 2025, at 17:08, zank--- via observium observium@lists.observium.org wrote:
Hello, thank you for the reply. So for the firewalls it's kind of weird because we have multiple devices on that gateway that show up normal (UP) only this one UPS is shown as down and it can't do snmpget from the observium host, but when I try from a different virtual machine snmpget goes through. Is it possible the firewall is only blocking this one poll? I am verry sorry if it seem as a dumb question I am new to this and they just launched me into the deep end since at the start 180 out of 226 devices showed down, and I got it down to only showing one down _______________________________________________ observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
Hello again! I am very sorry for the late reply, I was on sick leave for a few days. So here is the update: The snmpget doesn't get a response from the device and this is the output: snmpget -v2c -c <password> upssrv2.komunala.local 1.3.6.1.2.1.1.3.0 Timeout: No Response from upssrv2.komunala.local And after contacting my superior to check the firewall no trafic has been blocked. Should I just reinstall it (not sure if it will work since they already tried zabixx apparently after this first started happening and the result was the same) or is there another fix maybe? I also want to add that 1 device per week goes down until it reaches 195 devices, then it seems to be fine.
participants (3)
-
Adam Armstrong -
Eric Bates -
zank＠minicom.si