Hi,
I have a BGP alert checker / notification that doesn't seem to be working like it used to. The check applies to only about 140 peers. It's the standard test that the documentation provides. Enitity/Device match are * and the test is:
bgpPeerAdminStatus equals start bgpPeerState notequals established
I've noticed recently that I have only been getting some device notifications while the others don't even make it to the mail queue, however observium says that a notification was sent. I've tried deleting the alert checker and recreating it with the same results.
Created a fake BGP peer that is in Active mode right now and ran the debugger on it (also a debug on a current working peer)
*(no notification)*
$ ./test_alert.php -a 128226 -dd DEBUG! Load class 'Console_Color2' from '/opt/observium/libs/pear/Console/Color2.php': OK
SQL[SELECT `attrib_value` FROM `observium_attribs` WHERE `attrib_type` = 'dbSchema';] ROWS[1] SQL RUNTIME[0.00035691s]
SQL[SELECT @@SESSION.sql_mode;] ROWS[1] SQL RUNTIME[0.00018501s]
SQL[SELECT `attrib_value` FROM `observium_attribs` WHERE `attrib_type` = 'current_rev';] ROWS[1] SQL RUNTIME[0.00025105s]
SQL[SELECT * FROM `config`] ROWS[0] SQL RUNTIME[0.00022912s] Load class 'Net_IPv6' from '/opt/observium/libs/pear/Net/IPv6.php': OK Observium 0.16.10.8161 Test Alert Notification
CMD[/opt/observium/scripts/distro]
CMD EXITCODE[0] CMD RUNTIME[0.0168s] STDOUT[ Linux|3.10.0-327.36.1.el7.x86_64|amd64|CentOS|7.2.1511 ]
CMD[/usr/bin/env python --version 2>&1]
CMD EXITCODE[0] CMD RUNTIME[0.0047s] STDOUT[ Python 2.7.5 ]
SQL[SELECT version();] ROWS[1] SQL RUNTIME[0.00029111s]
CMD[/usr/bin/snmpget --version 2>&1]
CMD EXITCODE[0] CMD RUNTIME[0.0087s] STDOUT[ NET-SNMP version: 5.7.2 ]
CMD[/usr/bin/rrdtool --version | head -n1]
CMD EXITCODE[0] CMD RUNTIME[0.0086s] STDOUT[ RRDtool 1.4.8 Copyright 1997-2013 by Tobias Oetiker tobi@oetiker.ch ]
CMD[/usr/bin/rrdcached -h | head -n1]
CMD EXITCODE[0] CMD RUNTIME[0.0082s] STDOUT[ RRDCacheD 1.4.8 ]
CMD[/usr/bin/env apache2 -v | awk '/Server version:/ {print $3}']
CMD EXITCODE[0] CMD RUNTIME[0.004s] STDOUT[
]
CMD[/usr/bin/env httpd -v | awk '/Server version:/ {print $3}']
CMD EXITCODE[0] CMD RUNTIME[0.0346s] STDOUT[ Apache/2.4.6 ]
CMD[date "+%:z"]
CMD EXITCODE[0] CMD RUNTIME[0.0032s] STDOUT[ -07:00 ]
SQL[SELECT TIMEDIFF(NOW(), UTC_TIMESTAMP);] ROWS[1] SQL RUNTIME[0.00026107s]
SQL[SELECT @@SESSION.sql_mode;] ROWS[1] SQL RUNTIME[0.00019383s]
SQL[SHOW VARIABLES LIKE 'character_set_connection'] ROWS[1] SQL RUNTIME[0.00065804s]
##### Software versions #####
o OS Linux 3.10.0-327.36.1.el7.x86_64 [amd64] (CentOS 7.2.1511) o Apache 2.4.6 o PHP 5.4.16 o Python 2.7.5 o MySQL 5.5.50-MariaDB (extension: mysqli 5.5.50-MariaDB) o SNMP NET-SNMP 5.7.2 o RRDtool 1.4.8 (rrdcached 1.4.8: unix:/var/run/rrdcached/rrdcached.sock)
##### MySQL mode #####
o MySQL
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Thursday, 20-Oct-16 21:40:54 PDT o PHP -07:00 o MySQL -07:00
SQL[SELECT * FROM `alert_tests` WHERE 1] ROWS[32] SQL RUNTIME[0.00031996s] Cached 32 alert rules.
SQL[SELECT * FROM `alert_assoc`] ROWS[32] SQL RUNTIME[0.00021911s]
SQL[SELECT * FROM `alert_table` WHERE `alert_table_id` = '128226'] ROWS[1] SQL RUNTIME[0.00021100s]
SQL[SELECT * FROM `devices` WHERE `device_id` = '77'] ROWS[1] SQL RUNTIME[0.00027394s]
SQL[SELECT * FROM `device_graphs` WHERE `device_id` = '77'] ROWS[38] SQL RUNTIME[0.00034308s]
SQL[SELECT * FROM `bgpPeers` LEFT JOIN `bgpPeers-state` USING (`bgpPeer_id`) WHERE `bgpPeers`.`bgpPeer_id` = '3268'] ROWS[1] SQL RUNTIME[0.00026608s]
$graphtype - /opt/observium/includes/alerts.inc.php:1170
/opt/observium/html/includes/graphs/graph.inc.php:33 ======================================================== array( [0] => string(11) "bgp_updates" [type] => string(3) "bgp" [1] => string(3) "bgp" [subtype] => string(7) "updates" [2] => string(7) "updates" )
SQL[SELECT * FROM `bgpPeers` WHERE `bgpPeer_id` = '3268'] ROWS[1] SQL RUNTIME[0.00020003s] RRD PIPE OPEN[TRUE] RRD PIPE CLOSE[TRUE] RRD PIPE STDOUT[ OK u:0.03 s:0.01 r:0.03
]
RRD CMD[graph --daemon unix:/var/run/rrdcached/rrdcached.sock /tmp/1utnX5E8LqmthZw7.png -g -Y --lower-limit 0 --alt-autoscale-max --start 1476852054 --end 1477024854 --width 400 --height 150 -c BACK#EEEEEE00 -c SHADEA#EEEEEE00 -c SHADEB#EEEEEE00 -c FONT#000000 -c CANVAS#FFFFFF00 -c GRID#a5a5a5 -c MGRID#FF9999 -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal --font LEGEND:8:'DroidSansMono,DejaVuSansMono' --font AXIS:7:'DroidSansMono,DejaVuSansMono' --font-render-mode normal -E COMMENT:'Updates Last Avg Max \n' DEF:out=/opt/observium/rrd/asr1-sv2/bgp-100.65.0.49.rrd:bgpPeerOutUpdates:AVERAGE DEF:in=/opt/observium/rrd/asr1-sv2/bgp-100.65.0.49.rrd:bgpPeerInUpdates:AVERAGE DEF:out_max=/opt/observium/rrd/asr1-sv2/bgp-100.65.0.49.rrd:bgpPeerOutUpdates:MAX DEF:in_max=/opt/observium/rrd/asr1-sv2/bgp-100.65.0.49.rrd:bgpPeerInUpdates:MAX CDEF:dout_max=out_max,-1,* CDEF:dout=out,-1,* CDEF:both=in,out,+ CDEF:alloctets=out,in,+ CDEF:wrongin=alloctets,UN,INF,UNKN,IF CDEF:wrongout=wrongin,-1,* AREA:in#AA66AA90 LINE1.25:in#AA66AA:'In ' GPRINT:in:LAST:.2lf%s GPRINT:in:AVERAGE:.2lf%s GPRINT:in_max:MAX:.2lf%s COMMENT:'\n' AREA:dout#FF660090: LINE1.25:dout#FF6600:'Out ' GPRINT:out:LAST:.2lf%s GPRINT:out:AVERAGE:.2lf%s GPRINT:out_max:MAX:.2lf%s COMMENT:\n AREA:wrongin#FFF2F2 AREA:wrongout#FFF2F2 HRULE:0#999999] RRD RUNTIME[0.0418s] RRD STDOUT[481x189] RRD_STATUS[TRUE]
SQL[SELECT `attrib_value` FROM `entity_attribs` WHERE `entity_type` = 'device' AND `entity_id` = '77' AND `attrib_type` = 'disable_notify'] ROWS[1] SQL RUNTIME[0.00042892s]
*(one that works from the same check)*
./test_alert.php -a 128255 -dd DEBUG! Load class 'Console_Color2' from '/opt/observium/libs/pear/Console/Color2.php': OK
SQL[SELECT `attrib_value` FROM `observium_attribs` WHERE `attrib_type` = 'dbSchema';] ROWS[1] SQL RUNTIME[0.00032401s]
SQL[SELECT @@SESSION.sql_mode;] ROWS[1] SQL RUNTIME[0.00017214s]
SQL[SELECT `attrib_value` FROM `observium_attribs` WHERE `attrib_type` = 'current_rev';] ROWS[1] SQL RUNTIME[0.00025201s]
SQL[SELECT * FROM `config`] ROWS[0] SQL RUNTIME[0.00023389s] Load class 'Net_IPv6' from '/opt/observium/libs/pear/Net/IPv6.php': OK Observium 0.16.10.8161 Test Alert Notification
CMD[/opt/observium/scripts/distro]
CMD EXITCODE[0] CMD RUNTIME[0.0164s] STDOUT[ Linux|3.10.0-327.36.1.el7.x86_64|amd64|CentOS|7.2.1511 ]
CMD[/usr/bin/env python --version 2>&1]
CMD EXITCODE[0] CMD RUNTIME[0.0051s] STDOUT[ Python 2.7.5 ]
SQL[SELECT version();] ROWS[1] SQL RUNTIME[0.00030708s]
CMD[/usr/bin/snmpget --version 2>&1]
CMD EXITCODE[0] CMD RUNTIME[0.0081s] STDOUT[ NET-SNMP version: 5.7.2 ]
CMD[/usr/bin/rrdtool --version | head -n1]
CMD EXITCODE[0] CMD RUNTIME[0.0077s] STDOUT[ RRDtool 1.4.8 Copyright 1997-2013 by Tobias Oetiker tobi@oetiker.ch ]
CMD[/usr/bin/rrdcached -h | head -n1]
CMD EXITCODE[0] CMD RUNTIME[0.0074s] STDOUT[ RRDCacheD 1.4.8 ]
CMD[/usr/bin/env apache2 -v | awk '/Server version:/ {print $3}']
CMD EXITCODE[0] CMD RUNTIME[0.0042s] STDOUT[
]
CMD[/usr/bin/env httpd -v | awk '/Server version:/ {print $3}']
CMD EXITCODE[0] CMD RUNTIME[0.0364s] STDOUT[ Apache/2.4.6 ]
CMD[date "+%:z"]
Wrong process status! Issue in proc_get_status(), see: https://bugs.php.net/bug.php?id=69014 CMD EXITCODE[0] CMD RUNTIME[0.003s] STDOUT[ -07:00 ]
SQL[SELECT TIMEDIFF(NOW(), UTC_TIMESTAMP);] ROWS[1] SQL RUNTIME[0.00025392s]
SQL[SELECT @@SESSION.sql_mode;] ROWS[1] SQL RUNTIME[0.00019908s]
SQL[SHOW VARIABLES LIKE 'character_set_connection'] ROWS[1] SQL RUNTIME[0.00074387s]
##### Software versions #####
o OS Linux 3.10.0-327.36.1.el7.x86_64 [amd64] (CentOS 7.2.1511) o Apache 2.4.6 o PHP 5.4.16 o Python 2.7.5 o MySQL 5.5.50-MariaDB (extension: mysqli 5.5.50-MariaDB) o SNMP NET-SNMP 5.7.2 o RRDtool 1.4.8 (rrdcached 1.4.8: unix:/var/run/rrdcached/rrdcached.sock)
##### MySQL mode #####
o MySQL
##### Charset info #####
o PHP UTF-8 o MySQL utf8
##### Timezones info #####
o Date Thursday, 20-Oct-16 21:46:44 PDT o PHP -07:00 o MySQL -07:00
SQL[SELECT * FROM `alert_tests` WHERE 1] ROWS[32] SQL RUNTIME[0.00056601s] Cached 32 alert rules.
SQL[SELECT * FROM `alert_assoc`] ROWS[32] SQL RUNTIME[0.00039506s]
SQL[SELECT * FROM `alert_table` WHERE `alert_table_id` = '128255'] ROWS[1] SQL RUNTIME[0.00037503s]
SQL[SELECT * FROM `devices` WHERE `device_id` = '104'] ROWS[1] SQL RUNTIME[0.00044703s]
SQL[SELECT * FROM `device_graphs` WHERE `device_id` = '104'] ROWS[28] SQL RUNTIME[0.00048900s]
SQL[SELECT * FROM `bgpPeers` LEFT JOIN `bgpPeers-state` USING (`bgpPeer_id`) WHERE `bgpPeers`.`bgpPeer_id` = '509'] ROWS[1] SQL RUNTIME[0.00043297s]
$graphtype - /opt/observium/includes/alerts.inc.php:1170
/opt/observium/html/includes/graphs/graph.inc.php:33 ======================================================== array( [0] => string(11) "bgp_updates" [type] => string(3) "bgp" [1] => string(3) "bgp" [subtype] => string(7) "updates" [2] => string(7) "updates" )
SQL[SELECT * FROM `bgpPeers` WHERE `bgpPeer_id` = '509'] ROWS[1] SQL RUNTIME[0.00029802s] RRD PIPE OPEN[TRUE] RRD PIPE CLOSE[TRUE] RRD PIPE STDOUT[ OK u:0.03 s:0.01 r:0.03
]
RRD CMD[graph --daemon unix:/var/run/rrdcached/rrdcached.sock /tmp/5S0b4geqsVsq08iN.png -g -Y --lower-limit 0 --alt-autoscale-max --start 1476852404 --end 1477025204 --width 400 --height 150 -c BACK#EEEEEE00 -c SHADEA#EEEEEE00 -c SHADEB#EEEEEE00 -c FONT#000000 -c CANVAS#FFFFFF00 -c GRID#a5a5a5 -c MGRID#FF9999 -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal --font LEGEND:8:'DroidSansMono,DejaVuSansMono' --font AXIS:7:'DroidSansMono,DejaVuSansMono' --font-render-mode normal -E COMMENT:'Updates Last Avg Max \n' DEF:out=/opt/observium/rrd/us1inet2/bgp-x.x.x.x.rrd:bgpPeerOutUpdates:AVERAGE DEF:in=/opt/observium/rrd/us1inet2/bgp-x.x.x.x.rrd:bgpPeerInUpdates:AVERAGE DEF:out_max=/opt/observium/rrd/us1inet2/bgp-x.x.x.x.rrd:bgpPeerOutUpdates:MAX DEF:in_max=/opt/observium/rrd/us1inet2/bgp-x.x.x.x.rrd:bgpPeerInUpdates:MAX CDEF:dout_max=out_max,-1,* CDEF:dout=out,-1,* CDEF:both=in,out,+ CDEF:alloctets=out,in,+ CDEF:wrongin=alloctets,UN,INF,UNKN,IF CDEF:wrongout=wrongin,-1,* AREA:in#AA66AA90 LINE1.25:in#AA66AA:'In ' GPRINT:in:LAST:.2lf%s GPRINT:in:AVERAGE:.2lf%s GPRINT:in_max:MAX:.2lf%s COMMENT:'\n' AREA:dout#FF660090: LINE1.25:dout#FF6600:'Out ' GPRINT:out:LAST:.2lf%s GPRINT:out:AVERAGE:.2lf%s GPRINT:out_max:MAX:.2lf%s COMMENT:\n AREA:wrongin#FFF2F2 AREA:wrongout#FFF2F2 HRULE:0#999999] RRD RUNTIME[0.0418s] RRD STDOUT[481x189] RRD_STATUS[TRUE]
SQL[SELECT `attrib_value` FROM `entity_attribs` WHERE `entity_type` = 'device' AND `entity_id` = '104' AND `attrib_type` = 'disable_notify'] ROWS[0] SQL RUNTIME[0.00042915s]
SQL[SELECT * FROM `alert_contacts` WHERE `contact_disabled` = 0 AND `contact_id` IN (SELECT `contact_id` FROM `alert_contacts_assoc` WHERE `aca_type` = 'alert' AND `alert_checker_id` = '39');] ROWS[2] SQL RUNTIME[0.00054097s] o Notifying [hipchat] Networking Alert: {"room_id":"","from":"Observium","token":""}
REQUEST[https://api.hipchat.com/v2/room/xxxxxx/notification] REQUEST STATUS[204 No Content] REQUEST RUNTIME[0.5128s] RESPONSE[
]
$http_response_header - /opt/observium/includes/alerts.inc.php:1271
/opt/observium/includes/common.inc.php:3213 ========================================================= array( [0] => string(23) "HTTP/1.1 204 No Content" [1] => string(30) "Access-Control-Allow-Origin: *" [2] => string(119) "Access-Control-Expose-Headers: Date, ETag, Link, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-Backoff" [3] => string(23) "Content-Type: text/html" [4] => string(35) "Date: Fri, 21 Oct 2016 04:46:44 GMT" [5] => string(94) "Location: https://api.hipchat.com/v2/room/xxxxx/history/xxxxx-7c71-xxxx-xxxx-xxxxxxxxx... " [6] => string(13) "Server: nginx" [7] => string(43) "Strict-Transport-Security: max-age=31536000" [8] => string(31) "X-Content-Type-Options: nosniff" [9] => string(22) "X-Ratelimit-Limit: 100" [10] => string(26) "X-Ratelimit-Remaining: 100" [11] => string(29) "X-Ratelimit-Reset: 1477025505" [12] => string(31) "X-XSS-Protection: 1; mode=block" [13] => string(17) "Connection: Close" )
$opts - /opt/observium/includes/alerts.inc.php:1271
/opt/observium/includes/common.inc.php:3214 ================================================= array( [http] => array( [method] => string(4) "POST" [header] => string(174) "Connection: close Content-Type: application/json Content-Length: 275 Authorization: Bearer zomgWApDk5Wz9eK4kCReKjUDQxW1h9kMrRqDJnqy User-Agent: Observium/0.16.10.8161\r\n" [content] => string(275) "{"from":"Observium","color":"red","message_format":"html","message":"<a href="http://kansatsu.lol.com/device/device=104/tab=alert/alert_entry=128255/">ALERT: [us1inet2] [bgp_peer] [AS65028 x.x.x.x] </a><br />bgpPeerAdminStatus = start bgpPeerState = active"}" [timeout] => string(2) "15" ) ) o Notifying [email] Netops: {"email":"netops@lol.com"} Load class 'Mail_mime' from '/opt/observium/libs/pear/Mail/mime.php': OK Load class 'Mail' from '/opt/observium/libs/pear/Mail/Mail.php': OK PHP Strict Standards: Non-static method Mail::factory() should not be called statically in /opt/observium/includes/alerting/email.inc.php on line 176
Strict Standards: Non-static method Mail::factory() should not be called statically in /opt/observium/includes/alerting/email.inc.php on line 176 PHP Strict Standards: Non-static method PEAR::isError() should not be called statically in /opt/observium/includes/alerting/email.inc.php on line 180
Strict Standards: Non-static method PEAR::isError() should not be called statically in /opt/observium/includes/alerting/email.inc.php on line 180