That first alert has disable notifications set.

Look at the final SQL query it does before it decides not to bother.

:D

Adam.



Sent from BlueMail

On 21 Oct 2016, at 06:32, Pico Leto <picoleto420@gmail.com> wrote:
Hi,

I have a BGP alert checker / notification that doesn't seem to be working like it used to.  The check applies to only about 140 peers.  It's the standard test that the documentation provides.  Enitity/Device match are * and the test is:

bgpPeerAdminStatus equals start
bgpPeerState notequals established

I've noticed recently that I have only been getting some device notifications while the others don't even make it to the mail queue, however observium says that a notification was sent.  I've tried deleting the alert checker and recreating it with the same results.

Created a fake BGP peer that is in Active mode right now and ran the debugger on it (also a debug on a current working peer)

(no notification)

$ ./test_alert.php -a 128226 -dd
DEBUG!
Load class 'Console_Color2' from '/opt/observium/libs/pear/Console/Color2.php': OK

SQL[SELECT `attrib_value` FROM `observium_attribs` WHERE `attrib_type` = 'dbSchema';]
ROWS[1]
SQL RUNTIME[0.00035691s]

SQL[SELECT @@SESSION.sql_mode;]
ROWS[1]
SQL RUNTIME[0.00018501s]

SQL[SELECT `attrib_value` FROM `observium_attribs` WHERE `attrib_type` = 'current_rev';]
ROWS[1]
SQL RUNTIME[0.00025105s]

SQL[SELECT * FROM `config`]
ROWS[0]
SQL RUNTIME[0.00022912s]
Load class 'Net_IPv6' from '/opt/observium/libs/pear/Net/IPv6.php': OK
Observium 0.16.10.8161
Test Alert Notification


CMD[/opt/observium/scripts/distro]

CMD EXITCODE[0]
CMD RUNTIME[0.0168s]
STDOUT[
Linux|3.10.0-327.36.1.el7.x86_64|amd64|CentOS|7.2.1511
]

CMD[/usr/bin/env python --version 2>&1]

CMD EXITCODE[0]
CMD RUNTIME[0.0047s]
STDOUT[
Python 2.7.5
]

SQL[SELECT version();]
ROWS[1]
SQL RUNTIME[0.00029111s]

CMD[/usr/bin/snmpget --version 2>&1]

CMD EXITCODE[0]
CMD RUNTIME[0.0087s]
STDOUT[
NET-SNMP version: 5.7.2
]

CMD[/usr/bin/rrdtool --version | head -n1]

CMD EXITCODE[0]
CMD RUNTIME[0.0086s]
STDOUT[
RRDtool 1.4.8  Copyright 1997-2013 by Tobias Oetiker <tobi@oetiker.ch>
]

CMD[/usr/bin/rrdcached -h | head -n1]

CMD EXITCODE[0]
CMD RUNTIME[0.0082s]
STDOUT[
RRDCacheD 1.4.8
]

CMD[/usr/bin/env apache2 -v | awk '/Server version:/ {print $3}']

CMD EXITCODE[0]
CMD RUNTIME[0.004s]
STDOUT[

]

CMD[/usr/bin/env httpd -v | awk '/Server version:/ {print $3}']

CMD EXITCODE[0]
CMD RUNTIME[0.0346s]
STDOUT[
Apache/2.4.6
]

CMD[date "+%:z"]

CMD EXITCODE[0]
CMD RUNTIME[0.0032s]
STDOUT[
-07:00
]

SQL[SELECT TIMEDIFF(NOW(), UTC_TIMESTAMP);]
ROWS[1]
SQL RUNTIME[0.00026107s]

SQL[SELECT @@SESSION.sql_mode;]
ROWS[1]
SQL RUNTIME[0.00019383s]

SQL[SHOW VARIABLES LIKE 'character_set_connection']
ROWS[1]
SQL RUNTIME[0.00065804s]

#####  Software versions  #####

 o OS                   Linux 3.10.0-327.36.1.el7.x86_64 [amd64] (CentOS 7.2.1511)
 o Apache               2.4.6
 o PHP                  5.4.16
 o Python               2.7.5
 o MySQL                5.5.50-MariaDB (extension: mysqli 5.5.50-MariaDB)
 o SNMP                 NET-SNMP 5.7.2
 o RRDtool              1.4.8 (rrdcached 1.4.8: unix:/var/run/rrdcached/rrdcached.sock)

#####  MySQL mode  #####

 o MySQL

#####  Charset info  #####

 o PHP                  UTF-8
 o MySQL                utf8

#####  Timezones info  #####

 o Date                 Thursday, 20-Oct-16 21:40:54 PDT
 o PHP                  -07:00
 o MySQL                -07:00


SQL[SELECT * FROM `alert_tests` WHERE 1]
ROWS[32]
SQL RUNTIME[0.00031996s]
Cached 32 alert rules.

SQL[SELECT * FROM `alert_assoc`]
ROWS[32]
SQL RUNTIME[0.00021911s]

SQL[SELECT * FROM `alert_table` WHERE `alert_table_id` = '128226']
ROWS[1]
SQL RUNTIME[0.00021100s]

SQL[SELECT * FROM `devices` WHERE `device_id` = '77']
ROWS[1]
SQL RUNTIME[0.00027394s]

SQL[SELECT * FROM `device_graphs` WHERE `device_id` = '77']
ROWS[38]
SQL RUNTIME[0.00034308s]

SQL[SELECT * FROM `bgpPeers` LEFT JOIN `bgpPeers-state` USING (`bgpPeer_id`) WHERE `bgpPeers`.`bgpPeer_id` = '3268']
ROWS[1]
SQL RUNTIME[0.00026608s]


> $graphtype - /opt/observium/includes/alerts.inc.php:1170
/opt/observium/html/includes/graphs/graph.inc.php:33
========================================================
array(
  [0]       => string(11) "bgp_updates"
  [type]    => string(3) "bgp"
  [1]       => string(3) "bgp"
  [subtype] => string(7) "updates"
  [2]       => string(7) "updates"
)

SQL[SELECT * FROM `bgpPeers` WHERE `bgpPeer_id` = '3268']
ROWS[1]
SQL RUNTIME[0.00020003s]
RRD PIPE OPEN[TRUE]
RRD PIPE CLOSE[TRUE]
RRD PIPE STDOUT[
OK u:0.03 s:0.01 r:0.03

]

RRD CMD[graph --daemon unix:/var/run/rrdcached/rrdcached.sock /tmp/1utnX5E8LqmthZw7.png  -g -Y --lower-limit 0 --alt-autoscale-max  --start 1476852054 --end 1477024854 --width 400 --height 150 -c BACK#EEEEEE00 -c SHADEA#EEEEEE00 -c SHADEB#EEEEEE00 -c FONT#000000 -c CANVAS#FFFFFF00 -c GRID#a5a5a5 -c MGRID#FF9999 -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal --font LEGEND:8:'DroidSansMono,DejaVuSansMono' --font AXIS:7:'DroidSansMono,DejaVuSansMono' --font-render-mode normal -E COMMENT:'Updates           Last      Avg      Max \n' DEF:out=/opt/observium/rrd/asr1-sv2/bgp-100.65.0.49.rrd:bgpPeerOutUpdates:AVERAGE DEF:in=/opt/observium/rrd/asr1-sv2/bgp-100.65.0.49.rrd:bgpPeerInUpdates:AVERAGE DEF:out_max=/opt/observium/rrd/asr1-sv2/bgp-100.65.0.49.rrd:bgpPeerOutUpdates:MAX DEF:in_max=/opt/observium/rrd/asr1-sv2/bgp-100.65.0.49.rrd:bgpPeerInUpdates:MAX CDEF:dout_max=out_max,-1,* CDEF:dout=out,-1,* CDEF:both=in,out,+ CDEF:alloctets=out,in,+ CDEF:wrongin=alloctets,UN,INF,UNKN,IF CDEF:wrongout=wrongin,-1,* AREA:in#AA66AA90 LINE1.25:in#AA66AA:'In          ' GPRINT:in:LAST:.2lf%s GPRINT:in:AVERAGE:.2lf%s GPRINT:in_max:MAX:.2lf%s COMMENT:'\n' AREA:dout#FF660090: LINE1.25:dout#FF6600:'Out         ' GPRINT:out:LAST:.2lf%s GPRINT:out:AVERAGE:.2lf%s GPRINT:out_max:MAX:.2lf%s COMMENT:\n AREA:wrongin#FFF2F2 AREA:wrongout#FFF2F2 HRULE:0#999999]
RRD RUNTIME[0.0418s]
RRD STDOUT[481x189]
RRD_STATUS[TRUE]


SQL[SELECT `attrib_value` FROM `entity_attribs` WHERE `entity_type` = 'device' AND `entity_id` = '77' AND `attrib_type` = 'disable_notify']
ROWS[1]
SQL RUNTIME[0.00042892s]

(one that works from the same check)

 ./test_alert.php -a 128255 -dd
DEBUG!
Load class 'Console_Color2' from '/opt/observium/libs/pear/Console/Color2.php': OK

SQL[SELECT `attrib_value` FROM `observium_attribs` WHERE `attrib_type` = 'dbSchema';]
ROWS[1]
SQL RUNTIME[0.00032401s]

SQL[SELECT @@SESSION.sql_mode;]
ROWS[1]
SQL RUNTIME[0.00017214s]

SQL[SELECT `attrib_value` FROM `observium_attribs` WHERE `attrib_type` = 'current_rev';]
ROWS[1]
SQL RUNTIME[0.00025201s]

SQL[SELECT * FROM `config`]
ROWS[0]
SQL RUNTIME[0.00023389s]
Load class 'Net_IPv6' from '/opt/observium/libs/pear/Net/IPv6.php': OK
Observium 0.16.10.8161
Test Alert Notification

CMD[/opt/observium/scripts/distro]

CMD EXITCODE[0]
CMD RUNTIME[0.0164s]
STDOUT[
Linux|3.10.0-327.36.1.el7.x86_64|amd64|CentOS|7.2.1511
]

CMD[/usr/bin/env python --version 2>&1]

CMD EXITCODE[0]
CMD RUNTIME[0.0051s]
STDOUT[
Python 2.7.5
]

SQL[SELECT version();]
ROWS[1]
SQL RUNTIME[0.00030708s]

CMD[/usr/bin/snmpget --version 2>&1]

CMD EXITCODE[0]
CMD RUNTIME[0.0081s]
STDOUT[
NET-SNMP version: 5.7.2
]

CMD[/usr/bin/rrdtool --version | head -n1]

CMD EXITCODE[0]
CMD RUNTIME[0.0077s]
STDOUT[
RRDtool 1.4.8  Copyright 1997-2013 by Tobias Oetiker <tobi@oetiker.ch>
]

CMD[/usr/bin/rrdcached -h | head -n1]

CMD EXITCODE[0]
CMD RUNTIME[0.0074s]
STDOUT[
RRDCacheD 1.4.8
]

CMD[/usr/bin/env apache2 -v | awk '/Server version:/ {print $3}']

CMD EXITCODE[0]
CMD RUNTIME[0.0042s]
STDOUT[

]

CMD[/usr/bin/env httpd -v | awk '/Server version:/ {print $3}']

CMD EXITCODE[0]
CMD RUNTIME[0.0364s]
STDOUT[
Apache/2.4.6
]

CMD[date "+%:z"]

Wrong process status! Issue in proc_get_status(), see: https://bugs.php.net/bug.php?id=69014
CMD EXITCODE[0]
CMD RUNTIME[0.003s]
STDOUT[
-07:00
]

SQL[SELECT TIMEDIFF(NOW(), UTC_TIMESTAMP);]
ROWS[1]
SQL RUNTIME[0.00025392s]

SQL[SELECT @@SESSION.sql_mode;]
ROWS[1]
SQL RUNTIME[0.00019908s]

SQL[SHOW VARIABLES LIKE 'character_set_connection']
ROWS[1]
SQL RUNTIME[0.00074387s]

#####  Software versions  #####

 o OS                   Linux 3.10.0-327.36.1.el7.x86_64 [amd64] (CentOS 7.2.1511)
 o Apache               2.4.6
 o PHP                  5.4.16
 o Python               2.7.5
 o MySQL                5.5.50-MariaDB (extension: mysqli 5.5.50-MariaDB)
 o SNMP                 NET-SNMP 5.7.2
 o RRDtool              1.4.8 (rrdcached 1.4.8: unix:/var/run/rrdcached/rrdcached.sock)

#####  MySQL mode  #####

 o MySQL

#####  Charset info  #####

 o PHP                  UTF-8
 o MySQL                utf8

#####  Timezones info  #####

 o Date                 Thursday, 20-Oct-16 21:46:44 PDT
 o PHP                  -07:00
 o MySQL                -07:00


SQL[SELECT * FROM `alert_tests` WHERE 1]
ROWS[32]
SQL RUNTIME[0.00056601s]
Cached 32 alert rules.

SQL[SELECT * FROM `alert_assoc`]
ROWS[32]
SQL RUNTIME[0.00039506s]

SQL[SELECT * FROM `alert_table` WHERE `alert_table_id` = '128255']
ROWS[1]
SQL RUNTIME[0.00037503s]

SQL[SELECT * FROM `devices` WHERE `device_id` = '104']
ROWS[1]
SQL RUNTIME[0.00044703s]

SQL[SELECT * FROM `device_graphs` WHERE `device_id` = '104']
ROWS[28]
SQL RUNTIME[0.00048900s]

SQL[SELECT * FROM `bgpPeers` LEFT JOIN `bgpPeers-state` USING (`bgpPeer_id`) WHERE `bgpPeers`.`bgpPeer_id` = '509']
ROWS[1]
SQL RUNTIME[0.00043297s]


> $graphtype - /opt/observium/includes/alerts.inc.php:1170
/opt/observium/html/includes/graphs/graph.inc.php:33
========================================================
array(
  [0]       => string(11) "bgp_updates"
  [type]    => string(3) "bgp"
  [1]       => string(3) "bgp"
  [subtype] => string(7) "updates"
  [2]       => string(7) "updates"
)

SQL[SELECT * FROM `bgpPeers` WHERE `bgpPeer_id` = '509']
ROWS[1]
SQL RUNTIME[0.00029802s]
RRD PIPE OPEN[TRUE]
RRD PIPE CLOSE[TRUE]
RRD PIPE STDOUT[
OK u:0.03 s:0.01 r:0.03

]

RRD CMD[graph --daemon unix:/var/run/rrdcached/rrdcached.sock /tmp/5S0b4geqsVsq08iN.png  -g -Y --lower-limit 0 --alt-autoscale-max  --start 1476852404 --end 1477025204 --width 400 --height 150 -c BACK#EEEEEE00 -c SHADEA#EEEEEE00 -c SHADEB#EEEEEE00 -c FONT#000000 -c CANVAS#FFFFFF00 -c GRID#a5a5a5 -c MGRID#FF9999 -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal --font LEGEND:8:'DroidSansMono,DejaVuSansMono' --font AXIS:7:'DroidSansMono,DejaVuSansMono' --font-render-mode normal -E COMMENT:'Updates           Last      Avg      Max \n' DEF:out=/opt/observium/rrd/us1inet2/bgp-x.x.x.x.rrd:bgpPeerOutUpdates:AVERAGE DEF:in=/opt/observium/rrd/us1inet2/bgp-x.x.x.x.rrd:bgpPeerInUpdates:AVERAGE DEF:out_max=/opt/observium/rrd/us1inet2/bgp-x.x.x.x.rrd:bgpPeerOutUpdates:MAX DEF:in_max=/opt/observium/rrd/us1inet2/bgp-x.x.x.x.rrd:bgpPeerInUpdates:MAX CDEF:dout_max=out_max,-1,* CDEF:dout=out,-1,* CDEF:both=in,out,+ CDEF:alloctets=out,in,+ CDEF:wrongin=alloctets,UN,INF,UNKN,IF CDEF:wrongout=wrongin,-1,* AREA:in#AA66AA90 LINE1.25:in#AA66AA:'In          ' GPRINT:in:LAST:.2lf%s GPRINT:in:AVERAGE:.2lf%s GPRINT:in_max:MAX:.2lf%s COMMENT:'\n' AREA:dout#FF660090: LINE1.25:dout#FF6600:'Out         ' GPRINT:out:LAST:.2lf%s GPRINT:out:AVERAGE:.2lf%s GPRINT:out_max:MAX:.2lf%s COMMENT:\n AREA:wrongin#FFF2F2 AREA:wrongout#FFF2F2 HRULE:0#999999]
RRD RUNTIME[0.0418s]
RRD STDOUT[481x189]
RRD_STATUS[TRUE]


SQL[SELECT `attrib_value` FROM `entity_attribs` WHERE `entity_type` = 'device' AND `entity_id` = '104' AND `attrib_type` = 'disable_notify']
ROWS[0]
SQL RUNTIME[0.00042915s]

SQL[SELECT * FROM `alert_contacts` WHERE `contact_disabled` = 0 AND `contact_id` IN (SELECT `contact_id` FROM `alert_contacts_assoc` WHERE `aca_type` = 'alert' AND `alert_checker_id` = '39');]
ROWS[2]
SQL RUNTIME[0.00054097s]
 o Notifying            [hipchat] Networking Alert:
                        {"room_id":"","from":"Observium","token":""}

REQUEST[https://api.hipchat.com/v2/room/xxxxxx/notification]
REQUEST STATUS[204 No Content]
REQUEST RUNTIME[0.5128s]
RESPONSE[

]


> $http_response_header - /opt/observium/includes/alerts.inc.php:1271
/opt/observium/includes/common.inc.php:3213
=========================================================
array(
  [0]  => string(23) "HTTP/1.1 204 No Content"
  [1]  => string(30) "Access-Control-Allow-Origin: *"
  [2]  => string(119) "Access-Control-Expose-Headers: Date, ETag, Link, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-Backoff"
  [3]  => string(23) "Content-Type: text/html"
  [4]  => string(35) "Date: Fri, 21 Oct 2016 04:46:44 GMT"
  [5]  => string(94) "Location: https://api.hipchat.com/v2/room/xxxxx/history/xxxxx-7c71-xxxx-xxxx-xxxxxxxxxx"
  [6]  => string(13) "Server: nginx"
  [7]  => string(43) "Strict-Transport-Security: max-age=31536000"
  [8]  => string(31) "X-Content-Type-Options: nosniff"
  [9]  => string(22) "X-Ratelimit-Limit: 100"
  [10] => string(26) "X-Ratelimit-Remaining: 100"
  [11] => string(29) "X-Ratelimit-Reset: 1477025505"
  [12] => string(31) "X-XSS-Protection: 1; mode=block"
  [13] => string(17) "Connection: Close"
)


> $opts - /opt/observium/includes/alerts.inc.php:1271
/opt/observium/includes/common.inc.php:3214
=================================================
array(
  [http] => array(
              [method]  => string(4) "POST"
              [header]  => string(174) "Connection: close
Content-Type: application/json
Content-Length: 275
Authorization: Bearer zomgWApDk5Wz9eK4kCReKjUDQxW1h9kMrRqDJnqy
User-Agent: Observium/0.16.10.8161\r\n"
              [content] => string(275) "{"from":"Observium","color":"red","message_format":"html","message":"<a href=\"http:\/\/kansatsu.lol.com\/device\/device=104\/tab=alert\/alert_entry=128255\/\">ALERT: [us1inet2] [bgp_peer] [AS65028 x.x.x.x] <\/a><br \/>bgpPeerAdminStatus = start bgpPeerState = active"}"
              [timeout] => string(2) "15"
            )
)
 o Notifying            [email] Netops: {"email":"netops@lol.com"}
Load class 'Mail_mime' from '/opt/observium/libs/pear/Mail/mime.php': OK
Load class 'Mail' from '/opt/observium/libs/pear/Mail/Mail.php': OK
PHP Strict Standards:  Non-static method Mail::factory() should not be called statically in /opt/observium/includes/alerting/email.inc.php on line 176

Strict Standards: Non-static method Mail::factory() should not be called statically in /opt/observium/includes/alerting/email.inc.php on line 176
PHP Strict Standards:  Non-static method PEAR::isError() should not be called statically in /opt/observium/includes/alerting/email.inc.php on line 180

Strict Standards: Non-static method PEAR::isError() should not be called statically in /opt/observium/includes/alerting/email.inc.php on line 180


observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium