syslog alert notifications not firing
Howdy,
I’ve setup syslog integration in our copy of observium. In the UI’s /syslog_alerts/ we are seeing the alerts. I have the syslog rules turned on in both our email and telegram contacts. I’ve rebuilt the rule checkers. Unfortunately we’re not getting alerted. We’re getting other alerts, just not for syslogs. I can’t find anything in the obs log or the server’s syslog that would tell me why it’s not working. Can anyone point me to how I can troubleshoot this please?
Thanks,
Joey
Hi,
Syslog rules don't need to be rebuilt, since they don't have state entries.
Do you see syslog alert entries? Is it just the notifications which aren't appearing, or are the rules not matching?
Generally it's a good idea to provide screenshots of everything so people can see what's actually happening.
Adam.
Sent from BlueMail
On 4 Aug 2017, 20:53, at 20:53, Joey Stanford joey@stan4d.net wrote:
Howdy,
I’ve setup syslog integration in our copy of observium. In the UI’s /syslog_alerts/ we are seeing the alerts. I have the syslog rules turned on in both our email and telegram contacts. I’ve rebuilt the rule checkers. Unfortunately we’re not getting alerted. We’re getting other alerts, just not for syslogs. I can’t find anything in the obs log or the server’s syslog that would tell me why it’s not working. Can anyone point me to how I can troubleshoot this please?
Thanks,
Joey _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On Aug 4, 2017, at 14:06 , Adam Armstrong adama@memetic.org wrote:
Do you see syslog alert entries? Is it just the notifications which aren't appearing, or are the rules not matching?
Yes, I see the alert entries and it’s just the notifications which are not showing up.
Hmm. It's difficult to debug the syslog alert generation code path, since it all happens in a process executed by the syslog daemon.
If you cause a syslog entry to trigger, you should see an entry being created in the alert queue table.
It's possible these entries aren't being inserted. Do you have any relevant errors in db.log?
Adam.
Sent from BlueMail
On 4 Aug 2017, 21:24, at 21:24, Joey Stanford joey@stan4d.net wrote:
On Aug 4, 2017, at 14:06 , Adam Armstrong adama@memetic.org wrote:
Do you see syslog alert entries? Is it just the notifications which
aren't appearing, or are the rules not matching?
Yes, I see the alert entries and it’s just the notifications which are not showing up.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On Aug 4, 2017, at 14:31, Adam Armstrong adama@memetic.org wrote:
Hmm. It's difficult to debug the syslog alert generation code path, since it all happens in a process executed by the syslog daemon.
If you cause a syslog entry to trigger, you should see an entry being created in the alert queue table.
It's possible these entries aren't being inserted. Do you have any relevant errors in db.log?
db.log is a zero-byte file.
Alert contacts Assoc has the info correct:
| 33 | syslog | 1 | 2 | | 34 | syslog | 3 | 2 | | 35 | syslog | 5 | 2 | | 36 | syslog | 4 | 2 | | 37 | syslog | 2 | 2 | | 38 | syslog | 6 | 2 | | 39 | syslog | 1 | 3 | | 40 | syslog | 3 | 3 | | 41 | syslog | 5 | 3 | | 42 | syslog | 4 | 3 | | 43 | syslog | 2 | 3 | | 44 | syslog | 6 | 3 |
syslog_alerts is also correct
--------------+-----+-----+----------+ | 1 | 2017-08-04 13:18:26 | 118 | 5 | 54545 | login failure for user joey-test25 from 192.168.77.124 via winbox | SYSTEM,ERROR,CRITICAL | 1 | 0 | 0 | | 2 | 2017-08-04 13:37:46 | 118 | 5 | 227167 | login failure for user test:we-now-have-syslog-alerts-on-routers from 192.168.77.124 via winbox | SYSTEM,ERROR,CRITICAL | 1 | 0 | 0 | | 3 | 2017-08-04 13:39:49 | 118 | 5 | 282127 | login failure for user nv0n-testing:we-now-have-syslog-alerts-on-routers from 192.168.77.124 via winbox | SYSTEM,ERROR,CRITICAL | 1 | 0 | 0 |
Syslog_rules is also correct.
Interestingly, syslog_rules_assoc is empty.
I see nothing that I can translate to an alert queue table though.
Joey
On Aug 4, 2017, at 15:17, Joey Stanford joey@stan4d.net wrote:
alert queue table though
if this is meant to be alert_table …. the syslog alerts on not present, only “regular” alerts
Hi,
No, there's a notification queue table where alerts are put before being picked up by the notification code.
The operation of this depends upon the version used as it has changed a bit recently.
Adam.
Sent from BlueMail
On 4 Aug 2017, 22:31, at 22:31, Joey Stanford joey@stan4d.net wrote:
On Aug 4, 2017, at 15:17, Joey Stanford joey@stan4d.net wrote:
alert queue table though
if this is meant to be alert_table …. the syslog alerts on not present, only “regular” alerts _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
With no intervention on my part, syslog alerts starting notifying this morning.
How curious.
I should take a look at that code and see if there's anything not behaving quite right.
Adam.
Sent from BlueMail
On 8 Aug 2017, 16:10, at 16:10, Joey Stanford joey@stan4d.net wrote:
With no intervention on my part, syslog alerts starting notifying this morning. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Good news. tl;dr: I’m an idiot.
I got woken up to a “login failure” alert this morning. Thankfully it was a false alarm from one of my guys. Got me wondering why this fired. Turns out “disable alerts” was set on the router I was using to test syslog alerting. Apparently someone in the past did that I didn’t realize it. I really need to look closer at the ignored page.
Sorry for the noise.
Perhaps we should find a way to more easily communicate that devices have alerts disabled!
Adam.
Sent from BlueMail
On 8 Aug 2017, 19:50, at 19:50, Joey Stanford joey@stan4d.net wrote:
Good news. tl;dr: I’m an idiot.
I got woken up to a “login failure” alert this morning. Thankfully it was a false alarm from one of my guys. Got me wondering why this fired. Turns out “disable alerts” was set on the router I was using to test syslog alerting. Apparently someone in the past did that I didn’t realize it. I really need to look closer at the ignored page.
Sorry for the noise. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Teilnehmer (3)
-
Adam Armstrong -
Joey Stanford -
Youssef BENGELLOUN - ZAHR