Is there a way to group email alerts so there is not an email alert for every sensor that might go down at the same time? For example multiple fans on a server might show as an alert if the device itself is not responding
How are your alert checkers built?
While we have asked for alert dependencies forever (and may never get it) I don't get any individual alarms for sensors when a device is offline.
Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Androidhttps://aka.ms/AAb9ysg ________________________________ From: Luis Londoño via observium observium@lists.observium.org Sent: Wednesday, November 23, 2022 8:31:57 AM To: observium@lists.observium.org observium@lists.observium.org Cc: Luis Londoño llondono@disti.com Subject: [Observium] Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Is there a way to group email alerts so there is not an email alert for every sensor that might go down at the same time? For example multiple fans on a server might show as an alert if the device itself is not responding
You have received a message from the law firm Miller Canfield. The information contained in or attached to this electronic mail may be privileged and/or confidential. If you received this transmission and are not the intended recipient, you should not read this message and are hereby notified that any dissemination, distribution or copying of this communication and/or its attachments is strictly prohibited. If you have received this communication in error or are not sure whether it is privileged, please immediately notify us by return e-mail and delete or destroy the original and any copies, electronic, paper or otherwise, that you may have of this communication and any attachments.
There's not really a way to "group" alerts. It could potentially be done, but not with our current alert sending code.
If you frequently get situations where you get multiple alerts all at once, being annoyed by those alerts should serve as motivation to fix the problem, no? :D
adam.
Ryan, Spencer J. via observium wrote on 2022-11-23 13:42:
How are your alert checkers built?
While we have asked for alert dependencies forever (and may never get it) I don't get any individual alarms for sensors when a device is offline.
Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Android https://aka.ms/AAb9ysg
*From:* Luis Londoño via observium observium@lists.observium.org *Sent:* Wednesday, November 23, 2022 8:31:57 AM *To:* observium@lists.observium.org observium@lists.observium.org *Cc:* Luis Londoño llondono@disti.com *Subject:* [Observium] Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Is there a way to group email alerts so there is not an email alert for every sensor that might go down at the same time? For example multiple fans on a server might show as an alert if the device itself is not responding
You have received a message from the law firm Miller Canfield. The information contained in or attached to this electronic mail may be privileged and/or confidential. If you received this transmission and are not the intended recipient, you should not read this message and are hereby notified that any dissemination, distribution or copying of this communication and/or its attachments is strictly prohibited. If you have received this communication in error or are not sure whether it is privileged, please immediately notify us by return e-mail and delete or destroy the original and any copies, electronic, paper or otherwise, that you may have of this communication and any attachments.
observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
To give an example of something that happened today and why I ask; One of the servers we are monitoring we have an alert check for the fans being below the sensor threshold something timed out on the connection and caused the sensor values to all drop to 0 and then showed the device timed out as well. We got about 20 emails about the sensors both about the alert and the recovery. I did go ahead and add a alert delay to the checker so that would at least fix that for now but maybe grouping the alerts per host if its happening all at the same time would reduce the noise of the alerts and provide a better way to know that something on that host itself might be an issue.
From: Adam Armstrong via observium observium@lists.observium.org Sent: Wednesday, November 23, 2022 8:55 AM To: Ryan, Spencer J. via observium observium@lists.observium.org Cc: Adam Armstrong adama@observium.org Subject: {EXT}[Observium] Re: Group Email Alerts
There's not really a way to "group" alerts. It could potentially be done, but not with our current alert sending code.
If you frequently get situations where you get multiple alerts all at once, being annoyed by those alerts should serve as motivation to fix the problem, no? :D
adam.
Ryan, Spencer J. via observium wrote on 2022-11-23 13:42:
How are your alert checkers built?
While we have asked for alert dependencies forever (and may never get it) I don't get any individual alarms for sensors when a device is offline.
Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Androidhttps://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cllondono%40disti.com%7Cc7b2f418cab7447c44c408dacd5a6bc2%7Cd4b51790318d4698a7956c33a8d4b716%7C1%7C0%7C638048085494730628%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PhG1kij4k%2Fw%2BOidugfGwuFSVdMfqXzWl7LI%2BAI8gF44%3D&reserved=0 ________________________________ From: Luis Londoño via observium observium@lists.observium.orgmailto:observium@lists.observium.org Sent: Wednesday, November 23, 2022 8:31:57 AM To: observium@lists.observium.orgmailto:observium@lists.observium.org observium@lists.observium.orgmailto:observium@lists.observium.org Cc: Luis Londoño llondono@disti.commailto:llondono@disti.com Subject: [Observium] Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Is there a way to group email alerts so there is not an email alert for every sensor that might go down at the same time? For example multiple fans on a server might show as an alert if the device itself is not responding
You have received a message from the law firm Miller Canfield. The information contained in or attached to this electronic mail may be privileged and/or confidential. If you received this transmission and are not the intended recipient, you should not read this message and are hereby notified that any dissemination, distribution or copying of this communication and/or its attachments is strictly prohibited. If you have received this communication in error or are not sure whether it is privileged, please immediately notify us by return e-mail and delete or destroy the original and any copies, electronic, paper or otherwise, that you may have of this communication and any attachments.
_______________________________________________
observium mailing list -- observium@lists.observium.orgmailto:observium@lists.observium.org
To unsubscribe send an email to observium-leave@lists.observium.orgmailto:observium-leave@lists.observium.org
0, or null?
I have two similar checks (one for sensor below and one for above thresholds) but a while ago I had to add null checks to avoid this.
My Sensor High/Low alarms have a delay of 1 and set it so all conditions have to be true:
sensor_value less @sensor_limit_low @sensor_limit_low ne NULL
(Obviously replace low for high on the other check)
That seems to work for us.
________________________________ Spencer J. Ryan| Manager, Technology and Infrastructure Miller Canfield T +1.313.496.7979 | F +1.313.496.7500 ________________________________
From: Luis Londoño via observium observium@lists.observium.org Sent: Wednesday, November 23, 2022 9:02 AM To: Observium observium@lists.observium.org Cc: Adam Armstrong adama@observium.org; Luis Londoño llondono@disti.com Subject: [Observium] Re: {EXT}Re: Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
To give an example of something that happened today and why I ask; One of the servers we are monitoring we have an alert check for the fans being below the sensor threshold something timed out on the connection and caused the sensor values to all drop to 0 and then showed the device timed out as well. We got about 20 emails about the sensors both about the alert and the recovery. I did go ahead and add a alert delay to the checker so that would at least fix that for now but maybe grouping the alerts per host if its happening all at the same time would reduce the noise of the alerts and provide a better way to know that something on that host itself might be an issue.
From: Adam Armstrong via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Sent: Wednesday, November 23, 2022 8:55 AM To: Ryan, Spencer J. via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Cc: Adam Armstrong <adama@observium.orgmailto:adama@observium.org> Subject: {EXT}[Observium] Re: Group Email Alerts
There's not really a way to "group" alerts. It could potentially be done, but not with our current alert sending code.
If you frequently get situations where you get multiple alerts all at once, being annoyed by those alerts should serve as motivation to fix the problem, no? :D
adam.
Ryan, Spencer J. via observium wrote on 2022-11-23 13:42: How are your alert checkers built?
While we have asked for alert dependencies forever (and may never get it) I don't get any individual alarms for sensors when a device is offline.
Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Androidhttps://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cllondono%40disti.com%7Cc7b2f418cab7447c44c408dacd5a6bc2%7Cd4b51790318d4698a7956c33a8d4b716%7C1%7C0%7C638048085494730628%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PhG1kij4k%2Fw%2BOidugfGwuFSVdMfqXzWl7LI%2BAI8gF44%3D&reserved=0 ________________________________ From: Luis Londoño via observium observium@lists.observium.orgmailto:observium@lists.observium.org Sent: Wednesday, November 23, 2022 8:31:57 AM To: observium@lists.observium.orgmailto:observium@lists.observium.org observium@lists.observium.orgmailto:observium@lists.observium.org Cc: Luis Londoño llondono@disti.commailto:llondono@disti.com Subject: [Observium] Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Is there a way to group email alerts so there is not an email alert for every sensor that might go down at the same time? For example multiple fans on a server might show as an alert if the device itself is not responding
You have received a message from the law firm Miller Canfield. The information contained in or attached to this electronic mail may be privileged and/or confidential. If you received this transmission and are not the intended recipient, you should not read this message and are hereby notified that any dissemination, distribution or copying of this communication and/or its attachments is strictly prohibited. If you have received this communication in error or are not sure whether it is privileged, please immediately notify us by return e-mail and delete or destroy the original and any copies, electronic, paper or otherwise, that you may have of this communication and any attachments.
_______________________________________________
observium mailing list -- observium@lists.observium.orgmailto:observium@lists.observium.org
To unsubscribe send an email to observium-leave@lists.observium.orgmailto:observium-leave@lists.observium.org
Thank you for the suggestion. Do you get this error by any chance? [cid:image001.png@01D8FF1C.8B410150]
From: Ryan, Spencer J. Ryan@millercanfield.com Sent: Wednesday, November 23, 2022 9:13 AM To: Observium observium@lists.observium.org Cc: Adam Armstrong adama@observium.org; Luis Londoño llondono@disti.com Subject: RE: [Observium] Re: {EXT}Re: Group Email Alerts
You don't often get email from ryan@millercanfield.commailto:ryan@millercanfield.com. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification 0, or null?
I have two similar checks (one for sensor below and one for above thresholds) but a while ago I had to add null checks to avoid this.
My Sensor High/Low alarms have a delay of 1 and set it so all conditions have to be true:
sensor_value less @sensor_limit_low @sensor_limit_low ne NULL
(Obviously replace low for high on the other check)
That seems to work for us.
________________________________ Spencer J. Ryan| Manager, Technology and Infrastructure Miller Canfield T +1.313.496.7979 | F +1.313.496.7500 ________________________________
From: Luis Londoño via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Sent: Wednesday, November 23, 2022 9:02 AM To: Observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Cc: Adam Armstrong <adama@observium.orgmailto:adama@observium.org>; Luis Londoño <llondono@disti.commailto:llondono@disti.com> Subject: [Observium] Re: {EXT}Re: Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
To give an example of something that happened today and why I ask; One of the servers we are monitoring we have an alert check for the fans being below the sensor threshold something timed out on the connection and caused the sensor values to all drop to 0 and then showed the device timed out as well. We got about 20 emails about the sensors both about the alert and the recovery. I did go ahead and add a alert delay to the checker so that would at least fix that for now but maybe grouping the alerts per host if its happening all at the same time would reduce the noise of the alerts and provide a better way to know that something on that host itself might be an issue.
From: Adam Armstrong via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Sent: Wednesday, November 23, 2022 8:55 AM To: Ryan, Spencer J. via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Cc: Adam Armstrong <adama@observium.orgmailto:adama@observium.org> Subject: {EXT}[Observium] Re: Group Email Alerts
There's not really a way to "group" alerts. It could potentially be done, but not with our current alert sending code.
If you frequently get situations where you get multiple alerts all at once, being annoyed by those alerts should serve as motivation to fix the problem, no? :D
adam.
Ryan, Spencer J. via observium wrote on 2022-11-23 13:42: How are your alert checkers built?
While we have asked for alert dependencies forever (and may never get it) I don't get any individual alarms for sensors when a device is offline.
Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Androidhttps://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cllondono%40disti.com%7C7d01993ce32f4d93afc908dacd5cd194%7Cd4b51790318d4698a7956c33a8d4b716%7C1%7C0%7C638048095794850730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Y3noLRhM4Vie%2FESKB7IeCpME7owq8ZOWMcbbcrgRXss%3D&reserved=0 ________________________________ From: Luis Londoño via observium observium@lists.observium.orgmailto:observium@lists.observium.org Sent: Wednesday, November 23, 2022 8:31:57 AM To: observium@lists.observium.orgmailto:observium@lists.observium.org observium@lists.observium.orgmailto:observium@lists.observium.org Cc: Luis Londoño llondono@disti.commailto:llondono@disti.com Subject: [Observium] Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Is there a way to group email alerts so there is not an email alert for every sensor that might go down at the same time? For example multiple fans on a server might show as an alert if the device itself is not responding
You have received a message from the law firm Miller Canfield. The information contained in or attached to this electronic mail may be privileged and/or confidential. If you received this transmission and are not the intended recipient, you should not read this message and are hereby notified that any dissemination, distribution or copying of this communication and/or its attachments is strictly prohibited. If you have received this communication in error or are not sure whether it is privileged, please immediately notify us by return e-mail and delete or destroy the original and any copies, electronic, paper or otherwise, that you may have of this communication and any attachments.
_______________________________________________
observium mailing list -- observium@lists.observium.orgmailto:observium@lists.observium.org
To unsubscribe send an email to observium-leave@lists.observium.orgmailto:observium-leave@lists.observium.org
Funny enough, yes, although I've never seen that before today (when I loaded the check to grab you the conditions). I'm guessing they added some checks at some point for input sanity, but it seems to still work normally.
Adam might know (or be able to fix it!)
________________________________ Spencer J. Ryan| Manager, Technology and Infrastructure Miller Canfield T +1.313.496.7979 | F +1.313.496.7500 ________________________________
From: Luis Londoño llondono@disti.com Sent: Wednesday, November 23, 2022 9:18 AM To: Ryan, Spencer J. Ryan@millercanfield.com; Observium observium@lists.observium.org Cc: Adam Armstrong adama@observium.org Subject: RE: [Observium] Re: {EXT}Re: Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Thank you for the suggestion. Do you get this error by any chance? [cid:image001.png@01D8FF1C.DBA265D0]
From: Ryan, Spencer J. <Ryan@millercanfield.commailto:Ryan@millercanfield.com> Sent: Wednesday, November 23, 2022 9:13 AM To: Observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Cc: Adam Armstrong <adama@observium.orgmailto:adama@observium.org>; Luis Londoño <llondono@disti.commailto:llondono@disti.com> Subject: RE: [Observium] Re: {EXT}Re: Group Email Alerts
You don't often get email from ryan@millercanfield.commailto:ryan@millercanfield.com. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification 0, or null?
I have two similar checks (one for sensor below and one for above thresholds) but a while ago I had to add null checks to avoid this.
My Sensor High/Low alarms have a delay of 1 and set it so all conditions have to be true:
sensor_value less @sensor_limit_low @sensor_limit_low ne NULL
(Obviously replace low for high on the other check)
That seems to work for us.
________________________________ Spencer J. Ryan| Manager, Technology and Infrastructure Miller Canfield T +1.313.496.7979 | F +1.313.496.7500 ________________________________
From: Luis Londoño via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Sent: Wednesday, November 23, 2022 9:02 AM To: Observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Cc: Adam Armstrong <adama@observium.orgmailto:adama@observium.org>; Luis Londoño <llondono@disti.commailto:llondono@disti.com> Subject: [Observium] Re: {EXT}Re: Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
To give an example of something that happened today and why I ask; One of the servers we are monitoring we have an alert check for the fans being below the sensor threshold something timed out on the connection and caused the sensor values to all drop to 0 and then showed the device timed out as well. We got about 20 emails about the sensors both about the alert and the recovery. I did go ahead and add a alert delay to the checker so that would at least fix that for now but maybe grouping the alerts per host if its happening all at the same time would reduce the noise of the alerts and provide a better way to know that something on that host itself might be an issue.
From: Adam Armstrong via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Sent: Wednesday, November 23, 2022 8:55 AM To: Ryan, Spencer J. via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Cc: Adam Armstrong <adama@observium.orgmailto:adama@observium.org> Subject: {EXT}[Observium] Re: Group Email Alerts
There's not really a way to "group" alerts. It could potentially be done, but not with our current alert sending code.
If you frequently get situations where you get multiple alerts all at once, being annoyed by those alerts should serve as motivation to fix the problem, no? :D
adam.
Ryan, Spencer J. via observium wrote on 2022-11-23 13:42: How are your alert checkers built?
While we have asked for alert dependencies forever (and may never get it) I don't get any individual alarms for sensors when a device is offline.
Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Androidhttps://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cllondono%40disti.com%7C7d01993ce32f4d93afc908dacd5cd194%7Cd4b51790318d4698a7956c33a8d4b716%7C1%7C0%7C638048095794850730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Y3noLRhM4Vie%2FESKB7IeCpME7owq8ZOWMcbbcrgRXss%3D&reserved=0 ________________________________ From: Luis Londoño via observium observium@lists.observium.orgmailto:observium@lists.observium.org Sent: Wednesday, November 23, 2022 8:31:57 AM To: observium@lists.observium.orgmailto:observium@lists.observium.org observium@lists.observium.orgmailto:observium@lists.observium.org Cc: Luis Londoño llondono@disti.commailto:llondono@disti.com Subject: [Observium] Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Is there a way to group email alerts so there is not an email alert for every sensor that might go down at the same time? For example multiple fans on a server might show as an alert if the device itself is not responding
You have received a message from the law firm Miller Canfield. The information contained in or attached to this electronic mail may be privileged and/or confidential. If you received this transmission and are not the intended recipient, you should not read this message and are hereby notified that any dissemination, distribution or copying of this communication and/or its attachments is strictly prohibited. If you have received this communication in error or are not sure whether it is privileged, please immediately notify us by return e-mail and delete or destroy the original and any copies, electronic, paper or otherwise, that you may have of this communication and any attachments.
_______________________________________________
observium mailing list -- observium@lists.observium.orgmailto:observium@lists.observium.org
To unsubscribe send an email to observium-leave@lists.observium.orgmailto:observium-leave@lists.observium.org
Why are you checking if the threshold is null?
You want to check if the value is null :)
adam.
Ryan, Spencer J. via observium wrote on 2022-11-23 14:20:
Funny enough, yes, although I’ve never seen that before today (when I loaded the check to grab you the conditions). I’m guessing they added some checks at some point for input sanity, but it seems to still work normally.
Adam might know (or be able to fix it!)
Spencer J. Ryan| Manager, Technology and Infrastructure *Miller Canfield*** *T* +1.313.496.7979 | *F* +1.313.496.7500
*From:*Luis Londoño llondono@disti.com *Sent:* Wednesday, November 23, 2022 9:18 AM *To:* Ryan, Spencer J. Ryan@millercanfield.com; Observium observium@lists.observium.org *Cc:* Adam Armstrong adama@observium.org *Subject:* RE: [Observium] Re: {EXT}Re: Group Email Alerts
*Caution:*This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Thank you for the suggestion. Do you get this error by any chance?
*From:*Ryan, Spencer J. <Ryan@millercanfield.com mailto:Ryan@millercanfield.com> *Sent:* Wednesday, November 23, 2022 9:13 AM *To:* Observium <observium@lists.observium.org mailto:observium@lists.observium.org> *Cc:* Adam Armstrong <adama@observium.org mailto:adama@observium.org>; Luis Londoño <llondono@disti.com mailto:llondono@disti.com> *Subject:* RE: [Observium] Re: {EXT}Re: Group Email Alerts
You don't often get email from ryan@millercanfield.com mailto:ryan@millercanfield.com. Learn why this is important https://aka.ms/LearnAboutSenderIdentification
0, or null?
I have two similar checks (one for sensor below and one for above thresholds) but a while ago I had to add null checks to avoid this.
My Sensor High/Low alarms have a delay of 1 and set it so all conditions have to be true:
|sensor_value less @sensor_limit_low| |@sensor_limit_low ne NULL|
||
|(Obviously replace low for high on the other check)|
||
||
|That seems to work for us. |
Spencer J. Ryan| Manager, Technology and Infrastructure *Miller Canfield*** *T* +1.313.496.7979 | *F* +1.313.496.7500
*From:*Luis Londoño via observium <observium@lists.observium.org mailto:observium@lists.observium.org> *Sent:* Wednesday, November 23, 2022 9:02 AM *To:* Observium <observium@lists.observium.org mailto:observium@lists.observium.org> *Cc:* Adam Armstrong <adama@observium.org mailto:adama@observium.org>; Luis Londoño <llondono@disti.com mailto:llondono@disti.com> *Subject:* [Observium] Re: {EXT}Re: Group Email Alerts
*Caution:*This is an external email. Do not open attachments or click links from unknown or unexpected emails.
To give an example of something that happened today and why I ask; One of the servers we are monitoring we have an alert check for the fans being below the sensor threshold something timed out on the connection and caused the sensor values to all drop to 0 and then showed the device timed out as well. We got about 20 emails about the sensors both about the alert and the recovery. I did go ahead and add a alert delay to the checker so that would at least fix that for now but maybe grouping the alerts per host if its happening all at the same time would reduce the noise of the alerts and provide a better way to know that something on that host itself might be an issue.
*From:*Adam Armstrong via observium <observium@lists.observium.org mailto:observium@lists.observium.org> *Sent:* Wednesday, November 23, 2022 8:55 AM *To:* Ryan, Spencer J. via observium <observium@lists.observium.org mailto:observium@lists.observium.org> *Cc:* Adam Armstrong <adama@observium.org mailto:adama@observium.org> *Subject:* {EXT}[Observium] Re: Group Email Alerts
There's not really a way to "group" alerts. It could potentially be done, but not with our current alert sending code.
If you frequently get situations where you get multiple alerts all at once, being annoyed by those alerts should serve as motivation to fix the problem, no? :D
adam.
Ryan, Spencer J. via observium wrote on 2022-11-23 13:42:
How are your alert checkers built? While we have asked for alert dependencies forever (and may never get it) I don't get any individual alarms for sensors when a device is offline. Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Android <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cllondono%40disti.com%7C7d01993ce32f4d93afc908dacd5cd194%7Cd4b51790318d4698a7956c33a8d4b716%7C1%7C0%7C638048095794850730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Y3noLRhM4Vie%2FESKB7IeCpME7owq8ZOWMcbbcrgRXss%3D&reserved=0> ------------------------------------------------------------------------ *From:* Luis Londoño via observium <observium@lists.observium.org> <mailto:observium@lists.observium.org> *Sent:* Wednesday, November 23, 2022 8:31:57 AM *To:* observium@lists.observium.org <mailto:observium@lists.observium.org> <observium@lists.observium.org> <mailto:observium@lists.observium.org> *Cc:* Luis Londoño <llondono@disti.com> <mailto:llondono@disti.com> *Subject:* [Observium] Group Email Alerts *Caution:*This is an external email. Do not open attachments or click links from unknown or unexpected emails. Is there a way to group email alerts so there is not an email alert for every sensor that might go down at the same time? For example multiple fans on a server might show as an alert if the device itself is not responding You have received a message from the law firm Miller Canfield. The information contained in or attached to this electronic mail may be privileged and/or confidential. If you received this transmission and are not the intended recipient, you should not read this message and are hereby notified that any dissemination, distribution or copying of this communication and/or its attachments is strictly prohibited. If you have received this communication in error or are not sure whether it is privileged, please immediately notify us by return e-mail and delete or destroy the original and any copies, electronic, paper or otherwise, that you may have of this communication and any attachments. _______________________________________________ observium mailing list --observium@lists.observium.org <mailto:observium@lists.observium.org> To unsubscribe send an email toobservium-leave@lists.observium.org <mailto:observium-leave@lists.observium.org>--
observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
Because on random discovery failures the sensor threshold would go to NULL and then start alarming.
________________________________ Spencer J. Ryan| Manager, Technology and Infrastructure Miller Canfield T +1.313.496.7979 | F +1.313.496.7500 ________________________________
From: Adam Armstrong via observium observium@lists.observium.org Sent: Wednesday, November 23, 2022 9:53 AM To: Ryan, Spencer J. via observium observium@lists.observium.org Cc: Adam Armstrong adama@observium.org Subject: [Observium] Re: {EXT}Re: Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Why are you checking if the threshold is null?
You want to check if the value is null :)
adam.
Ryan, Spencer J. via observium wrote on 2022-11-23 14:20:
Funny enough, yes, although I've never seen that before today (when I loaded the check to grab you the conditions). I'm guessing they added some checks at some point for input sanity, but it seems to still work normally.
Adam might know (or be able to fix it!)
________________________________ Spencer J. Ryan| Manager, Technology and Infrastructure Miller Canfield T +1.313.496.7979 | F +1.313.496.7500 ________________________________
From: Luis Londoño llondono@disti.commailto:llondono@disti.com Sent: Wednesday, November 23, 2022 9:18 AM To: Ryan, Spencer J. Ryan@millercanfield.commailto:Ryan@millercanfield.com; Observium observium@lists.observium.orgmailto:observium@lists.observium.org Cc: Adam Armstrong adama@observium.orgmailto:adama@observium.org Subject: RE: [Observium] Re: {EXT}Re: Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Thank you for the suggestion. Do you get this error by any chance? [cid:image001.png@01D8FF21.CFA7BD70]
From: Ryan, Spencer J. <Ryan@millercanfield.commailto:Ryan@millercanfield.com> Sent: Wednesday, November 23, 2022 9:13 AM To: Observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Cc: Adam Armstrong <adama@observium.orgmailto:adama@observium.org>; Luis Londoño <llondono@disti.commailto:llondono@disti.com> Subject: RE: [Observium] Re: {EXT}Re: Group Email Alerts
You don't often get email from ryan@millercanfield.commailto:ryan@millercanfield.com. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification 0, or null?
I have two similar checks (one for sensor below and one for above thresholds) but a while ago I had to add null checks to avoid this.
My Sensor High/Low alarms have a delay of 1 and set it so all conditions have to be true:
sensor_value less @sensor_limit_low @sensor_limit_low ne NULL
(Obviously replace low for high on the other check)
That seems to work for us.
________________________________ Spencer J. Ryan| Manager, Technology and Infrastructure Miller Canfield T +1.313.496.7979 | F +1.313.496.7500 ________________________________
From: Luis Londoño via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Sent: Wednesday, November 23, 2022 9:02 AM To: Observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Cc: Adam Armstrong <adama@observium.orgmailto:adama@observium.org>; Luis Londoño <llondono@disti.commailto:llondono@disti.com> Subject: [Observium] Re: {EXT}Re: Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
To give an example of something that happened today and why I ask; One of the servers we are monitoring we have an alert check for the fans being below the sensor threshold something timed out on the connection and caused the sensor values to all drop to 0 and then showed the device timed out as well. We got about 20 emails about the sensors both about the alert and the recovery. I did go ahead and add a alert delay to the checker so that would at least fix that for now but maybe grouping the alerts per host if its happening all at the same time would reduce the noise of the alerts and provide a better way to know that something on that host itself might be an issue.
From: Adam Armstrong via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Sent: Wednesday, November 23, 2022 8:55 AM To: Ryan, Spencer J. via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Cc: Adam Armstrong <adama@observium.orgmailto:adama@observium.org> Subject: {EXT}[Observium] Re: Group Email Alerts
There's not really a way to "group" alerts. It could potentially be done, but not with our current alert sending code.
If you frequently get situations where you get multiple alerts all at once, being annoyed by those alerts should serve as motivation to fix the problem, no? :D
adam.
Ryan, Spencer J. via observium wrote on 2022-11-23 13:42: How are your alert checkers built?
While we have asked for alert dependencies forever (and may never get it) I don't get any individual alarms for sensors when a device is offline.
Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Androidhttps://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cllondono%40disti.com%7C7d01993ce32f4d93afc908dacd5cd194%7Cd4b51790318d4698a7956c33a8d4b716%7C1%7C0%7C638048095794850730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Y3noLRhM4Vie%2FESKB7IeCpME7owq8ZOWMcbbcrgRXss%3D&reserved=0 ________________________________ From: Luis Londoño via observium observium@lists.observium.orgmailto:observium@lists.observium.org Sent: Wednesday, November 23, 2022 8:31:57 AM To: observium@lists.observium.orgmailto:observium@lists.observium.org observium@lists.observium.orgmailto:observium@lists.observium.org Cc: Luis Londoño llondono@disti.commailto:llondono@disti.com Subject: [Observium] Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Is there a way to group email alerts so there is not an email alert for every sensor that might go down at the same time? For example multiple fans on a server might show as an alert if the device itself is not responding
You have received a message from the law firm Miller Canfield. The information contained in or attached to this electronic mail may be privileged and/or confidential. If you received this transmission and are not the intended recipient, you should not read this message and are hereby notified that any dissemination, distribution or copying of this communication and/or its attachments is strictly prohibited. If you have received this communication in error or are not sure whether it is privileged, please immediately notify us by return e-mail and delete or destroy the original and any copies, electronic, paper or otherwise, that you may have of this communication and any attachments.
_______________________________________________
observium mailing list -- observium@lists.observium.orgmailto:observium@lists.observium.org
To unsubscribe send an email to observium-leave@lists.observium.orgmailto:observium-leave@lists.observium.org
_______________________________________________
observium mailing list -- observium@lists.observium.orgmailto:observium@lists.observium.org
To unsubscribe send an email to observium-leave@lists.observium.orgmailto:observium-leave@lists.observium.org
-- Sent from Postboxhttps://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach
That's a clever hack ;)
Cheers.
Le mer. 23 nov. 2022, à 09 h 58, Ryan, Spencer J. via observium < observium@lists.observium.org> a écrit :
Because on random discovery failures the sensor threshold would go to NULL and then start alarming.
Spencer J. Ryan| Manager, Technology and Infrastructure *Miller Canfield*
*T* +1.313.496.7979 | *F* +1.313.496.7500
*From:* Adam Armstrong via observium observium@lists.observium.org *Sent:* Wednesday, November 23, 2022 9:53 AM *To:* Ryan, Spencer J. via observium observium@lists.observium.org *Cc:* Adam Armstrong adama@observium.org *Subject:* [Observium] Re: {EXT}Re: Group Email Alerts
*Caution:* This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Why are you checking if the threshold is null?
You want to check if the value is null :)
adam.
Ryan, Spencer J. via observium wrote on 2022-11-23 14:20:
Funny enough, yes, although I’ve never seen that before today (when I loaded the check to grab you the conditions). I’m guessing they added some checks at some point for input sanity, but it seems to still work normally.
Adam might know (or be able to fix it!)
Spencer J. Ryan| Manager, Technology and Infrastructure *Miller Canfield*
*T* +1.313.496.7979 | *F* +1.313.496.7500
*From:* Luis Londoño llondono@disti.com llondono@disti.com *Sent:* Wednesday, November 23, 2022 9:18 AM *To:* Ryan, Spencer J. Ryan@millercanfield.com Ryan@millercanfield.com; Observium observium@lists.observium.org observium@lists.observium.org *Cc:* Adam Armstrong adama@observium.org adama@observium.org *Subject:* RE: [Observium] Re: {EXT}Re: Group Email Alerts
*Caution:* This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Thank you for the suggestion. Do you get this error by any chance?
*From:* Ryan, Spencer J. Ryan@millercanfield.com *Sent:* Wednesday, November 23, 2022 9:13 AM *To:* Observium observium@lists.observium.org *Cc:* Adam Armstrong adama@observium.org; Luis Londoño < llondono@disti.com> *Subject:* RE: [Observium] Re: {EXT}Re: Group Email Alerts
You don't often get email from ryan@millercanfield.com. Learn why this is important https://aka.ms/LearnAboutSenderIdentification
0, or null?
I have two similar checks (one for sensor below and one for above thresholds) but a while ago I had to add null checks to avoid this.
My Sensor High/Low alarms have a delay of 1 and set it so all conditions have to be true:
sensor_value less @sensor_limit_low @sensor_limit_low ne NULL
(Obviously replace low for high on the other check)
That seems to work for us.
Spencer J. Ryan| Manager, Technology and Infrastructure *Miller Canfield*
*T* +1.313.496.7979 | *F* +1.313.496.7500
*From:* Luis Londoño via observium observium@lists.observium.org *Sent:* Wednesday, November 23, 2022 9:02 AM *To:* Observium observium@lists.observium.org *Cc:* Adam Armstrong adama@observium.org; Luis Londoño < llondono@disti.com> *Subject:* [Observium] Re: {EXT}Re: Group Email Alerts
*Caution:* This is an external email. Do not open attachments or click links from unknown or unexpected emails.
To give an example of something that happened today and why I ask; One of the servers we are monitoring we have an alert check for the fans being below the sensor threshold something timed out on the connection and caused the sensor values to all drop to 0 and then showed the device timed out as well. We got about 20 emails about the sensors both about the alert and the recovery. I did go ahead and add a alert delay to the checker so that would at least fix that for now but maybe grouping the alerts per host if its happening all at the same time would reduce the noise of the alerts and provide a better way to know that something on that host itself might be an issue.
*From:* Adam Armstrong via observium observium@lists.observium.org *Sent:* Wednesday, November 23, 2022 8:55 AM *To:* Ryan, Spencer J. via observium observium@lists.observium.org *Cc:* Adam Armstrong adama@observium.org *Subject:* {EXT}[Observium] Re: Group Email Alerts
There's not really a way to "group" alerts. It could potentially be done, but not with our current alert sending code.
If you frequently get situations where you get multiple alerts all at once, being annoyed by those alerts should serve as motivation to fix the problem, no? :D
adam.
Ryan, Spencer J. via observium wrote on 2022-11-23 13:42:
How are your alert checkers built?
While we have asked for alert dependencies forever (and may never get it) I don't get any individual alarms for sensors when a device is offline.
Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Android
https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cllondono%40disti.com%7C7d01993ce32f4d93afc908dacd5cd194%7Cd4b51790318d4698a7956c33a8d4b716%7C1%7C0%7C638048095794850730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Y3noLRhM4Vie%2FESKB7IeCpME7owq8ZOWMcbbcrgRXss%3D&reserved=0
*From:* Luis Londoño via observium observium@lists.observium.org observium@lists.observium.org *Sent:* Wednesday, November 23, 2022 8:31:57 AM *To:* observium@lists.observium.org observium@lists.observium.org observium@lists.observium.org *Cc:* Luis Londoño llondono@disti.com llondono@disti.com *Subject:* [Observium] Group Email Alerts
*Caution:* This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Is there a way to group email alerts so there is not an email alert for every sensor that might go down at the same time? For example multiple fans on a server might show as an alert if the device itself is not responding
You have received a message from the law firm Miller Canfield. The information contained in or attached to this electronic mail may be privileged and/or confidential. If you received this transmission and are not the intended recipient, you should not read this message and are hereby notified that any dissemination, distribution or copying of this communication and/or its attachments is strictly prohibited. If you have received this communication in error or are not sure whether it is privileged, please immediately notify us by return e-mail and delete or destroy the original and any copies, electronic, paper or otherwise, that you may have of this communication and any attachments.
observium mailing list -- observium@lists.observium.org
To unsubscribe send an email to observium-leave@lists.observium.org
--
observium mailing list -- observium@lists.observium.org
To unsubscribe send an email to observium-leave@lists.observium.org
--
Sent from Postbox https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach _______________________________________________ observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
Hmm. Are you sure this has ever worked?
The thresholds shouldn't have ever been present in the metric array. The code the does the check doesn't pass these as metrics, and the code that rewrites the @thresholds on the test side doesn't work on the "metric" side of the comparison.
adam.
Ahmed Rahal via observium wrote on 2022-11-23 14:59:
That's a clever hack ;)
Cheers.
Le mer. 23 nov. 2022, à 09 h 58, Ryan, Spencer J. via observium <observium@lists.observium.org mailto:observium@lists.observium.org> a écrit :
Because on random discovery failures the sensor threshold would go to NULL and then start alarming. ------------------------------------------------------------------------ Spencer J. Ryan| Manager, Technology and Infrastructure *Miller Canfield*** *T* +1.313.496.7979 | *F* +1.313.496.7500 ------------------------------------------------------------------------ *From:*Adam Armstrong via observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> *Sent:* Wednesday, November 23, 2022 9:53 AM *To:* Ryan, Spencer J. via observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> *Cc:* Adam Armstrong <adama@observium.org <mailto:adama@observium.org>> *Subject:* [Observium] Re: {EXT}Re: Group Email Alerts *Caution:*This is an external email. Do not open attachments or click links from unknown or unexpected emails. Why are you checking if the threshold is null? You want to check if the value is null :) adam. Ryan, Spencer J. via observium wrote on 2022-11-23 14:20: Funny enough, yes, although I’ve never seen that before today (when I loaded the check to grab you the conditions). I’m guessing they added some checks at some point for input sanity, but it seems to still work normally. Adam might know (or be able to fix it!) ------------------------------------------------------------------------ Spencer J. Ryan| Manager, Technology and Infrastructure *Miller Canfield*** *T* +1.313.496.7979 | *F* +1.313.496.7500 ------------------------------------------------------------------------ *From:*Luis Londoño <llondono@disti.com> <mailto:llondono@disti.com> *Sent:* Wednesday, November 23, 2022 9:18 AM *To:* Ryan, Spencer J. <Ryan@millercanfield.com> <mailto:Ryan@millercanfield.com>; Observium <observium@lists.observium.org> <mailto:observium@lists.observium.org> *Cc:* Adam Armstrong <adama@observium.org> <mailto:adama@observium.org> *Subject:* RE: [Observium] Re: {EXT}Re: Group Email Alerts *Caution:*This is an external email. Do not open attachments or click links from unknown or unexpected emails. Thank you for the suggestion. Do you get this error by any chance? *From:*Ryan, Spencer J. <Ryan@millercanfield.com <mailto:Ryan@millercanfield.com>> *Sent:* Wednesday, November 23, 2022 9:13 AM *To:* Observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> *Cc:* Adam Armstrong <adama@observium.org <mailto:adama@observium.org>>; Luis Londoño <llondono@disti.com <mailto:llondono@disti.com>> *Subject:* RE: [Observium] Re: {EXT}Re: Group Email Alerts You don't often get email from ryan@millercanfield.com <mailto:ryan@millercanfield.com>. Learn why this is important <https://aka.ms/LearnAboutSenderIdentification> 0, or null? I have two similar checks (one for sensor below and one for above thresholds) but a while ago I had to add null checks to avoid this. My Sensor High/Low alarms have a delay of 1 and set it so all conditions have to be true: |sensor_value less @sensor_limit_low| |@sensor_limit_low ne NULL| || |(Obviously replace low for high on the other check)| || || |That seems to work for us. | ------------------------------------------------------------------------ Spencer J. Ryan| Manager, Technology and Infrastructure *Miller Canfield*** *T* +1.313.496.7979 | *F* +1.313.496.7500 ------------------------------------------------------------------------ *From:*Luis Londoño via observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> *Sent:* Wednesday, November 23, 2022 9:02 AM *To:* Observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> *Cc:* Adam Armstrong <adama@observium.org <mailto:adama@observium.org>>; Luis Londoño <llondono@disti.com <mailto:llondono@disti.com>> *Subject:* [Observium] Re: {EXT}Re: Group Email Alerts *Caution:*This is an external email. Do not open attachments or click links from unknown or unexpected emails. To give an example of something that happened today and why I ask; One of the servers we are monitoring we have an alert check for the fans being below the sensor threshold something timed out on the connection and caused the sensor values to all drop to 0 and then showed the device timed out as well. We got about 20 emails about the sensors both about the alert and the recovery. I did go ahead and add a alert delay to the checker so that would at least fix that for now but maybe grouping the alerts per host if its happening all at the same time would reduce the noise of the alerts and provide a better way to know that something on that host itself might be an issue. *From:*Adam Armstrong via observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> *Sent:* Wednesday, November 23, 2022 8:55 AM *To:* Ryan, Spencer J. via observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> *Cc:* Adam Armstrong <adama@observium.org <mailto:adama@observium.org>> *Subject:* {EXT}[Observium] Re: Group Email Alerts There's not really a way to "group" alerts. It could potentially be done, but not with our current alert sending code. If you frequently get situations where you get multiple alerts all at once, being annoyed by those alerts should serve as motivation to fix the problem, no? :D adam. Ryan, Spencer J. via observium wrote on 2022-11-23 13:42: How are your alert checkers built? While we have asked for alert dependencies forever (and may never get it) I don't get any individual alarms for sensors when a device is offline. Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Android <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cllondono%40disti.com%7C7d01993ce32f4d93afc908dacd5cd194%7Cd4b51790318d4698a7956c33a8d4b716%7C1%7C0%7C638048095794850730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Y3noLRhM4Vie%2FESKB7IeCpME7owq8ZOWMcbbcrgRXss%3D&reserved=0> ------------------------------------------------------------------------ *From:* Luis Londoño via observium <observium@lists.observium.org> <mailto:observium@lists.observium.org> *Sent:* Wednesday, November 23, 2022 8:31:57 AM *To:* observium@lists.observium.org <mailto:observium@lists.observium.org> <observium@lists.observium.org> <mailto:observium@lists.observium.org> *Cc:* Luis Londoño <llondono@disti.com> <mailto:llondono@disti.com> *Subject:* [Observium] Group Email Alerts *Caution:*This is an external email. Do not open attachments or click links from unknown or unexpected emails. Is there a way to group email alerts so there is not an email alert for every sensor that might go down at the same time? For example multiple fans on a server might show as an alert if the device itself is not responding You have received a message from the law firm Miller Canfield. The information contained in or attached to this electronic mail may be privileged and/or confidential. If you received this transmission and are not the intended recipient, you should not read this message and are hereby notified that any dissemination, distribution or copying of this communication and/or its attachments is strictly prohibited. If you have received this communication in error or are not sure whether it is privileged, please immediately notify us by return e-mail and delete or destroy the original and any copies, electronic, paper or otherwise, that you may have of this communication and any attachments. _______________________________________________ observium mailing list --observium@lists.observium.org <mailto:observium@lists.observium.org> To unsubscribe send an email toobservium-leave@lists.observium.org <mailto:observium-leave@lists.observium.org> -- Sent from Postbox <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fwww.postbox-inc.com%2F%3Futm_source%3Demail%26utm_medium%3Dsiglink%26utm_campaign%3Dreach&data=05%7C01%7Cllondono%40disti.com%7C7d01993ce32f4d93afc908dacd5cd194%7Cd4b51790318d4698a7956c33a8d4b716%7C1%7C0%7C638048095794850730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=givrKgpt7EmK32gCgqqobO4%2FktvYCEM%2FI%2FcqMQDOcpc%3D&reserved=0> _______________________________________________ observium mailing list --observium@lists.observium.org <mailto:observium@lists.observium.org> To unsubscribe send an email toobservium-leave@lists.observium.org <mailto:observium-leave@lists.observium.org> -- Sent from Postbox <https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach> _______________________________________________ observium mailing list -- observium@lists.observium.org <mailto:observium@lists.observium.org> To unsubscribe send an email to observium-leave@lists.observium.org <mailto:observium-leave@lists.observium.org>-- Ahmed Rahal Administrateur de Systèmes / Systems Administrator/ / *Videotron/Fibrenoire* - www.fibrenoire.ca http://www.fibrenoire.ca/ A: 612 Rue Saint-Jacques, Montréal, QC H3C 1E2 ahmed.rahal@videotron.com mailto:ahmed.rahal@videotron.com Twitter: @fibrenoire
observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
100%, because at some point in the last year or two, some code changed that would cause the (some) alert code to trigger when the sensor value was NULL, we added this logic and the issues went away.
Someone else posted the solution to this list on 7/9/21 and it worked for us.
________________________________ Spencer J. Ryan| Manager, Technology and Infrastructure Miller Canfield T +1.313.496.7979 | F +1.313.496.7500 ________________________________
From: Adam Armstrong via observium observium@lists.observium.org Sent: Wednesday, November 23, 2022 10:21 AM To: Ahmed Rahal via observium observium@lists.observium.org Cc: Adam Armstrong adama@observium.org Subject: [Observium] Re: {EXT}Re: Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Hmm. Are you sure this has ever worked?
The thresholds shouldn't have ever been present in the metric array. The code the does the check doesn't pass these as metrics, and the code that rewrites the @thresholds on the test side doesn't work on the "metric" side of the comparison.
adam.
Ahmed Rahal via observium wrote on 2022-11-23 14:59:
That's a clever hack ;)
Cheers.
Le mer. 23 nov. 2022, à 09 h 58, Ryan, Spencer J. via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> a écrit : Because on random discovery failures the sensor threshold would go to NULL and then start alarming.
________________________________ Spencer J. Ryan| Manager, Technology and Infrastructure Miller Canfield T +1.313.496.7979 | F +1.313.496.7500 ________________________________
From: Adam Armstrong via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Sent: Wednesday, November 23, 2022 9:53 AM To: Ryan, Spencer J. via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Cc: Adam Armstrong <adama@observium.orgmailto:adama@observium.org> Subject: [Observium] Re: {EXT}Re: Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Why are you checking if the threshold is null?
You want to check if the value is null :)
adam.
Ryan, Spencer J. via observium wrote on 2022-11-23 14:20: Funny enough, yes, although I’ve never seen that before today (when I loaded the check to grab you the conditions). I’m guessing they added some checks at some point for input sanity, but it seems to still work normally.
Adam might know (or be able to fix it!)
________________________________ Spencer J. Ryan| Manager, Technology and Infrastructure Miller Canfield T +1.313.496.7979 | F +1.313.496.7500 ________________________________
From: Luis Londoño llondono@disti.commailto:llondono@disti.com Sent: Wednesday, November 23, 2022 9:18 AM To: Ryan, Spencer J. Ryan@millercanfield.commailto:Ryan@millercanfield.com; Observium observium@lists.observium.orgmailto:observium@lists.observium.org Cc: Adam Armstrong adama@observium.orgmailto:adama@observium.org Subject: RE: [Observium] Re: {EXT}Re: Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Thank you for the suggestion. Do you get this error by any chance? [cid:image001.png@01D8FF25.B273DC80]
From: Ryan, Spencer J. <Ryan@millercanfield.commailto:Ryan@millercanfield.com> Sent: Wednesday, November 23, 2022 9:13 AM To: Observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Cc: Adam Armstrong <adama@observium.orgmailto:adama@observium.org>; Luis Londoño <llondono@disti.commailto:llondono@disti.com> Subject: RE: [Observium] Re: {EXT}Re: Group Email Alerts
You don't often get email from ryan@millercanfield.commailto:ryan@millercanfield.com. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification 0, or null?
I have two similar checks (one for sensor below and one for above thresholds) but a while ago I had to add null checks to avoid this.
My Sensor High/Low alarms have a delay of 1 and set it so all conditions have to be true:
sensor_value less @sensor_limit_low @sensor_limit_low ne NULL
(Obviously replace low for high on the other check)
That seems to work for us.
________________________________ Spencer J. Ryan| Manager, Technology and Infrastructure Miller Canfield T +1.313.496.7979 | F +1.313.496.7500 ________________________________
From: Luis Londoño via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Sent: Wednesday, November 23, 2022 9:02 AM To: Observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Cc: Adam Armstrong <adama@observium.orgmailto:adama@observium.org>; Luis Londoño <llondono@disti.commailto:llondono@disti.com> Subject: [Observium] Re: {EXT}Re: Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
To give an example of something that happened today and why I ask; One of the servers we are monitoring we have an alert check for the fans being below the sensor threshold something timed out on the connection and caused the sensor values to all drop to 0 and then showed the device timed out as well. We got about 20 emails about the sensors both about the alert and the recovery. I did go ahead and add a alert delay to the checker so that would at least fix that for now but maybe grouping the alerts per host if its happening all at the same time would reduce the noise of the alerts and provide a better way to know that something on that host itself might be an issue.
From: Adam Armstrong via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Sent: Wednesday, November 23, 2022 8:55 AM To: Ryan, Spencer J. via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> Cc: Adam Armstrong <adama@observium.orgmailto:adama@observium.org> Subject: {EXT}[Observium] Re: Group Email Alerts
There's not really a way to "group" alerts. It could potentially be done, but not with our current alert sending code.
If you frequently get situations where you get multiple alerts all at once, being annoyed by those alerts should serve as motivation to fix the problem, no? :D
adam.
Ryan, Spencer J. via observium wrote on 2022-11-23 13:42: How are your alert checkers built?
While we have asked for alert dependencies forever (and may never get it) I don't get any individual alarms for sensors when a device is offline.
Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Androidhttps://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cllondono%40disti.com%7C7d01993ce32f4d93afc908dacd5cd194%7Cd4b51790318d4698a7956c33a8d4b716%7C1%7C0%7C638048095794850730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Y3noLRhM4Vie%2FESKB7IeCpME7owq8ZOWMcbbcrgRXss%3D&reserved=0 ________________________________ From: Luis Londoño via observium observium@lists.observium.orgmailto:observium@lists.observium.org Sent: Wednesday, November 23, 2022 8:31:57 AM To: observium@lists.observium.orgmailto:observium@lists.observium.org observium@lists.observium.orgmailto:observium@lists.observium.org Cc: Luis Londoño llondono@disti.commailto:llondono@disti.com Subject: [Observium] Group Email Alerts
Caution: This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Is there a way to group email alerts so there is not an email alert for every sensor that might go down at the same time? For example multiple fans on a server might show as an alert if the device itself is not responding
You have received a message from the law firm Miller Canfield. The information contained in or attached to this electronic mail may be privileged and/or confidential. If you received this transmission and are not the intended recipient, you should not read this message and are hereby notified that any dissemination, distribution or copying of this communication and/or its attachments is strictly prohibited. If you have received this communication in error or are not sure whether it is privileged, please immediately notify us by return e-mail and delete or destroy the original and any copies, electronic, paper or otherwise, that you may have of this communication and any attachments.
_______________________________________________
observium mailing list -- observium@lists.observium.orgmailto:observium@lists.observium.org
To unsubscribe send an email to observium-leave@lists.observium.orgmailto:observium-leave@lists.observium.org
_______________________________________________
observium mailing list -- observium@lists.observium.orgmailto:observium@lists.observium.org
To unsubscribe send an email to observium-leave@lists.observium.orgmailto:observium-leave@lists.observium.org
-- Sent from Postboxhttps://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach _______________________________________________ observium mailing list -- observium@lists.observium.orgmailto:observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.orgmailto:observium-leave@lists.observium.org
-- Ahmed Rahal Administrateur de Systèmes / Systems Administrator Videotron/Fibrenoire - www.fibrenoire.cahttp://www.fibrenoire.ca/ A: 612 Rue Saint-Jacques, Montréal, QC H3C 1E2 ahmed.rahal@videotron.commailto:ahmed.rahal@videotron.com Twitter: @fibrenoire
_______________________________________________
observium mailing list -- observium@lists.observium.orgmailto:observium@lists.observium.org
To unsubscribe send an email to observium-leave@lists.observium.orgmailto:observium-leave@lists.observium.org
-- Sent from Postboxhttps://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach
Are you sure this didn't just make it so that the alert would /never/ trigger, because you were checking a metric that didn't exist?
adam.
Ryan, Spencer J. via observium wrote on 2022-11-23 15:25:
100%, because at some point in the last year or two, some code changed that would cause the (some) alert code to trigger when the sensor value was NULL, we added this logic and the issues went away.
Someone else posted the solution to this list on 7/9/21 and it worked for us.
Spencer J. Ryan| Manager, Technology and Infrastructure *Miller Canfield*** *T* +1.313.496.7979 | *F* +1.313.496.7500
*From:*Adam Armstrong via observium observium@lists.observium.org *Sent:* Wednesday, November 23, 2022 10:21 AM *To:* Ahmed Rahal via observium observium@lists.observium.org *Cc:* Adam Armstrong adama@observium.org *Subject:* [Observium] Re: {EXT}Re: Group Email Alerts
*Caution:*This is an external email. Do not open attachments or click links from unknown or unexpected emails.
Hmm. Are you sure this has ever worked?
The thresholds shouldn't have ever been present in the metric array. The code the does the check doesn't pass these as metrics, and the code that rewrites the @thresholds on the test side doesn't work on the "metric" side of the comparison.
adam.
Ahmed Rahal via observium wrote on 2022-11-23 14:59:
That's a clever hack ;) Cheers. Le mer. 23 nov. 2022, à 09 h 58, Ryan, Spencer J. via observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> a écrit : Because on random discovery failures the sensor threshold would go to NULL and then start alarming. ------------------------------------------------------------------------ Spencer J. Ryan| Manager, Technology and Infrastructure *Miller Canfield*** *T* +1.313.496.7979 | *F* +1.313.496.7500 ------------------------------------------------------------------------ *From:*Adam Armstrong via observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> *Sent:* Wednesday, November 23, 2022 9:53 AM *To:* Ryan, Spencer J. via observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> *Cc:* Adam Armstrong <adama@observium.org <mailto:adama@observium.org>> *Subject:* [Observium] Re: {EXT}Re: Group Email Alerts *Caution:*This is an external email. Do not open attachments or click links from unknown or unexpected emails. Why are you checking if the threshold is null? You want to check if the value is null :) adam. Ryan, Spencer J. via observium wrote on 2022-11-23 14:20: Funny enough, yes, although I’ve never seen that before today (when I loaded the check to grab you the conditions). I’m guessing they added some checks at some point for input sanity, but it seems to still work normally. Adam might know (or be able to fix it!) ------------------------------------------------------------------------ Spencer J. Ryan| Manager, Technology and Infrastructure *Miller Canfield*** *T* +1.313.496.7979 | *F* +1.313.496.7500 ------------------------------------------------------------------------ *From:*Luis Londoño <llondono@disti.com> <mailto:llondono@disti.com> *Sent:* Wednesday, November 23, 2022 9:18 AM *To:* Ryan, Spencer J. <Ryan@millercanfield.com> <mailto:Ryan@millercanfield.com>; Observium <observium@lists.observium.org> <mailto:observium@lists.observium.org> *Cc:* Adam Armstrong <adama@observium.org> <mailto:adama@observium.org> *Subject:* RE: [Observium] Re: {EXT}Re: Group Email Alerts *Caution:*This is an external email. Do not open attachments or click links from unknown or unexpected emails. Thank you for the suggestion. Do you get this error by any chance? *From:*Ryan, Spencer J. <Ryan@millercanfield.com <mailto:Ryan@millercanfield.com>> *Sent:* Wednesday, November 23, 2022 9:13 AM *To:* Observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> *Cc:* Adam Armstrong <adama@observium.org <mailto:adama@observium.org>>; Luis Londoño <llondono@disti.com <mailto:llondono@disti.com>> *Subject:* RE: [Observium] Re: {EXT}Re: Group Email Alerts You don't often get email from ryan@millercanfield.com <mailto:ryan@millercanfield.com>. Learn why this is important <https://aka.ms/LearnAboutSenderIdentification> 0, or null? I have two similar checks (one for sensor below and one for above thresholds) but a while ago I had to add null checks to avoid this. My Sensor High/Low alarms have a delay of 1 and set it so all conditions have to be true: |sensor_value less @sensor_limit_low| |@sensor_limit_low ne NULL| || |(Obviously replace low for high on the other check)| || || |That seems to work for us. | ------------------------------------------------------------------------ Spencer J. Ryan| Manager, Technology and Infrastructure *Miller Canfield*** *T* +1.313.496.7979 | *F* +1.313.496.7500 ------------------------------------------------------------------------ *From:*Luis Londoño via observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> *Sent:* Wednesday, November 23, 2022 9:02 AM *To:* Observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> *Cc:* Adam Armstrong <adama@observium.org <mailto:adama@observium.org>>; Luis Londoño <llondono@disti.com <mailto:llondono@disti.com>> *Subject:* [Observium] Re: {EXT}Re: Group Email Alerts *Caution:*This is an external email. Do not open attachments or click links from unknown or unexpected emails. To give an example of something that happened today and why I ask; One of the servers we are monitoring we have an alert check for the fans being below the sensor threshold something timed out on the connection and caused the sensor values to all drop to 0 and then showed the device timed out as well. We got about 20 emails about the sensors both about the alert and the recovery. I did go ahead and add a alert delay to the checker so that would at least fix that for now but maybe grouping the alerts per host if its happening all at the same time would reduce the noise of the alerts and provide a better way to know that something on that host itself might be an issue. *From:*Adam Armstrong via observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> *Sent:* Wednesday, November 23, 2022 8:55 AM *To:* Ryan, Spencer J. via observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> *Cc:* Adam Armstrong <adama@observium.org <mailto:adama@observium.org>> *Subject:* {EXT}[Observium] Re: Group Email Alerts There's not really a way to "group" alerts. It could potentially be done, but not with our current alert sending code. If you frequently get situations where you get multiple alerts all at once, being annoyed by those alerts should serve as motivation to fix the problem, no? :D adam. Ryan, Spencer J. via observium wrote on 2022-11-23 13:42: How are your alert checkers built? While we have asked for alert dependencies forever (and may never get it) I don't get any individual alarms for sensors when a device is offline. Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Android <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cllondono%40disti.com%7C7d01993ce32f4d93afc908dacd5cd194%7Cd4b51790318d4698a7956c33a8d4b716%7C1%7C0%7C638048095794850730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Y3noLRhM4Vie%2FESKB7IeCpME7owq8ZOWMcbbcrgRXss%3D&reserved=0> ------------------------------------------------------------------------ *From:* Luis Londoño via observium <observium@lists.observium.org> <mailto:observium@lists.observium.org> *Sent:* Wednesday, November 23, 2022 8:31:57 AM *To:* observium@lists.observium.org <mailto:observium@lists.observium.org> <observium@lists.observium.org> <mailto:observium@lists.observium.org> *Cc:* Luis Londoño <llondono@disti.com> <mailto:llondono@disti.com> *Subject:* [Observium] Group Email Alerts *Caution:*This is an external email. Do not open attachments or click links from unknown or unexpected emails. Is there a way to group email alerts so there is not an email alert for every sensor that might go down at the same time? For example multiple fans on a server might show as an alert if the device itself is not responding You have received a message from the law firm Miller Canfield. The information contained in or attached to this electronic mail may be privileged and/or confidential. If you received this transmission and are not the intended recipient, you should not read this message and are hereby notified that any dissemination, distribution or copying of this communication and/or its attachments is strictly prohibited. If you have received this communication in error or are not sure whether it is privileged, please immediately notify us by return e-mail and delete or destroy the original and any copies, electronic, paper or otherwise, that you may have of this communication and any attachments. _______________________________________________ observium mailing list --observium@lists.observium.org <mailto:observium@lists.observium.org> To unsubscribe send an email toobservium-leave@lists.observium.org <mailto:observium-leave@lists.observium.org> -- Sent from Postbox <https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fwww.postbox-inc.com%2F%3Futm_source%3Demail%26utm_medium%3Dsiglink%26utm_campaign%3Dreach&data=05%7C01%7Cllondono%40disti.com%7C7d01993ce32f4d93afc908dacd5cd194%7Cd4b51790318d4698a7956c33a8d4b716%7C1%7C0%7C638048095794850730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=givrKgpt7EmK32gCgqqobO4%2FktvYCEM%2FI%2FcqMQDOcpc%3D&reserved=0> _______________________________________________ observium mailing list --observium@lists.observium.org <mailto:observium@lists.observium.org> To unsubscribe send an email toobservium-leave@lists.observium.org <mailto:observium-leave@lists.observium.org> -- Sent from Postbox <https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach> _______________________________________________ observium mailing list -- observium@lists.observium.org <mailto:observium@lists.observium.org> To unsubscribe send an email to observium-leave@lists.observium.org <mailto:observium-leave@lists.observium.org> -- Ahmed Rahal Administrateur de Systèmes / Systems Administrator *Videotron**/Fibre**noire* - www.fibrenoire.ca <http://www.fibrenoire.ca/> A: 612 Rue Saint-Jacques, Montréal, QC H3C 1E2 ahmed.rahal@videotron.com <mailto:ahmed.rahal@videotron.com> Twitter: @fibrenoire _______________________________________________ observium mailing list --observium@lists.observium.org <mailto:observium@lists.observium.org> To unsubscribe send an email toobservium-leave@lists.observium.org <mailto:observium-leave@lists.observium.org>--
Sent from Postbox https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach
observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
participants (4)
-
Adam Armstrong -
Ahmed Rahal -
Luis Londoño -
Ryan, Spencer J.