Hi, Is it possible to aggregate alert emails? When a switch does fail, it might generate thousands of alerts. For our case maybe even thousands of txt messages to a pager. Is it possible to configure observium to detect such kind of massive alerts and aggregate them to only send a few alerts out?
Thanks.
—Zhenhui
It isn't, and it won't.
Alerts are only generated when things are polled. If a device goes down, only a device status alert is generated.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 17 December 2014 15:56:01 Zhenhui Liang ZhenhuiLiang@eharmony.com wrote:
Hi, Is it possible to aggregate alert emails? When a switch does fail, it might generate thousands of alerts. For our case maybe even thousands of txt messages to a pager. Is it possible to configure observium to detect such kind of massive alerts and aggregate them to only send a few alerts out?
Thanks.
—Zhenhui _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
My take on this would be to write my own alert tool to summarize alert status from the database. I have not looked into this in depth yet, but it is how I have handled similar issues with other tools.
Regards, Mark
Sent from my iPad
On Dec 17, 2014, at 2:01 PM, Adam Armstrong adama@memetic.org wrote:
It isn't, and it won't.
Alerts are only generated when things are polled. If a device goes down, only a device status alert is generated.
Adam.
Sent with AquaMail for Android https://canit.willingminds.com/canit/urlproxy.php?q=aHR0cDovL3d3dy5hcXVhLW1h...
On 17 December 2014 15:56:01 Zhenhui Liang ZhenhuiLiang@eharmony.com wrote:
Hi, Is it possible to aggregate alert emails? When a switch does fail, it might generate thousands of alerts. For our case maybe even thousands of txt messages to a pager. Is it possible to configure observium to detect such kind of massive alerts and aggregate them to only send a few alerts out?
Thanks.
—Zhenhui _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
This message has been scanned by CanIt-PRO.
-- BEGIN-ANTISPAM-VOTING-LINKS
Teach CanIt if this mail (ID 03NsK2cmG) is spam: Spam: https://canit.willingminds.com/canit/b.php?i=03NsK2cmG&m=7f1bfbbf5a96&am... Not spam: https://canit.willingminds.com/canit/b.php?i=03NsK2cmG&m=7f1bfbbf5a96&am... Forget vote: https://canit.willingminds.com/canit/b.php?i=03NsK2cmG&m=7f1bfbbf5a96&am...
END-ANTISPAM-VOTING-LINKS
Say my alert condition is ifAdmin up ifOper down. Observium send out alerts on both conditions true.
If a device is down, it receives no answer. I guess my question is, if it get no answer for configured conditions, would it alert?
Seems your answer is no it won¹t alert. If that¹s the case, then we are good.
On 12/17/14, 2:01 PM, "Adam Armstrong" adama@memetic.org wrote:
It isn't, and it won't.
Alerts are only generated when things are polled. If a device goes down, only a device status alert is generated.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 17 December 2014 15:56:01 Zhenhui Liang ZhenhuiLiang@eharmony.com wrote:
Hi, Is it possible to aggregate alert emails? When a switch does fail, it might generate thousands of alerts. For our case maybe even thousands of txt messages to a pager. Is it possible to configure observium to detect such kind of massive alerts and aggregate them to only send a few alerts out?
Thanks.
‹Zhenhui _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
No, it won't, because those poller modules would never be executed.
adam.
------ Original Message ------ From: "Zhenhui Liang" ZhenhuiLiang@eharmony.com To: "Observium Network Observation System" observium@observium.org Sent: 12/17/2014 4:16:35 PM Subject: Re: [Observium] aggregate alert emails?
Say my alert condition is ifAdmin up ifOper down. Observium send out alerts on both conditions true.
If a device is down, it receives no answer. I guess my question is, if it get no answer for configured conditions, would it alert?
Seems your answer is no it won¹t alert. If that¹s the case, then we are good.
On 12/17/14, 2:01 PM, "Adam Armstrong" adama@memetic.org wrote:
It isn't, and it won't.
Alerts are only generated when things are polled. If a device goes down, only a device status alert is generated.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 17 December 2014 15:56:01 Zhenhui Liang ZhenhuiLiang@eharmony.com wrote:
Hi, Is it possible to aggregate alert emails? When a switch does fail, it might generate thousands of alerts. For our case maybe even thousands of txt messages to a pager. Is it possible to configure observium to detect such kind of massive alerts and aggregate them to only send a few alerts out?
Thanks.
‹Zhenhui _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Yup, it only gets to alert when it's polling the entity configured in your alert checker, but if the device is not polled (due to being down) it will not run the checkers and as such not alert.
Tom
On 17/12/2014 23:16, Zhenhui Liang wrote:
Say my alert condition is ifAdmin up ifOper down. Observium send out alerts on both conditions true.
If a device is down, it receives no answer. I guess my question is, if it get no answer for configured conditions, would it alert?
Seems your answer is no it won¹t alert. If that¹s the case, then we are good.
On 12/17/14, 2:01 PM, "Adam Armstrong" adama@memetic.org wrote:
It isn't, and it won't.
Alerts are only generated when things are polled. If a device goes down, only a device status alert is generated.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 17 December 2014 15:56:01 Zhenhui Liang ZhenhuiLiang@eharmony.com wrote:
Hi, Is it possible to aggregate alert emails? When a switch does fail, it might generate thousands of alerts. For our case maybe even thousands of txt messages to a pager. Is it possible to configure observium to detect such kind of massive alerts and aggregate them to only send a few alerts out?
Thanks.
‹Zhenhui _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Device polling supersedes entity polling. I’ve had multiple device down alarms with no trailing port or other entity alarms associated with that device.
However, if you have a bunch of devices connected to “switch-a” and it goes down, you’re going to get alerts from all of those downstream device ports, assuming you have a port up/down alert configured. Maybe that’s what the OP was referring to?
Some large (arguably, bloated) commercial NMS tools will map inter-device connections and silence alarms for downstream ports connected to a device that went down.
On 12/17/14, 3:17 PM, "Tom Laermans" tom.laermans@powersource.cx wrote:
Yup, it only gets to alert when it's polling the entity configured in your alert checker, but if the device is not polled (due to being down) it will not run the checkers and as such not alert.
Tom
On 17/12/2014 23:16, Zhenhui Liang wrote:
Say my alert condition is ifAdmin up ifOper down. Observium send out alerts on both conditions true.
If a device is down, it receives no answer. I guess my question is, if it get no answer for configured conditions, would it alert?
Seems your answer is no it won¹t alert. If that¹s the case, then we are good.
On 12/17/14, 2:01 PM, "Adam Armstrong" adama@memetic.org wrote:
It isn't, and it won't.
Alerts are only generated when things are polled. If a device goes down, only a device status alert is generated.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 17 December 2014 15:56:01 Zhenhui Liang ZhenhuiLiang@eharmony.com wrote:
Hi, Is it possible to aggregate alert emails? When a switch does fail, it might generate thousands of alerts. For our case maybe even thousands of txt messages to a pager. Is it possible to configure observium to detect such kind of massive alerts and aggregate them to only send a few alerts out?
Thanks.
‹Zhenhui _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Founded in 2007, IO provides the data center as a service to businesses and governments around the world.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.
Who cares, really? It's a whole metric fucktonne of work for a couple of extra emails a few times a year when a device breaks.
It'd involve some pretty big architectural changes, and I'm not even sure we could do it properly without introducing even greater delays.
Those other applications probably aren't interpreted PHP without any persistent in-memory state. It's way easier to do that kind of thing if you always know the state of everything at every point in the code without having to drag the database.
For example
observium === switch === router
If switch goes down, router is also unreachable. Router is polled and found to be down.
When do we alert for router? After we polled switch? What if switch was actually up 15 seconds ago when it was polled, do we wait 5 minutes? Where do we make that decision?
adam.
------ Original Message ------ From: "Pedersen, Sean" spedersen@io.com To: "Observium Network Observation System" observium@observium.org Sent: 12/18/2014 7:03:29 AM Subject: Re: [Observium] aggregate alert emails?
Device polling supersedes entity polling. I’ve had multiple device down alarms with no trailing port or other entity alarms associated with that device.
However, if you have a bunch of devices connected to “switch-a” and it goes down, you’re going to get alerts from all of those downstream device ports, assuming you have a port up/down alert configured. Maybe that’s what the OP was referring to?
Some large (arguably, bloated) commercial NMS tools will map inter-device connections and silence alarms for downstream ports connected to a device that went down.
On 12/17/14, 3:17 PM, "Tom Laermans" tom.laermans@powersource.cx wrote:
Yup, it only gets to alert when it's polling the entity configured in your alert checker, but if the device is not polled (due to being down) it will not run the checkers and as such not alert.
Tom
On 17/12/2014 23:16, Zhenhui Liang wrote:
Say my alert condition is ifAdmin up ifOper down. Observium send out alerts on both conditions true.
If a device is down, it receives no answer. I guess my question is, if it get no answer for configured conditions, would it alert?
Seems your answer is no it won¹t alert. If that¹s the case, then we are good.
On 12/17/14, 2:01 PM, "Adam Armstrong" adama@memetic.org wrote:
It isn't, and it won't.
Alerts are only generated when things are polled. If a device goes down, only a device status alert is generated.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 17 December 2014 15:56:01 Zhenhui Liang ZhenhuiLiang@eharmony.com wrote:
Hi, Is it possible to aggregate alert emails? When a switch does fail, it might generate thousands of alerts. For our case maybe even thousands of txt messages to a pager. Is it possible to configure observium to detect such kind of massive alerts and aggregate them to only send a few alerts out?
Thanks.
‹Zhenhui _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Founded in 2007, IO provides the data center as a service to businesses and governments around the world.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
I've asked about that before since I have hundreds of remote firewalls with our systems behind them at customer locations. Firewall/VPN tunnel/etc. drops, and I get 50 alerts. I was given the response that it was too big a change.
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Pedersen, Sean Sent: Thursday, December 18, 2014 7:03 AM To: Observium Network Observation System Subject: Re: [Observium] aggregate alert emails?
Device polling supersedes entity polling. I’ve had multiple device down alarms with no trailing port or other entity alarms associated with that device.
However, if you have a bunch of devices connected to “switch-a” and it goes down, you’re going to get alerts from all of those downstream device ports, assuming you have a port up/down alert configured. Maybe that’s what the OP was referring to?
Some large (arguably, bloated) commercial NMS tools will map inter-device connections and silence alarms for downstream ports connected to a device that went down.
On 12/17/14, 3:17 PM, "Tom Laermans" tom.laermans@powersource.cx wrote:
Yup, it only gets to alert when it's polling the entity configured in your alert checker, but if the device is not polled (due to being down) it will not run the checkers and as such not alert.
Tom
On 17/12/2014 23:16, Zhenhui Liang wrote:
Say my alert condition is ifAdmin up ifOper down. Observium send out alerts on both conditions true.
If a device is down, it receives no answer. I guess my question is, if it get no answer for configured conditions, would it alert?
Seems your answer is no it won¹t alert. If that¹s the case, then we are good.
On 12/17/14, 2:01 PM, "Adam Armstrong" adama@memetic.org wrote:
It isn't, and it won't.
Alerts are only generated when things are polled. If a device goes down, only a device status alert is generated.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 17 December 2014 15:56:01 Zhenhui Liang ZhenhuiLiang@eharmony.com wrote:
Hi, Is it possible to aggregate alert emails? When a switch does fail, it might generate thousands of alerts. For our case maybe even thousands of txt messages to a pager. Is it possible to configure observium to detect such kind of massive alerts and aggregate them to only send a few alerts out?
Thanks.
‹Zhenhui _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Founded in 2007, IO provides the data center as a service to businesses and governments around the world.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
________________________________
This e-mail and any documents accompanying it may contain legally privileged and/or confidential information belonging to Exegy, Inc. Such information may be protected from disclosure by law. The information is intended for use by only the addressee. If you are not the intended recipient, you are hereby notified that any disclosure or use of the information is strictly prohibited. If you have received this e-mail in error, please immediately contact the sender by e-mail or phone regarding instructions for return or destruction and do not use or disclose the content to others.
Yeah. Because we check and send alerts per-device, its pretty hard for us to have aggregate emails including multiple devices.
Aggregate emails including multiple alerts from the same device is significantly easier, since all alerts for one host are sent by the same script execution.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 22 December 2014 11:00:39 "Hamilton, Kent" KHamilton@exegy.com wrote:
I've asked about that before since I have hundreds of remote firewalls with our systems behind them at customer locations. Firewall/VPN tunnel/etc. drops, and I get 50 alerts. I was given the response that it was too big a change.
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Pedersen, Sean Sent: Thursday, December 18, 2014 7:03 AM To: Observium Network Observation System Subject: Re: [Observium] aggregate alert emails?
Device polling supersedes entity polling. I’ve had multiple device down alarms with no trailing port or other entity alarms associated with that device.
However, if you have a bunch of devices connected to “switch-a” and it goes down, you’re going to get alerts from all of those downstream device ports, assuming you have a port up/down alert configured. Maybe that’s what the OP was referring to?
Some large (arguably, bloated) commercial NMS tools will map inter-device connections and silence alarms for downstream ports connected to a device that went down.
On 12/17/14, 3:17 PM, "Tom Laermans" tom.laermans@powersource.cx wrote:
Yup, it only gets to alert when it's polling the entity configured in your alert checker, but if the device is not polled (due to being down) it will not run the checkers and as such not alert.
Tom
On 17/12/2014 23:16, Zhenhui Liang wrote:
Say my alert condition is ifAdmin up ifOper down. Observium send out alerts on both conditions true.
If a device is down, it receives no answer. I guess my question is, if it get no answer for configured conditions, would it alert?
Seems your answer is no it won¹t alert. If that¹s the case, then we are good.
On 12/17/14, 2:01 PM, "Adam Armstrong" adama@memetic.org wrote:
It isn't, and it won't.
Alerts are only generated when things are polled. If a device goes down, only a device status alert is generated.
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 17 December 2014 15:56:01 Zhenhui Liang ZhenhuiLiang@eharmony.com wrote:
Hi, Is it possible to aggregate alert emails? When a switch does fail, it might generate thousands of alerts. For our case maybe even thousands of txt messages to a pager. Is it possible to configure observium to detect such kind of massive alerts and aggregate them to only send a few alerts out?
Thanks.
‹Zhenhui _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Founded in 2007, IO provides the data center as a service to businesses and governments around the world.
The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
This e-mail and any documents accompanying it may contain legally privileged and/or confidential information belonging to Exegy, Inc. Such information may be protected from disclosure by law. The information is intended for use by only the addressee. If you are not the intended recipient, you are hereby notified that any disclosure or use of the information is strictly prohibited. If you have received this e-mail in error, please immediately contact the sender by e-mail or phone regarding instructions for return or destruction and do not use or disclose the content to others. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (6)
-
Adam Armstrong -
Hamilton, Kent -
Mark Nagel -
Pedersen, Sean -
Tom Laermans -
Zhenhui Liang