Hmm...
Originally the plan was to try to implement alert coalescing so that multiple alerts would be sent in a single email, but that isn't as easy when you don't only use email.
I could potentially add *device* dependencies, which might be easier to implement and handle.
adam. On 2019-02-01 00:23:22, Joey Stanford via observium observium@observium.org wrote:
In reality we don't actually know if anything is up or down, we only know that something /was/ up or down at some point in the past.
At least in our situation, there might a programatic solution that could work.
1) set a delay to the alert (we have this already as part of the checker)
2) set a variable for the number of pending (concurrent) alerts. e.g. 10
3) If number of pending alerts > threshold send alert storm alert and purge alert queue
Something along those lines would probably work to cut down the spam at least in our case.
ps one of my guys said "We need to get Joey to program Obs so that when we get this many alerts it plays a tune😁” I told them that tune would like be an air raid siren.