From my colleagues:
What we would REALLY like is the ability to turn off alerts on a particular device for a specified amount of time. For example, when Winter Park goes flakey but it is really Comcast or Centurylink that need to fix the problem, I don't want alerts every 10 minutes to tell me the link still sucks.
There are several possible solutions to this. One is a manual "suppress alerts for X hours". It is important that the alerts be turned back on after the specified time because otherwise we tend to forget to turn it back on.
Another option is a rate limit on alerts by device. After 5 alerts I probably figured out the link sucks.
A third option is a "delay on recover". Right now we get a recover as soon as one check passes. If it is flakey we may want to have more checks pass before declaring it recovered. This would probably be something we want to do by device, because from most I want to know as soon as it is back.