Hello Adam,
Tuesday, June 3, 2014, 5:21:00 AM, you wrote:
Any suggestions on how to maintain state for this kind of thing in a generic way would be helpful :)
If I understand things right: Currently there's an assumption, that Observium monitors devices and device MUST be an SNMP capable device and respond to pings.
I'm coming from The Dude here and I think their approach to things was better. They had objects (devices, links, subnets) and devices had services (Ping, SNMP, whatever). All that device required to have is a unique name. It didn't even need an IP address.
I believe current state of things can be migrated in this direction. Have devices that are empty husks, that can contain entities like Ports, Services (either detected via SNMP, or added manualy) - this way you could add server.domain.com, that's SNMP capable - SNMP probing would add entities: Network Interfaces (Ports) Sensors Ping
You could then manually add more entities, like for example scripts (local, running on observium box), that returned either boolean result, a number, or whatever. Or custom OIDs. You could graph the results etc. This way you also drop specific code for temp/voltage sensors, since they are just another entity.
In case of SNMP-incapable host, probing would add only Ping entity.
In summary, you no longer monitor devices, you monitor services on them (or entities, or whatever you decide to call them). Less core assumptions, more freedom to users. I could add few Internet hosts just to monitor their availability and graph ping times, and add my Eltek Minipacks and monitor their voltage and battery state with custom OIDs.
This would maybe open the possibility to write some custom pollers that poll only specific entities/services/whatever; for example some pingd, that only concerns with pinging hosts and does it every second, speeding reaction times, creating more accurate results, monitoring packet loss, etc.
Actually I strongly suggest installing Mikrotik's The Dude and fiddling with it a bit -- lots of ideas to look at. It's really great piece of monitoring software, appreciated by many and really hard to migrate from due to nice drag & drop map and astonishing flexibility (especially considering who created it ;>)
Hope this helps you in some way, or gets you thinking at least :)