- observium - lists.observium.org

Observium SNMP polling crashes various Adtran devices
by John Brown 17 Dec '14

17 Dec '14

Hi, We recently installed Observium in an effort to replace our aging Cacti and Nagios systems. We are running the latest code checked out of SVN (We paid the money to Adam for the Professional Version) Today we added 180+ Adtran based devices to the system. All 180+ devices began rebooting. They would become stable for a few hours and then start rebooting again. Important to note that most of these items have been in our network for 3+ years, are running recent code, and have been rock solid stable for their entire service life. Cacti and Nagios poll them every minute and have been doing so for the past 3 years. Insert Observium and the devices start failing. My initial feeling is that there is something in the way Observium does its SNMP. Curious if anyone else has experienced this or something like this. Its caused us to halt putting more devices on Observium. Devices are TA 5000 / TA 352 GPON ONT's, TA 9xx iad's When I add a single device via add_device <name> and then do a discover.php -h new as soon as the code hits ENTITY-SENSOR-MIB the adtran device reboots noc@observium:/home/observium# ./discovery.php -h new Observium 0.14.12.6107 Discovery ont-03.sip.pon-1-6-1.fttx.XXX.YYYYY.net 38 adtran-aos (adtran-aos) Module [ os ] time: 0.0005s Ports : .... Module [ ports ] time: 0.1889s Port Stacks: Module [ ports-stack ] time: 0.0774s Processors : ADTRAN-AOSCPU + hrDevice: Module [ processors ] time: 0.2734s Memory : ADTRAN-AOSCPU + Module [ mempools ] time: 0.2019s IPv4 Addresses : .. Module [ ipv4-addresses ] time: 0.1366s IPv6 Addresses : Module [ ipv6-addresses ] time: 0.5068s Sensors: ENTITY-SENSOR-MIB

4 3

Re: [Observium] junos logical sub interface state
by Zhenhui Liang 17 Dec '14

17 Dec '14

Ok, I added ifDescr !match *.* In the condition, seems working. Although I don¹t feel that clean about it. I would like to only care about physical interface, not logical. But I tried all snmp mibs in the mib list, none of them I found tell you it¹s a logical interface. So I rely on the pattern *dot* in junos to indicate a logical port. On 12/16/14, 1:18 PM, "Zhenhui Liang" <ZhenhuiLiang(a)eharmony.com> wrote: >Hi, > >I shutdown all unused ports to make the alerter life easier. But after I >done so, all good except ae ports sub interfaces still shows on observium >as failed even though the main ae physical ports already administratively >down. > >For instace, ae11: >show interfaces ae11 >Physical interface: ae11, Administratively down, Physical link is Down > >But observium still complain ae11.0 failed check: >show interfaces ae11.0 > Logical interface ae11.0 (Index 153) (SNMP ifIndex 982) > > >How do we ignore logical interfaces when setting up alerter? >My current condition is: > >ifAdminStatus equals up >ifOperStatus equals down > > >Thank you. > > ‹Zhenhui >_______________________________________________ >observium mailing list >observium(a)observium.org >http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

2 5

Example alert checkers, and background information
by Moerman, Maarten 16 Dec '14

16 Dec '14

Hey Guys, I had some spare time, and decided to write up some more documentation on alert-checkers for observium, and provide a whole bunch of example alerts which could be adapted for your use: Document: http://www.maartenmoerman.nl/?p=612 I have provided examples for: - Device down - Processor usage is above 80% - Memory usage is above 70% - State sensor is in ALERT state! - Fanspeed is above or under treshold - Traffic exceeds 85% - BGP Session down - Storage exceeds 85% of disk capacity - Port has encountered errors or discards - Port is enabled, but operationally down Let me know if this is helpful and what you'd like to see added/improved. Thanks, Maarten -- Maarten Moerman | Mgr, Network Engineering | eBay Classifieds | +31-655122247 | mmoerman(a)ebay.com

4 4

Re: [Observium] Sort ports based on total data usage
by Adam Armstrong 16 Dec '14

16 Dec '14

Hmm. That's a lot of I/O time. Seems it's unfeasible to do at the same frequency as the poller. Perhaps it's something we can do daily, but I'm not sure if it's really useful enough to do. adam. ------ Original Message ------ From: "Pedersen, Sean" <spedersen(a)io.com> To: "Adam Armstrong" <adama(a)memetic.org> Sent: 12/15/2014 7:39:34 AM Subject: Re: [Observium] Sort ports based on total data usage >real1m45.617s >user0m1.800s >sys0m3.724s > >20320 > >About 200 devices, and it’s a VM. > >From: Adam Armstrong <adama(a)memetic.org> >Reply-To: Adam Armstrong <adama(a)memetic.org>, Observium Network >Observation System <observium(a)observium.org> >Date: Saturday, December 13, 2014 at 11:41 PM >To: Observium Network Observation System <observium(a)observium.org> >Subject: Re: [Observium] Sort ports based on total data usage > >So. > >I ran a test to see how quickly we could extract the "traffic in the >last 24 hours" statistic from rrdtool for all ports on my development >install : > >root@omega:/mnt/ramdisk/observium_dev# time for file in `ls >*/port-*.rrd | grep -v adsl | grep -v dot3 | xargs`; do rrdtool graphv >/dev/null DEF:outoctets=$file:OUTOCTETS:AVERAGE >DEF:inoctets=$file:INOCTETS:AVERAGE CDEF:octets=inoctets,outoctets,+ >VDEF:totin=inoctets,TOTAL VDEF:totout=outoctets,TOTAL >PRINT:totin:"%4.2lf" PRINT:totout:"%4.2lf"; done > >real 0m10.953s >user 0m3.526s >sys 0m7.670s > > >root@omega:/mnt/ramdisk/observium_dev# ls */port-*.rrd | grep -v adsl | >grep -v dot3 | wc -l >3327 > > >I'm running from a ramdisk though, so it would be nice if some other >people with large non-ramdisk installs could see how long this takes :) > >adam. > > >------ Original Message ------ >From: "Michael Loftis" <mloftis(a)wgops.com> >To: "Adam Armstrong" <adama(a)memetic.org>; "Observium Network >Observation System" <observium(a)observium.org> >Sent: 12/13/2014 11:20:19 PM >Subject: Re: [Observium] Sort ports based on total data usage > >>I'll second what Adam is saying - whats in the DB is not anywhere >>remotely useful like you think. Maybe sort of on a single device but >>even then dubious at best. Your option is really to have to total >>every RRD file over the timeframe wished in order to get a result, >>which isn't going to be very fast. My biggest Observium installation >>is (IMO) very small and is still ~30G of RRD data. You'd end up >>scanning a good portion of that 30G to come up with a sort >>order/totalling. One of the big switches is ~4G by itself, loading >>it's ports page is.....ugly pulling in all the mini graphs, but at >>least the basic view is up fast. Loading all the mini graphs after >>that...I'm not sure how long that takes, I've never waited around for >>it to finish, at least a couple minutes I'd imagine. It'd be around >>that timeframe to produce a sorted list and spit out the main view. >>Granted that switch is probably pretty close to a worst case in terms >>of port count. >> >> >>On Sat, Dec 13, 2014 at 7:04 PM, Adam Armstrong <adama(a)memetic.org> >>wrote: >>>That number is in the database to calculate rates. It's functionally >>>useless unless you know when the counter was last reset. >>> >>>It isn't exposed in the UI precisely because it's such a functionally >>>useless number. >>> >>>What /might/ be vaguely useful would be calculating an average >>>traffic rate for the port by tracking the "last reset" time for the >>>counter. But even then, this is only really useful if the port has >>>*always* been active for the uptime of the switch. It would take >>>literally weeks for a port flatlining 1GB to overtake a port which >>>had been pushing 100mbit for a few months. >>> >>>There are other possibilities, such as storing a series of values at >>>various timepoints so we can "sort of" give a value like "xbps over >>>the past 24 hours", but these get terribly inefficient and slow, >>>especially considering that we're using MEMORY tables to hold this >>>data, and pretty irritating things happen when those tables fill up. >>> >>>I don't think exposing a value which is only comparable with other >>>ports on the same device active for the same amount of time which >>>haven't had their counters reset is particularly useful (and seems >>>particularly confusing). >>> >>>adam. >>> >>> >>>------ Original Message ------ >>>From: "Justin Miller" <observium(a)dynstatic.net> >>>To: "Observium Network Observation System" <observium(a)observium.org> >>>Sent: 12/12/2014 1:21:55 PM >>>Subject: Re: [Observium] Sort ports based on total data usage >>> >>>>In the db observium.ports-state, there is ifInOctlets, >>>>ifInOctlets_delta, and ifInOctets_rate. It’s also there for >>>>ifOutOctlets, ifOutOctlets_delta, and ifOutOctets_rate. So the >>>>values I want are already in the database. >>>> >>>>Furthermore, when the html/pages/devie/port.inc.php invokes >>>>includes/print-interface.inc.php to print each interface, $port >>>>contains those values. >>>> >>>>Hacking it into the view >>>> >>>> '>>>style="',$port['bps_in_style'], '"> >>>style="',$port['bps_in_style'], '">' , >>>>formatStorage($port['ifInOctets']) , ' ', >>>> '>>>style="',$port['bps_out_style'],'"> >>>style="',$port['bps_out_style'],'">' , >>>>formatStorage($port['ifOutOctets']) , ' ', >>>> >>>>Produces somewhat correct results which are good enough for what I >>>>need. I just need to sort it now. Maybe make a new page. >>>> >>>> >>>> >>>> >>>>>On Dec 12, 2014, at 12:19 PM, Tom Laermans >>>>><tom.laermans(a)powersource.cx> wrote: >>>>> >>>>>Nope. >>>>> >>>>>On 12/12/2014 06:11 PM, Justin Miller wrote: >>>>>>Because it’s a round robin database and it has to sum the octlets >>>>>>for that given time, right? >>>>>> >>>>>>Doesn’t the poller read how many packets have been send or >>>>>>received on every port anyways? ifOutOclets / ifInOclets ? Isn’t >>>>>>that a running total? Then to get the rate you take the difference >>>>>>between the current poll and the last poll and divide it by the >>>>>>time between polls? >>>>>> >>>>>>There has to be an OID for this which would be easy to store at >>>>>>poll time, right? >>>>>> >>>>>>Justin >>>>>> >>>>>> >>>>>>>On Dec 12, 2014, at 10:50 AM, Tom Laermans >>>>>>><tom.laermans(a)powersource.cx> wrote: >>>>>>> >>>>>>>It will take about 15 minutes to generate that page, then. >>>>>>> >>>>>>>On 12/12/2014 04:14 PM, Justin Miller wrote: >>>>>>>>This information is already in the RRD file is it not? Can’t you >>>>>>>>export the numerical value for the TOTAL from the graph for each >>>>>>>>port and then sort based on that? >>>>>>>> >>>>>>>>It does not have to be 100% accurate, I’m not billing for it. I >>>>>>>>just need to see what the top 10 busiest ports are on a switch. >>>>>>>> >>>>>>>> >>>>>>>><Mail Attachment.png> >>>>>>>> >>>>>>>> >>>>>>>>>On Dec 12, 2014, at 2:24 AM, Adam Armstrong <adama(a)memetic.org> >>>>>>>>>wrote: >>>>>>>>> >>>>>>>>>Total in what time frame? Day, week, month, year, decade? >>>>>>>>>Whence would this information come? >>>>>>>>> >>>>>>>>>No, doing this isn't really realistically possible to do in a >>>>>>>>>non-slow way (or useful) >>>>>>>>> >>>>>>>>>adam. >>>>>>>>> >>>>>>>>>------ Original Message ------ >>>>>>>>>From: "Justin Miller" <observium(a)dynstatic.net> >>>>>>>>>To: "Observium Network Observation System" >>>>>>>>><observium(a)observium.org> >>>>>>>>>Sent: 12/11/2014 12:17:56 PM >>>>>>>>>Subject: Re: [Observium] Sort ports based on total data usage >>>>>>>>> >>>>>>>>>>That appears to be instantaneous amount when the poller >>>>>>>>>>collected it. I need how many bytes total. Not how many bits >>>>>>>>>>per second. >>>>>>>>>> >>>>>>>>>>I need to see that this port downloaded 3.4 TB of data, not >>>>>>>>>>that it’s doing 7 megabits per second. >>>>>>>>>> >>>>>>>>>>Justin >>>>>>>>>> >>>>>>>>>>>On Dec 11, 2014, at 1:12 PM, Spencer Gaw <spencerg(a)frii.net> >>>>>>>>>>>wrote: >>>>>>>>>>> >>>>>>>>>>>Click on the Traffic column on the ports page. >>>>>>>>>>> >>>>>>>>>>>Regards, >>>>>>>>>>> >>>>>>>>>>>SG >>>>>>>>>>> >>>>>>>>>>>On 12/11/2014 11:08 AM, Justin Miller wrote: >>>>>>>>>>>>We need to view the top X ports on switches which have used >>>>>>>>>>>>the most traffic (in & out) for a given time. I haven't been >>>>>>>>>>>>able to find this view. Is there a way to see this or will >>>>>>>>>>>>we have to create a special page to view this? >>>>>>>>>>>> >>>>>>>>>>>>Justin >>>>>>>>>>>>_______________________________________________ >>>>>>>>>>>>observium mailing list >>>>>>>>>>>>observium(a)observium.org >>>>>>>>>>>>http://postman.memetic.org/cgi-bin/mailman/listinfo/observium >>>>>>>>>>> >>>>>>>>>>>_______________________________________________ >>>>>>>>>>>observium mailing list >>>>>>>>>>>observium(a)observium.org >>>>>>>>>>>http://postman.memetic.org/cgi-bin/mailman/listinfo/observium >>>>>>>>>> >>>>>>>>>_______________________________________________ >>>>>>>>>observium mailing list >>>>>>>>>observium(a)observium.org >>>>>>>>>http://postman.memetic.org/cgi-bin/mailman/listinfo/observium >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>_______________________________________________ observium >>>>>>>>mailing list >>>>>>>>observium@observium.orghttp://postman.memetic.org/cgi-bin/mailman/listinfo/observium >>>>>>> >>>>>>>_______________________________________________ >>>>>>>observium mailing list >>>>>>>observium(a)observium.org >>>>>>>http://postman.memetic.org/cgi-bin/mailman/listinfo/observium >>>>>> >>>>>> >>>>>> >>>>>>_______________________________________________ observium mailing >>>>>>list >>>>>>observium@observium.orghttp://postman.memetic.org/cgi-bin/mailman/listinfo/observium >>>>> >>>>>_______________________________________________ >>>>>observium mailing list >>>>>observium(a)observium.org >>>>>http://postman.memetic.org/cgi-bin/mailman/listinfo/observium >>>> >>> >>>_______________________________________________ >>>observium mailing list >>>observium(a)observium.org >>>http://postman.memetic.org/cgi-bin/mailman/listinfo/observium >>> >> >> >>-- >> >>"Genius might be described as a supreme capacity for getting its >>possessors >>into trouble of all kinds." >>-- Samuel Butler > > >Founded in 2007, IO provides the data center as a service to businesses >and governments around the world. > >The communication contained in this e-mail is confidential and is >intended only for the named recipient(s) and may contain information >that is privileged, proprietary, attorney work product or exempt from >disclosure under applicable law. If you have received this message in >error, or are not the named recipient(s), please note that any form of >distribution, copying or use of this communication or the information >in it is strictly prohibited and may be unlawful. Please immediately >notify the sender of the error, and delete this communication including >any attached files from your system. Thank you for your cooperation.

3 2

TEXT/BLOB fields in MySQL
by Eduardo Schoedler 16 Dec '14

16 Dec '14

Hi, Any chance to change text/blob fields to another type in MySQL? # ./tuning-primer.sh TEMP TABLES Current max_heap_table_size = 512 M Current tmp_table_size = 512 M Of 17956 temp tables, 29% were created on disk Perhaps you should increase your tmp_table_size and/or max_heap_table_size to reduce the number of disk-based temporary tables *Note! BLOB and TEXT columns are not allow in memory tables.If you are using these columns raising these values might not impact yourratio of on disk temp tables.* Thanks. -- Eduardo Schoedler

3 4

Re: [Observium] Billing per VM host
by Ron Marosko, Jr. 16 Dec '14

16 Dec '14

I think it does. ;-) Meaning, yes, it graphs the 1000v just like it were any other switch. Thus, make sure you have all your "description" entries defined in the interfaces before you discover the device. And the vEth port does not change when the VM gets vmotioned from one host to another, so stats are consistent. Sent from my Verizon Wireless 4G LTE DROID Markus Klock <markus(a)best-practice.se> wrote: >_______________________________________________ >observium mailing list >observium(a)observium.org >http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

1 0

Re: [Observium] Billing per VM host
by Ron Marosko, Jr. 16 Dec '14

16 Dec '14

I'm running the Cisco freemium version of the Nexus 1000v virtual switch on my ESXi 4.1 boxen, mainly because I needed LACP uplink capability, which was not available in the vSwitch. Then I discovered that you can get SNMP stats on a per-vEthernet port (meaning per host!) from the 1000v. Sweet! The poller takes a bit longer to run on the 1000v, just like it does on all other Nexus products, but well worth having the information. And yes, you can set up traffic accounting based on the polled information. ...Ron Sent from my Verizon Wireless 4G LTE DROID Xavier Beaudouin <xavier.beaudouin(a)hsoftware.com> wrote: >Hi, > >> Are there any plans to include billing per VM host in Observium?. I currently >> have Vmware ESXi hosts and I would like to be able to bill traffic per VM >> machine. > >> All VM hosts are under VDS switch but observium doesn’t recognise separate >> ports. > >I really think the best way to do that is add a snmpd daemon on these hosts... > >Or use somewhat virtual switch from cisco... (didn't tested this solution btw) > >Xavier > > >-- >Xavier Beaudouin - Senior Network and System Administrator >Infrastructure and network director >Horizon Software - http://www.hsoftware.com/ >13 rue La Fayette - 75009 PARIS - France >Phone: +33 (0)1 4260 9490 Fax: +33 (0)1 44 56 97 01 > > >_______________________________________________ >observium mailing list >observium(a)observium.org >http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > >

2 1

Billing per VM host
by Zalewski, Dominik 16 Dec '14

16 Dec '14

Hi, Are there any plans to include billing per VM host in Observium?. I currently have Vmware ESXi hosts and I would like to be able to bill traffic per VM machine. All VM hosts are under VDS switch but observium doesn’t recognise separate ports. Thanks Dominik

2 1

Progressive search and wrong answers
by Kilian Krause 16 Dec '14

16 Dec '14

Hi, since the progressive search seems to start right away and for some reason our setup is too slow/too large to get instant SQL replies the answers seem to be dropping in as the SQL server finishes and them the client will render them as soon as it receives them. This unfortunately gives answers for shorter strings later even if the search field already does seem to be filled with a longer query string. Is there anyone who has implemented some JS/AJAX fix for this and ensure the answers processed by the client are only those matching the current search string (i.e. add a "static" version/revision/serial field into the SQL select and check for that in the results)? Cheers, Kilian -- Kilian Krause Rechenzentrum Universität Stuttgart (RUS) Abt. NKS - Netze und Kommunikationssysteme Tel.: +49 (711) 685-64512 Fax.: +49 (711) 685-54512 (PC) Fax.: +49 (711) 68 23 57 Allmandring 30a 70550 Stuttgart DEUTSCHLAND

5 6

Vlan Interface Usage Alert
by Chip Pleasants 16 Dec '14

16 Dec '14

Hi All, We created a Alert Checker for interface usage greater than 80%. We are seeing interface usage alerts for Vlan SVI interfaces. These don't mean as much to us as physical interfaces do. Looks like the default bandwidth on a Vlan SVI interface is 1G on our platform. Is there any way we can configure Observium not alert on vlan interfaces? Our Device Match is "os equals ios" and our Entity Match is "*" for the interface usage alerts. I see ifType is an attribute option. Could I do something like "not ifType vlan"? Any suggestions are appreciated. Thanks, -Chip

4 5