- observium - lists.observium.org

Very Large Environment
by Joe Hoh 29 Jul '13

29 Jul '13

Thanks in advance for anyone who can help. We are deploying Observium in a very large environment. We are upwards of 6,000 devices and 300,000 ports. I don't think we are going to be able to poll this in 5 minutes. This is what we have done so far: - Distributed polling - Polling - 8 VMs - 4core, 8GB RAM each - MySQL - 4 core, 8GB RAM - Web Interface - 4 core, 8GB RAM - NFS Server (for centralized /opt/observium - except for the MIBS directory, which is copied across each server) - moving to an EMC VNX in 2 weeks. - rrdcached being implemented (any assistance here is helpful) - Modified poller-wrapper.py to distribute the devices that poll within 120s across multiple instances of poller-wrapper.py running on multiple hosts. Devices that poll in more than 120s are polled on separate servers at 15 minute intervals. - poller-wrapper has been modified to allow for multiple instances just like poller.php with MODULUS distribution - Each instance of poller-wrapper.py gets an instance number and the total number of instances. - All of the devices with the last poll time < 120 seconds are MOD'ed with the device_id and the total number of instances and compared to the instance number - device_id MOD total_instances = this_instance - Tuning threads in each poller-wrapper.py - currently at 16 threads and 2 instances on each 4 vCPU server for 32 threads running at once or 8 threads per core - The DC is on the west coast and that presents latency problems. We may need to address with distributed polling - We are at 1024MB in php.ini - We are using xcache (tuning help is appreciated - or should we just turn it off)? My questions are: - How can I change the default RRD behavior to use 15 minute intervals instead of 5 minute intervals - 15 minutes x 2 weeks - 2 hours x 5 weeks - 4 hours x 12 months - We want to keep the max/min/average/87.5th percentile (since only 8 measurements per 2 hours) - I don't see the configuration items for that. - Would we be better with a few big boxes rather than small VMs?

3 4

Re: [Observium] Graphs
by Adam Armstrong 28 Jul '13

28 Jul '13

Not enough spoon? Tom Laermans <tom.laermans(a)powersource.cx> wrote: >_______________________________________________ >observium mailing list >observium(a)observium.org >http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

2 1

Re: [Observium] Graphs
by Ryan Milton - MVS USA 27 Jul '13

27 Jul '13

So, I uninstalled the whole mess, reinstalled everything. ONe thing that immediately worked was the emails from observium! This time I added only one machine manually. I don't really see that I did anything different. In fact, I kept my config.php and just pasted it in after I did the install and voila! Seems to be working now. However, I would add that on the installation procedures, it should be clarified that with the mysql, the coder needs to add the passwords for the users manually. Regards, Ryan Milton Network Manager Hawthorne, NJ o: 201-447-1505 x124 c: 862-249-5230 rmilton(a)mvsusa.com<mailto:rmilton@mvsusa.com> On Jul 27, 2013, at 1:56 PM, Tracy Phillips <tracphil(a)mantoso.com<mailto:tracphil@mantoso.com>> wrote: On 07/27/2013 1:50 pm, Ryan Milton - MVS USA wrote: What do you think? It's really odd. Everything looks normal. Is /opt/observium on your local filesystem or is it in NFS mount? How many hosts are in your observium install so far? I am a little baffled. One thing that I do... not sure if it matters, is that I don't add any hosts by hand like in the install instructions. That is the only place I deviate. Tracy

2 1

Re: [Observium] Graphs
by Adam Armstrong 27 Jul '13

27 Jul '13

All installation problems are solved by following the install guide properly. We long ago decided that trying to diagnose what people typed wrong just isn't worth the time and just recommend that people reinstall in most cases. Unless they want to pay us €1000 to fix it for them, of course :-) Adam. Ryan Milton - MVS USA <RMilton(a)mvsusa.com> wrote: >_______________________________________________ >observium mailing list >observium(a)observium.org >http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

1 0

Re: [Observium] Graphs
by Adam Armstrong 26 Jul '13

26 Jul '13

Please provide screenshots, otherwise no one will know what you actually mean. Adam. Ryan Milton - MVS USA <RMilton(a)mvsusa.com> wrote: >_______________________________________________ >observium mailing list >observium(a)observium.org >http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

2 4

Graphs
by Ryan Milton - MVS USA 26 Jul '13

26 Jul '13

I read the FAQ, and indeed that mysterious space after the ?> turned out to be the first problem. However, the problem now is that the live data doesn't show on the graphs anywhere. Regards, Ryan Milton Network Manager o: 201-447-1505 x124 c: 862-249-5230

1 0

ifLastchanged
by Benjamin Ballmert 26 Jul '13

26 Jul '13

2 1

Re: [Observium] Memory draw error
by Adam Armstrong 25 Jul '13

25 Jul '13

That output is not actually related to memory pools, its from the ucd MiB used on Unix hosts. Adam. Austin Marsh <amarsh(a)clearwateranalytics.com> wrote: >_______________________________________________ >observium mailing list >observium(a)observium.org >http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

1 0

Memory draw error
by Austin Marsh 25 Jul '13

25 Jul '13

I just did a fresh install of the newest observium (0.13.7.4200). Everything is working well except for my nexus 5k devices show a draw error for memory. Memory stats for the switches was working in previous observium releases. Here is the output from the poller below. I can get the memory from snmp via command line, I assume some mibs are messed up? /usr/bin/snmpget -v2c -c cwanro -OQUs -m UCD-SNMP-MIB -M /opt/observium/mibs udp:boi-sw10:161 memTotalSwap.0 memAvailSwap.0 memTotalReal.0 memAvailReal.0 memTotalFree.0 memShared.0 memBuffer.0 memCached.0 memTotalSwap.0 = No Such Object available on this agent at this OID memAvailSwap.0 = No Such Object available on this agent at this OID memTotalReal.0 = No Such Object available on this agent at this OID memAvailReal.0 = No Such Object available on this agent at this OID memTotalFree.0 = No Such Object available on this agent at this OID memShared.0 = No Such Object available on this agent at this OID memBuffer.0 = No Such Object available on this agent at this OID memCached.0 = No Such Object available on this agent at this OID ________________________________ NOTICE: This e-mail message, together with any attachments, contains information of Clearwater Analytics and/or its affiliates that may be confidential, proprietary copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named on this message. If you are not the intended recipient, and have received this message in error, please immediately delete it. The information we provide is from sources Clearwater Analytics considers reliable, but Clearwater Analytics provides no warranties regarding the accuracy of the information. Further, nothing in the email should be construed as legal, financial, or tax advice, and any questions regarding the intended recipient's individual circumstances should be addressed to that recipient's lawyer and/or accountant.

1 0

Re: [Observium] vrf dance
by Rob VanHooren 25 Jul '13

25 Jul '13

well then if nobody else out there gets it right, then that just screams "opportunity" for Observium! we can just all hope that it doesn't wind up giving Adam Mike and Tom too much brain damage. :-) R. PS: DNS separation I could see as logical for VDCs. But for VRFs? That's banannas to have to try do. Don't even know that it'd fully function. (e.g., traps etc. are pinned to source from just one vrf; how to map back the event blame of $other_vrf_"device" as culprit?) Screws loose in San Jose, for sure. Chris & Darius wrote: > Date: Wed, 24 Jul 2013 16:01:59 -0400 > From: Chris Moody <chris(a)node-nine.com> > To: Observium Network Observation System <observium(a)observium.org> > Subject: Re: [Observium] remedies to overcome vrf dance? > Message-ID: <51F032B7.5060607(a)node-nine.com> > Content-Type: text/plain; charset="iso-8859-1"; Format="flowed" > > I can say we've had nothing but pain in trying to get all the > context-mapping stuff recognized by pretty much every tool we use. > Because of how Cisco implemented this, even some of our inventory apps > are confused and think we have 3x more Nexus chassis than we actually do > because in their logic, each context-mapped VRF is a separate device. > IMHO, I -hate- the way Cisco has "updated" SNMP on the Nexus platform. > > One workaround we've had to use is to use DNS names per L3 interface, so > that we can refer to the vrf/device as say 'Nexus1_mgmt0.foo.com'(which > is part of the 'management' vrf) & 'Nexus1_eth1-1.foo.com'(which is part > of the 'vrf1' vrf). This way applications think of them as separate > devices due to the hostname/IP differences. > > -Chris > > On 7/24/13 1:29 PM, Darius Seroka wrote: >> Hi Rob, >> >> I posted a similar comment a few weeks/months back but my problem was >> with c6500's. Indeed it was the way cisco describes the vrf's so only >> thing that can be done is getting them to do their oid's more >> uniformly across their product range. I was not able to open a support >> request with the c6500s so dont have any tickets to refer to. >

1 0