- observium - lists.observium.org

Re: [Observium] Java JVM Monitoring
by Jurasovic Kresimir 01 Feb '15

01 Feb '15

Hi Tom, Thank you for the feeedback. I will look into code for shoutcast and memcached for implementation details. Right now I am trying to cover as much as information as possible. Code it already publicly available on Bitbucket. I am willing to share the results when I complete all the graphs I need and make some usefull tutorial. Will send all the info when I am done. Best regards, Krešimir === Heya, That sounds very interesting! I hope you'll share the result with us. Our app system is a bit basic, but it supports application "instances". I've never actually implemented something with those myself, but things like shoutcast or memcached can support multiple installs on a machine. I think that's the direction you need, but I have no practical information.. check their observium-side code to see how to use, feel free to sollicit feedback either by this list or on IRC. Tom On 23/01/2015 22:18, Kresimir Jurasovic wrote: > Hi, > > I am in the beginning phase of using Observium. I have everything > set-up and working. No problems so far. > > One feature I am missing is Java JVM Monitoring. In your Bug Tracker I > found the following ticket: > > http://jira.observium.org/browse/OBSERVIUM-554?jql=text%20~%20%22JVM%22 > > In it you mention that this should be implemented using unix-agent. > This is perfectly fine by me. > I have implemented a Java application which is run by unix-agent and > retrieves data from the JVM using the JMX protocol. I have graphs > set-up and so far every is running ok. > > There is only one feature I cannot figure out how to implement. This > feature is very important for me. In most cases there will be more > then one JVM on a single machine. From monitoring perspective this > should be treated as separate application but handled by the same > agent. Is this even possible on Observium? If it is can you please > point me to the right code so I can see how this is done. > > Thanks, > Kre?imir > _______________________________________________ > observium mailing list > observium(a)observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

4 3

Monitoring Logs
by Constantinos Coudounaris 31 Jan '15

31 Jan '15

Hello everyone, I'm trying to setup observium to monitor logs but i'm having some issues. I've configured rsyslog and i've followed the instructions at http://www.observium.org/wiki/Rsyslog_Syslog_Server however I get the following error in observium. I've configured the following files: /etc/rsyslog.conf - added forwarding rule to observium rule aswell as the following. The spool files get created successfully. $WorkDirectory /var/lib/rsyslog # where to place spool files $ActionQueueFileName fwdRule1 # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown $ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount -1 # infinite retries if host is down /etc/rsyslog.d/30-observium.conf - added as per observium documentation Error Message I get in Observium: No syslog entries found! This device does not have any syslog entries. Check that the syslog daemon and Observium configuration options are set correctly, that this device is configured to send syslog to Observium and that there are no firewalls blocking the messages. Thank you. -- Constantinos Coudounaris

1 0

rrdcached worked for me! (Or "How to solve gaps in your graphs")
by Tristan Rhodes 30 Jan '15

30 Jan '15

I was seeing gaps in my Observium graphs, so I knew that something was wrong with my system. The output from the Observium "poller-wrapper.py" cron job has statistics on how long it takes to complete each polling cycle. These are scheduled every 5 minutes (300 seconds). Here is what I found: "INFO: poller-wrapper polled 312 devices in 1142 seconds with 20 workers WARNING: the process took more than 5 minutes to finish, you need faster hardware or more threads INFO: in sequential style polling the elapsed time would have been: 21663 seconds WARNING: Consider setting a minimum of 77 threads. (This does not constitute professional advice!)" I tried increasing threads on the poller-wrapper.py cron job, but this did not speed things up. The "top" command showed me that this system was always waiting for disk I/O, which is indicated by a high value for "wa". I decided to give rrdcached a try, after reading this blog: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html With rrdcached enabled, I am finally completing my polls with 5-minutes! INFO: poller-wrapper polled 312 devices in 51 seconds with 20 workers Yes, it went from 1142 seconds to 51 seconds, over 22 times faster! I know people have said the web interface would be slower, but from my perspective it is loading much faster now that rrdcached is saving some disk IOPS for Apache. (Disclaimer: I have compiled and installed rrdtool 1.4.9 to try to maximize speed on the web-interface) I hope this help people who are bottle-necked on IOPS from their disks, like I was. Cheers, Tristan *Tristan Rhodes* Network Engineer Weber State University 801.626.8549

3 2

Slow polling for ports.inc.php module
by Jaynee Shah 30 Jan '15

30 Jan '15

Hi, We have heavily loaded Brocade MLXe routers, and polling time is very high for these devices. The same is seen for few Arista switches (at longer distances) as well. This certainly is related to IF-MIB, trying to poll lots of data statistics for all interfaces, resulting in ports.inc.php module run time of ~150sec+. Ideally, we want to reduce this polling time to <60sec. time /usr/bin/snmpbulkwalk -Cr60 -v2c -c 'jtv' -OQUs -m IF-MIB -M /home/jtv/observium//mibs/rfc:/home/jtv/observium//mibs/net-snmp 'udp':'br02.lhr02':'161' ifEntry real 0m57.616s user 0m0.044s sys 0m0.012s time /usr/bin/snmpbulkwalk -Cr60 -v2c -c 'jtv' -OQUs -m IF-MIB -M /home/jtv/observium//mibs/rfc:/home/jtv/observium//mibs/net-snmp 'udp':'br02.lhr02':'161' ifXEntry real 0m52.763s user 0m0.032s sys 0m0.024s We do not care about some of the interface statistics, for example, mtu, speed, duplex, promiscuous mode, trunk, vlan, etc. We only want interface traffic in/out and error statistics. Is there a way to selectively modify any discovery/polling/MIB files, and only keep the statistics we are interested in? Thanks, Jaynee

3 5

Not Receiving Alerts ??
by Andres P 29 Jan '15

29 Jan '15

Hi Observium Team, >From last two weeks I left to receive my Alerts. I did a test

1 0

Re: [Observium] mysql
by Rob VanHooren 29 Jan '15

29 Jan '15

theres a perl script that's pretty common, go hit mysqltuner.com and grok the results against your db also works with maria R. Typogaphical errors courtesy of iPhone > On Jan 29, 2015, at 11:37, observium-request(a)observium.org wrote: > > Send observium mailing list submissions to > observium(a)observium.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > or, via email, send a message with subject or body 'help' to > observium-request(a)observium.org > > You can reach the person managing the list at > observium-owner(a)observium.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of observium digest..." > > > Today's Topics: > > 1. Re: webpage no longer loads or takes forever (Ron Marosko) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 29 Jan 2015 10:37:32 -0600 > From: "Ron Marosko" <ron(a)rjr-services.com> > To: "'Observium Network Observation System'" <observium(a)observium.org> > Subject: Re: [Observium] webpage no longer loads or takes forever > Message-ID: <001801d03be1$e1db2420$a5916c60$(a)rjr-services.com> > Content-Type: text/plain; charset="utf-8" > > I had this same issue once we upped our device count to roughly 600 devices. Edited the my.cnf with > > max_connections = 512 > > And restarted, and no more problems. > > > > Agree with Tristan, that?s something that needs to be added to the wiki. > > > > From: observium [mailto:observium-bounces@observium.org] On Behalf Of Jerome Jenkins > Sent: Thursday, January 29, 2015 10:26 AM > To: Observium Network Observation System > Subject: Re: [Observium] webpage no longer loads or takes forever > > > > Thanks. Looks like my max_connections setting is commented out. I?ll uncomment it and change it to 500 and see if that helps. > > > > Below are the settings in the Fine Tuning section of the my.cnf on my box: > > > > # * Fine Tuning > > # > > key_buffer = 16M > > max_allowed_packet = 16M > > thread_stack = 192K > > thread_cache_size = 8 > > # This replaces the startup script and checks MyISAM tables if needed > > # the first time they are touched > > myisam-recover = BACKUP > > #max_connections = 100 > > #table_cache = 64 > > #thread_concurrency = 10 > > # > > # * Query Cache Configuration > > # > > query_cache_limit = 1M > > query_cache_size = 16M > > > > From: observium [mailto:observium-bounces@observium.org] On Behalf Of Tristan Rhodes > Sent: Thursday, January 29, 2015 11:21 AM > To: Observium Network Observation System > Subject: Re: [Observium] webpage no longer loads or takes forever > > > > Sorry! I was mixing this up with another error. The heap size fix above is for when you search "/var/log/mysql/error.log" and find: > > > > [ERROR] /usr/sbin/mysqld: The table 'alert_table-state' is full > > > > > > On the other hand, if you get the ?MySQL Error: Too many connections? in the web-page, then you need to increase the number of connections in "/etc/mysql/my.cnf": > > > > max_connections = 500 > > (I think the default is 151) > > > > I will try to add these common issues to the Observium wiki. > > > > Good luck! > > > > > Tristan Rhodes > Network Engineer > > Weber State University > > 801.626.8549 > > <http://www.weber.edu/wsuimages/brand/templates/emailsig_sig1.jpg> > > > > > > On Thu, Jan 29, 2015 at 9:01 AM, Tristan Rhodes <tristanrhodes(a)weber.edu> wrote: > > Jerome, > > > > I ran into this problem too. I think it is common for anyone with a few hundred devices, using the default MYSQL settings. > > > > Edit "/etc/mysql/my.cnf " and add this setting: > > > > max_heap_table_size = 512M > > > > I was told the default is 16M, so you can pick any larger number you want (512M might be over-kill). > > > > Cheers, > > > > Tristan > > > > > Tristan Rhodes > Network Engineer > > Weber State University > > 801.626.8549 > > <http://www.weber.edu/wsuimages/brand/templates/emailsig_sig1.jpg> > > > > > > On Thu, Jan 29, 2015 at 7:37 AM, Jerome Jenkins <JJENKINS(a)wpahs.org> wrote: > > I was able to gather the stats shortly after rebooting by holding the power button. After running for about 30mins now?.the webpage now gives the error: > > > > ?MySQL Error: Too many connections? > > > > From: Jerome Jenkins > Sent: Thursday, January 29, 2015 9:10 AM > To: Observium Network Observation System > Subject: RE: [Observium] webpage no longer loads or takes forever > > > > Appologies for lack of info: > > > > Shows total of 1027 devices, NOT 1200 (original list was 1200 but not all of it imported) > > > > Observium is stored on /opt (which is a raid-array of (6) 146GB drives) > > > > root@observium:~# df -h > > Filesystem Size Used Avail Use% Mounted on > > /dev/mapper/root-root 9.7G 2.4G 6.8G 27% / > > none 4.0K 0 4.0K 0% /sys/fs/cgroup > > udev 12G 4.0K 12G 1% /dev > > tmpfs 2.4G 612K 2.4G 1% /run > > none 5.0M 0 5.0M 0% /run/lock > > none 12G 0 12G 0% /run/shm > > none 100M 0 100M 0% /run/user > > /dev/mapper/observium--data-data 684G 195G 455G 30% /opt > > > > > > (2) Quad-core processors: > > > > cpu family : 6 > > model : 23 > > model name : Intel(R) Xeon(R) CPU L5420 @ 2.50GHz > > stepping : 10 > > microcode : 0xa07 > > cpu MHz : 2499.803 > > cache size : 6144 KB > > > > > > > > Meminfo: > > > > MemTotal: 24689672 kB > > MemFree: 19383548 kB > > Buffers: 124572 kB > > Cached: 3489096 kB > > SwapCached: 0 kB > > Active: 1476312 kB > > Inactive: 3352276 kB > > Active(anon): 1220624 kB > > Inactive(anon): 23092 kB > > Active(file): 255688 kB > > Inactive(file): 3329184 kB > > Unevictable: 0 kB > > Mlocked: 0 kB > > SwapTotal: 25161724 kB > > SwapFree: 25161724 kB > > Dirty: 353976 kB > > Writeback: 2704 kB > > AnonPages: 1214728 kB > > Mapped: 56804 kB > > Shmem: 28796 kB > > Slab: 210032 kB > > SReclaimable: 178204 kB > > SUnreclaim: 31828 kB > > KernelStack: 3552 kB > > PageTables: 30896 kB > > NFS_Unstable: 0 kB > > Bounce: 0 kB > > WritebackTmp: 0 kB > > CommitLimit: 37506560 kB > > Committed_AS: 2858996 kB > > VmallocTotal: 34359738367 kB > > VmallocUsed: 328028 kB > > VmallocChunk: 34359404252 kB > > HardwareCorrupted: 0 kB > > AnonHugePages: 106496 kB > > HugePages_Total: 0 > > HugePages_Free: 0 > > HugePages_Rsvd: 0 > > HugePages_Surp: 0 > > Hugepagesize: 2048 kB > > DirectMap4k: 59728 kB > > DirectMap2M: 25104384 kB > > > > > > > > From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong > Sent: Tuesday, January 27, 2015 5:18 PM > To: Observium Network Observation System > Subject: Re: [Observium] webpage no longer loads or takes forever > > > > Nice that you told us what ram and CPU you have. Less nice that you didn't tell us what storage you're using. > > Thus, insufficient information... > > Adam. > > Sent with AquaMail for Android > http://www.aqua-mail.com > > On 27 January 2015 13:56:17 Jerome Jenkins <JJENKINS(a)wpahs.org> wrote: > > Running on Ubuntu Server LTS 14.0.1 > > > > Server is 8 cores with 24GB of memory > > > > Was working perfectly during initial install of the software. Once I let it discover all of our routers and switches (approx. 1200 devices) things went downhill. Login to the cli of the server takes very long and most times will time-out. Webpage never loads. Only way to fix is to reboot the server and once it begins polling everything again?nothing is accessible. I changed the setting for php to use 3GB of memory and change the poller instances per the performance tuning guide for observium to > > > > */5 * * * * root /opt/observium/poller-wrapper.py 10 >> /dev/null 2>&1 > > > > Seems that the memory and processor are probably being consumed by the poller? > > > > Jerome S. Jenkins, Sr. > > Sr. Network Engineer > > jjenkins(a)wpahs.org > > > > ************************************************************************* > Confidentiality Notice: This e-mail message and any attachments are > intended solely for the individual or individuals designated above. It may > contain confidential or proprietary information and may be subject to > confidentiality protections. If you are not the intended recipient, you > are not authorized to read, copy, retain or distribute this message. If > you receive this message in error, please notify the sender by reply e-mail > and delete this message. The sender does not waive any privilege or right > of privacy or confidentiality that may attach to this communication. Thank > you. > ************************************************************************* > > > _______________________________________________ > observium mailing list > observium(a)observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > ************************************************************************* > Confidentiality Notice: This e-mail message and any attachments are intended solely for the individual or individuals designated above. It may contain confidential or proprietary information and may be subject to confidentiality protections. If you are not the intended recipient, you are not authorized to read, copy, retain or distribute this message. If you receive this message in error, please notify the sender by reply e-mail and delete this message. The sender does not waive any privilege or right of privacy or confidentiality that may attach to this communication. Thank you. > ************************************************************************* > > > > _______________________________________________ > observium mailing list > observium(a)observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > > > > > ************************************************************************* > Confidentiality Notice: This e-mail message and any attachments are intended solely for the individual or individuals designated above. It may contain confidential or proprietary information and may be subject to confidentiality protections. If you are not the intended recipient, you are not authorized to read, copy, retain or distribute this message. If you receive this message in error, please notify the sender by reply e-mail and delete this message. The sender does not waive any privilege or right of privacy or confidentiality that may attach to this communication. Thank you. > ************************************************************************* > > > > -- > This message has been scanned for viruses and dangerous content by > <http://www.efa-project.org> E.F.A. Project, and is believed to be clean. > Click here to report this message as spam. <http://smtp.rjr-services.com/cgi-bin/learn-msg.cgi?id=7808B604CC.A41F0&toke…> > >

1 0

webpage no longer loads or takes forever
by Jerome Jenkins 29 Jan '15

29 Jan '15

Running on Ubuntu Server LTS 14.0.1 Server is 8 cores with 24GB of memory Was working perfectly during initial install of the software. Once I let it discover all of our routers and switches (approx. 1200 devices) things went downhill. Login to the cli of the server takes very long and most times will time-out. Webpage never loads. Only way to fix is to reboot the server and once it begins polling everything again...nothing is accessible. I changed the setting for php to use 3GB of memory and change the poller instances per the performance tuning guide for observium to */5 * * * * root /opt/observium/poller-wrapper.py 10 >> /dev/null 2>&1 Seems that the memory and processor are probably being consumed by the poller? Jerome S. Jenkins, Sr. Sr. Network Engineer jjenkins(a)wpahs.org ************************************************************************* Confidentiality Notice: This e-mail message and any attachments are intended solely for the individual or individuals designated above. It may contain confidential or proprietary information and may be subject to confidentiality protections. If you are not the intended recipient, you are not authorized to read, copy, retain or distribute this message. If you receive this message in error, please notify the sender by reply e-mail and delete this message. The sender does not waive any privilege or right of privacy or confidentiality that may attach to this communication. Thank you. *************************************************************************

6 10

Device alerts for ignored devices
by Robert Williams 29 Jan '15

29 Jan '15

Hi, There seems to be a potential issue starting with r6211 which causes ignored ports and ignored devices to show in the “Device Alerts” list. If I roll back to r6210 then it is ok again, but anything after that causes them all to pop up in the list again. Any ideas? Cheers! Robert Williams Custodian Data Centre Email: Robert(a)CustodianDC.com http://www.CustodianDC.com

2 2

change log
by Zach Underwood 28 Jan '15

28 Jan '15

I would normally go to http://fisheye.observium.org/changelog/Observium to see all of the updates but it seems that it is no longer working. -- Zach Underwood (RHCE,RHCSA,RHCT,UACA) greenvilletowers.com My website <http://zachunderwood.me>

1 0

L2TP ports
by Teemu Grönqvist 28 Jan '15

28 Jan '15

Hello, How are L2TP ports defined? I have a few L2TP ports on a MikroTik as seen here: http://puu.sh/eRKAR/78df1434a9.png Observium however won't add these to Ports -> L2TP list. How does Observium decide which ports end up on this list and which don't? Thank you, Teemu Grönqvist Net9 Oy

2 3