- observium - lists.observium.org

OT: GMail (WAS: RE: Not Receiving Alerts ??)
by Phillip Baker 02 Feb '15

02 Feb '15

Off topic rant: In case this wasn’t already obvious from this symptom, nobody should use Gmail for anything important. Some of our customers (one a google apps “partner” or whatever) have seen it silently drop mails including invoice mails etc (despite aligned SPF, DKIM, DMARC, reverse dns, and no current reputational issue). Even if anything is wrong with the message, they shouldn’t be silently dropping mail on the floor. It’s unbelievably broken as a model for handling “spam” which is what I presume they think they’re doing. It’s just about adequate for a personal searchable mailing list archive. That’s about it. Phil From: observium [mailto:observium-bounces@observium.org] On Behalf Of Torsten-Sven Urbas Sent: 02 February 2015 07:12 To: Observium Network Observation System Subject: Re: [Observium] Not Receiving Alerts ?? Thats interesting. We do see the same issue with google mail. Basically, only recoveries make it through, i barely see a single alert in my mailbox. /Torsten 2015-01-31 3:21 GMT+01:00 Michael Loftis <mloftis(a)wgops.com<mailto:mloftis@wgops.com>>: Gmail randomly eats mail nowadays. I have had an ongoing g issue with it discarding (not flagging as spam) alerts from observium. Specifically my port down alerts. Recovery for the same port comes through just fine. Other alerts and recoveries are all fine. -- Torsten-Sven Urbas ==== Bitte beachten Sie unsere neuen Rufnummern ==== ProfitBricks GmbH • Greifswalder Str. 207 • D - 10405 Berlin Tel: +49 (30) 57700 8234 • Mobile: +49 (170) 77 38 251 • Fax: +49 (30) 57700 8296 www.profitbricks.de<http://www.profitbricks.de/> • torsten-sven.urbas(a)profitbricks.com<mailto:torsten-sven.urbas@profitbricks.com> Sitz der Gesellschaft: Berlin. Registergericht: Amtsgericht Charlottenburg, HRB 125506 B. Geschäftsführer: Andreas Gauger, Achim Weiss. Please consider the environment before printing this email.

1 0

Speed up graph rendering
by Louis Bailleul 02 Feb '15

02 Feb '15

Hi, I am currently using Observium on 149 switches with a total of 10173 ports. I can't say that anything is wrong with it, but the web interface seems a bit slow especially when you want to display pages with a lot of graphs. Observium is currently running in a VM with 4 xeon E5-2650 cores and 3Gb of ram. It's virtual disks are on a local raid5 composed of 10K SAS disks. Observium report that it's VM use on average 25% of its CPUs with spikes at 40% when the web interface is heavily used. The disk I/O doesn't go past 150 ops OUT and rare spikes at 50 IN. Anyone could advise on what kind of hardware or software tweaking could help speed up the thing ? Are they special requirements/tunnings for the rrd storage or the mysql database ? Best regards, Louis

13 35

Not Receiving Alerts ??
by Andres P 02 Feb '15

02 Feb '15

Hi Observium Team! Since 14 Jan I'm not receiving Alerts It seems to be a problem of the mail server. I did the following test, and not receive it. Any idea? root@Observium:/opt/observium# ./test_mail.php -h rcn-rou-01 Observium 0.15.1.6218 Test E-mail Configuration See mail backend options here: http://www.observium.org/wiki/Configuration_Options#Email_backend_settings My Hostname: Observium.Observium.cabase.org.ar E-mails ENABLED globally ($config['email']['enable']) E-mails use PHP-builtin MAIL backend ($config['email']['backend']) E-mails use these default recipient(s) ($config['email']['default']): noc(a)cabase.org.ar E-mails send from ($config['email']['from']): "Observium" <observium(a)Observium.Observium.cabase.org.ar> E-mails send to recipients based on default recipient from config and/or device sysContact. Try send test notification for rcn-rou-01 to these email(s): noc(a)cabase.org.ar Notification for rcn-rou-01 SENT.

4 3

Re: [Observium] Java JVM Monitoring
by Jurasovic Kresimir 01 Feb '15

01 Feb '15

Hi Tom, Thank you for the feeedback. I will look into code for shoutcast and memcached for implementation details. Right now I am trying to cover as much as information as possible. Code it already publicly available on Bitbucket. I am willing to share the results when I complete all the graphs I need and make some usefull tutorial. Will send all the info when I am done. Best regards, Krešimir === Heya, That sounds very interesting! I hope you'll share the result with us. Our app system is a bit basic, but it supports application "instances". I've never actually implemented something with those myself, but things like shoutcast or memcached can support multiple installs on a machine. I think that's the direction you need, but I have no practical information.. check their observium-side code to see how to use, feel free to sollicit feedback either by this list or on IRC. Tom On 23/01/2015 22:18, Kresimir Jurasovic wrote: > Hi, > > I am in the beginning phase of using Observium. I have everything > set-up and working. No problems so far. > > One feature I am missing is Java JVM Monitoring. In your Bug Tracker I > found the following ticket: > > http://jira.observium.org/browse/OBSERVIUM-554?jql=text%20~%20%22JVM%22 > > In it you mention that this should be implemented using unix-agent. > This is perfectly fine by me. > I have implemented a Java application which is run by unix-agent and > retrieves data from the JVM using the JMX protocol. I have graphs > set-up and so far every is running ok. > > There is only one feature I cannot figure out how to implement. This > feature is very important for me. In most cases there will be more > then one JVM on a single machine. From monitoring perspective this > should be treated as separate application but handled by the same > agent. Is this even possible on Observium? If it is can you please > point me to the right code so I can see how this is done. > > Thanks, > Kre?imir > _______________________________________________ > observium mailing list > observium(a)observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

4 3

Monitoring Logs
by Constantinos Coudounaris 31 Jan '15

31 Jan '15

Hello everyone, I'm trying to setup observium to monitor logs but i'm having some issues. I've configured rsyslog and i've followed the instructions at http://www.observium.org/wiki/Rsyslog_Syslog_Server however I get the following error in observium. I've configured the following files: /etc/rsyslog.conf - added forwarding rule to observium rule aswell as the following. The spool files get created successfully. $WorkDirectory /var/lib/rsyslog # where to place spool files $ActionQueueFileName fwdRule1 # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown $ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount -1 # infinite retries if host is down /etc/rsyslog.d/30-observium.conf - added as per observium documentation Error Message I get in Observium: No syslog entries found! This device does not have any syslog entries. Check that the syslog daemon and Observium configuration options are set correctly, that this device is configured to send syslog to Observium and that there are no firewalls blocking the messages. Thank you. -- Constantinos Coudounaris

1 0

rrdcached worked for me! (Or "How to solve gaps in your graphs")
by Tristan Rhodes 30 Jan '15

30 Jan '15

I was seeing gaps in my Observium graphs, so I knew that something was wrong with my system. The output from the Observium "poller-wrapper.py" cron job has statistics on how long it takes to complete each polling cycle. These are scheduled every 5 minutes (300 seconds). Here is what I found: "INFO: poller-wrapper polled 312 devices in 1142 seconds with 20 workers WARNING: the process took more than 5 minutes to finish, you need faster hardware or more threads INFO: in sequential style polling the elapsed time would have been: 21663 seconds WARNING: Consider setting a minimum of 77 threads. (This does not constitute professional advice!)" I tried increasing threads on the poller-wrapper.py cron job, but this did not speed things up. The "top" command showed me that this system was always waiting for disk I/O, which is indicated by a high value for "wa". I decided to give rrdcached a try, after reading this blog: http://blog.best-practice.se/2014/10/using-rrdcached-with-observium.html With rrdcached enabled, I am finally completing my polls with 5-minutes! INFO: poller-wrapper polled 312 devices in 51 seconds with 20 workers Yes, it went from 1142 seconds to 51 seconds, over 22 times faster! I know people have said the web interface would be slower, but from my perspective it is loading much faster now that rrdcached is saving some disk IOPS for Apache. (Disclaimer: I have compiled and installed rrdtool 1.4.9 to try to maximize speed on the web-interface) I hope this help people who are bottle-necked on IOPS from their disks, like I was. Cheers, Tristan *Tristan Rhodes* Network Engineer Weber State University 801.626.8549

3 2

Slow polling for ports.inc.php module
by Jaynee Shah 30 Jan '15

30 Jan '15

Hi, We have heavily loaded Brocade MLXe routers, and polling time is very high for these devices. The same is seen for few Arista switches (at longer distances) as well. This certainly is related to IF-MIB, trying to poll lots of data statistics for all interfaces, resulting in ports.inc.php module run time of ~150sec+. Ideally, we want to reduce this polling time to <60sec. time /usr/bin/snmpbulkwalk -Cr60 -v2c -c 'jtv' -OQUs -m IF-MIB -M /home/jtv/observium//mibs/rfc:/home/jtv/observium//mibs/net-snmp 'udp':'br02.lhr02':'161' ifEntry real 0m57.616s user 0m0.044s sys 0m0.012s time /usr/bin/snmpbulkwalk -Cr60 -v2c -c 'jtv' -OQUs -m IF-MIB -M /home/jtv/observium//mibs/rfc:/home/jtv/observium//mibs/net-snmp 'udp':'br02.lhr02':'161' ifXEntry real 0m52.763s user 0m0.032s sys 0m0.024s We do not care about some of the interface statistics, for example, mtu, speed, duplex, promiscuous mode, trunk, vlan, etc. We only want interface traffic in/out and error statistics. Is there a way to selectively modify any discovery/polling/MIB files, and only keep the statistics we are interested in? Thanks, Jaynee

3 5

Not Receiving Alerts ??
by Andres P 29 Jan '15

29 Jan '15

Hi Observium Team, >From last two weeks I left to receive my Alerts. I did a test

1 0

Re: [Observium] mysql
by Rob VanHooren 29 Jan '15

29 Jan '15

theres a perl script that's pretty common, go hit mysqltuner.com and grok the results against your db also works with maria R. Typogaphical errors courtesy of iPhone > On Jan 29, 2015, at 11:37, observium-request(a)observium.org wrote: > > Send observium mailing list submissions to > observium(a)observium.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > or, via email, send a message with subject or body 'help' to > observium-request(a)observium.org > > You can reach the person managing the list at > observium-owner(a)observium.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of observium digest..." > > > Today's Topics: > > 1. Re: webpage no longer loads or takes forever (Ron Marosko) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 29 Jan 2015 10:37:32 -0600 > From: "Ron Marosko" <ron(a)rjr-services.com> > To: "'Observium Network Observation System'" <observium(a)observium.org> > Subject: Re: [Observium] webpage no longer loads or takes forever > Message-ID: <001801d03be1$e1db2420$a5916c60$(a)rjr-services.com> > Content-Type: text/plain; charset="utf-8" > > I had this same issue once we upped our device count to roughly 600 devices. Edited the my.cnf with > > max_connections = 512 > > And restarted, and no more problems. > > > > Agree with Tristan, that?s something that needs to be added to the wiki. > > > > From: observium [mailto:observium-bounces@observium.org] On Behalf Of Jerome Jenkins > Sent: Thursday, January 29, 2015 10:26 AM > To: Observium Network Observation System > Subject: Re: [Observium] webpage no longer loads or takes forever > > > > Thanks. Looks like my max_connections setting is commented out. I?ll uncomment it and change it to 500 and see if that helps. > > > > Below are the settings in the Fine Tuning section of the my.cnf on my box: > > > > # * Fine Tuning > > # > > key_buffer = 16M > > max_allowed_packet = 16M > > thread_stack = 192K > > thread_cache_size = 8 > > # This replaces the startup script and checks MyISAM tables if needed > > # the first time they are touched > > myisam-recover = BACKUP > > #max_connections = 100 > > #table_cache = 64 > > #thread_concurrency = 10 > > # > > # * Query Cache Configuration > > # > > query_cache_limit = 1M > > query_cache_size = 16M > > > > From: observium [mailto:observium-bounces@observium.org] On Behalf Of Tristan Rhodes > Sent: Thursday, January 29, 2015 11:21 AM > To: Observium Network Observation System > Subject: Re: [Observium] webpage no longer loads or takes forever > > > > Sorry! I was mixing this up with another error. The heap size fix above is for when you search "/var/log/mysql/error.log" and find: > > > > [ERROR] /usr/sbin/mysqld: The table 'alert_table-state' is full > > > > > > On the other hand, if you get the ?MySQL Error: Too many connections? in the web-page, then you need to increase the number of connections in "/etc/mysql/my.cnf": > > > > max_connections = 500 > > (I think the default is 151) > > > > I will try to add these common issues to the Observium wiki. > > > > Good luck! > > > > > Tristan Rhodes > Network Engineer > > Weber State University > > 801.626.8549 > > <http://www.weber.edu/wsuimages/brand/templates/emailsig_sig1.jpg> > > > > > > On Thu, Jan 29, 2015 at 9:01 AM, Tristan Rhodes <tristanrhodes(a)weber.edu> wrote: > > Jerome, > > > > I ran into this problem too. I think it is common for anyone with a few hundred devices, using the default MYSQL settings. > > > > Edit "/etc/mysql/my.cnf " and add this setting: > > > > max_heap_table_size = 512M > > > > I was told the default is 16M, so you can pick any larger number you want (512M might be over-kill). > > > > Cheers, > > > > Tristan > > > > > Tristan Rhodes > Network Engineer > > Weber State University > > 801.626.8549 > > <http://www.weber.edu/wsuimages/brand/templates/emailsig_sig1.jpg> > > > > > > On Thu, Jan 29, 2015 at 7:37 AM, Jerome Jenkins <JJENKINS(a)wpahs.org> wrote: > > I was able to gather the stats shortly after rebooting by holding the power button. After running for about 30mins now?.the webpage now gives the error: > > > > ?MySQL Error: Too many connections? > > > > From: Jerome Jenkins > Sent: Thursday, January 29, 2015 9:10 AM > To: Observium Network Observation System > Subject: RE: [Observium] webpage no longer loads or takes forever > > > > Appologies for lack of info: > > > > Shows total of 1027 devices, NOT 1200 (original list was 1200 but not all of it imported) > > > > Observium is stored on /opt (which is a raid-array of (6) 146GB drives) > > > > root@observium:~# df -h > > Filesystem Size Used Avail Use% Mounted on > > /dev/mapper/root-root 9.7G 2.4G 6.8G 27% / > > none 4.0K 0 4.0K 0% /sys/fs/cgroup > > udev 12G 4.0K 12G 1% /dev > > tmpfs 2.4G 612K 2.4G 1% /run > > none 5.0M 0 5.0M 0% /run/lock > > none 12G 0 12G 0% /run/shm > > none 100M 0 100M 0% /run/user > > /dev/mapper/observium--data-data 684G 195G 455G 30% /opt > > > > > > (2) Quad-core processors: > > > > cpu family : 6 > > model : 23 > > model name : Intel(R) Xeon(R) CPU L5420 @ 2.50GHz > > stepping : 10 > > microcode : 0xa07 > > cpu MHz : 2499.803 > > cache size : 6144 KB > > > > > > > > Meminfo: > > > > MemTotal: 24689672 kB > > MemFree: 19383548 kB > > Buffers: 124572 kB > > Cached: 3489096 kB > > SwapCached: 0 kB > > Active: 1476312 kB > > Inactive: 3352276 kB > > Active(anon): 1220624 kB > > Inactive(anon): 23092 kB > > Active(file): 255688 kB > > Inactive(file): 3329184 kB > > Unevictable: 0 kB > > Mlocked: 0 kB > > SwapTotal: 25161724 kB > > SwapFree: 25161724 kB > > Dirty: 353976 kB > > Writeback: 2704 kB > > AnonPages: 1214728 kB > > Mapped: 56804 kB > > Shmem: 28796 kB > > Slab: 210032 kB > > SReclaimable: 178204 kB > > SUnreclaim: 31828 kB > > KernelStack: 3552 kB > > PageTables: 30896 kB > > NFS_Unstable: 0 kB > > Bounce: 0 kB > > WritebackTmp: 0 kB > > CommitLimit: 37506560 kB > > Committed_AS: 2858996 kB > > VmallocTotal: 34359738367 kB > > VmallocUsed: 328028 kB > > VmallocChunk: 34359404252 kB > > HardwareCorrupted: 0 kB > > AnonHugePages: 106496 kB > > HugePages_Total: 0 > > HugePages_Free: 0 > > HugePages_Rsvd: 0 > > HugePages_Surp: 0 > > Hugepagesize: 2048 kB > > DirectMap4k: 59728 kB > > DirectMap2M: 25104384 kB > > > > > > > > From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong > Sent: Tuesday, January 27, 2015 5:18 PM > To: Observium Network Observation System > Subject: Re: [Observium] webpage no longer loads or takes forever > > > > Nice that you told us what ram and CPU you have. Less nice that you didn't tell us what storage you're using. > > Thus, insufficient information... > > Adam. > > Sent with AquaMail for Android > http://www.aqua-mail.com > > On 27 January 2015 13:56:17 Jerome Jenkins <JJENKINS(a)wpahs.org> wrote: > > Running on Ubuntu Server LTS 14.0.1 > > > > Server is 8 cores with 24GB of memory > > > > Was working perfectly during initial install of the software. Once I let it discover all of our routers and switches (approx. 1200 devices) things went downhill. Login to the cli of the server takes very long and most times will time-out. Webpage never loads. Only way to fix is to reboot the server and once it begins polling everything again?nothing is accessible. I changed the setting for php to use 3GB of memory and change the poller instances per the performance tuning guide for observium to > > > > */5 * * * * root /opt/observium/poller-wrapper.py 10 >> /dev/null 2>&1 > > > > Seems that the memory and processor are probably being consumed by the poller? > > > > Jerome S. Jenkins, Sr. > > Sr. Network Engineer > > jjenkins(a)wpahs.org > > > > ************************************************************************* > Confidentiality Notice: This e-mail message and any attachments are > intended solely for the individual or individuals designated above. It may > contain confidential or proprietary information and may be subject to > confidentiality protections. If you are not the intended recipient, you > are not authorized to read, copy, retain or distribute this message. If > you receive this message in error, please notify the sender by reply e-mail > and delete this message. The sender does not waive any privilege or right > of privacy or confidentiality that may attach to this communication. Thank > you. > ************************************************************************* > > > _______________________________________________ > observium mailing list > observium(a)observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > ************************************************************************* > Confidentiality Notice: This e-mail message and any attachments are intended solely for the individual or individuals designated above. It may contain confidential or proprietary information and may be subject to confidentiality protections. If you are not the intended recipient, you are not authorized to read, copy, retain or distribute this message. If you receive this message in error, please notify the sender by reply e-mail and delete this message. The sender does not waive any privilege or right of privacy or confidentiality that may attach to this communication. Thank you. > ************************************************************************* > > > > _______________________________________________ > observium mailing list > observium(a)observium.org > http://postman.memetic.org/cgi-bin/mailman/listinfo/observium > > > > > > ************************************************************************* > Confidentiality Notice: This e-mail message and any attachments are intended solely for the individual or individuals designated above. It may contain confidential or proprietary information and may be subject to confidentiality protections. If you are not the intended recipient, you are not authorized to read, copy, retain or distribute this message. If you receive this message in error, please notify the sender by reply e-mail and delete this message. The sender does not waive any privilege or right of privacy or confidentiality that may attach to this communication. Thank you. > ************************************************************************* > > > > -- > This message has been scanned for viruses and dangerous content by > <http://www.efa-project.org> E.F.A. Project, and is believed to be clean. > Click here to report this message as spam. <http://smtp.rjr-services.com/cgi-bin/learn-msg.cgi?id=7808B604CC.A41F0&toke…> > >

1 0

webpage no longer loads or takes forever
by Jerome Jenkins 29 Jan '15

29 Jan '15

Running on Ubuntu Server LTS 14.0.1 Server is 8 cores with 24GB of memory Was working perfectly during initial install of the software. Once I let it discover all of our routers and switches (approx. 1200 devices) things went downhill. Login to the cli of the server takes very long and most times will time-out. Webpage never loads. Only way to fix is to reboot the server and once it begins polling everything again...nothing is accessible. I changed the setting for php to use 3GB of memory and change the poller instances per the performance tuning guide for observium to */5 * * * * root /opt/observium/poller-wrapper.py 10 >> /dev/null 2>&1 Seems that the memory and processor are probably being consumed by the poller? Jerome S. Jenkins, Sr. Sr. Network Engineer jjenkins(a)wpahs.org ************************************************************************* Confidentiality Notice: This e-mail message and any attachments are intended solely for the individual or individuals designated above. It may contain confidential or proprietary information and may be subject to confidentiality protections. If you are not the intended recipient, you are not authorized to read, copy, retain or distribute this message. If you receive this message in error, please notify the sender by reply e-mail and delete this message. The sender does not waive any privilege or right of privacy or confidentiality that may attach to this communication. Thank you. *************************************************************************

6 10