Hey all,
New to Observium and loving it so far.
I've done some reading, but I can not find an answer to my issue. I would really appreciate some pointers here.
After about 6 hours, i start seeing a looping PTR request:
23:14:37.602724 IP 192.168.155.13.39891 > 8.8.8.8.domain: 54851+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.604162 IP 192.168.155.13.53879 > 8.8.8.8.domain: 58763+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.605541 IP 192.168.155.13.52487 > 8.8.8.8.domain: 30717+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.606941 IP 192.168.155.13.34473 > 8.8.8.8.domain: 5645+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.608351 IP 192.168.155.13.45956 > 8.8.8.8.domain: 5585+ PTR? 30.151.168.192.in-addr.arpa. (45) 23:14:37.609741 IP 192.168.155.13.39834 > 8.8.8.8.domain: 4414+ PTR? 30.151.168.192.in-addr.arpa. (45) 23:14:37.611099 IP 192.168.155.13.35512 > 8.8.8.8.domain: 3031+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.612479 IP 192.168.155.13.49633 > 8.8.8.8.domain: 12185+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.613858 IP 192.168.155.13.59192 > 8.8.8.8.domain: 1926+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.615309 IP 192.168.155.13.51916 > 8.8.8.8.domain: 36203+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.616667 IP 192.168.155.13.49695 > 8.8.8.8.domain: 28584+ PTR? 30.151.168.192.in-addr.arpa. (45) 23:14:37.618245 IP 192.168.155.13.50620 > 8.8.8.8.domain: 13819+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.619849 IP 192.168.155.13.42869 > 8.8.8.8.domain: 48052+ PTR? 15.155.168.192.in-addr.arpa. (45)
Manual lookup: root@freshwater ~]host 192.168.155.15 Host 15.155.168.192.in-addr.arpa. not found: 3(NXDOMAIN) [root@freshwater ~]# host 192.168.155.15 8.8.8.8 Using domain server: Name: 8.8.8.8 Address: 8.8.8.8#53 Aliases:
Host 15.155.168.192.in-addr.arpa. not found: 3(NXDOMAIN) [root@freshwater ~]#
I'm not running PTR on these IPs and have turned off discovery (i think) (i dont need it)
From my config:
$config['autodiscovery']['ip_nets'] = array("127.0.0.0/8"); $config['autodiscovery']['xdp'] = FALSE; $config['autodiscovery']['ospf'] = FALSE; $config['autodiscovery']['bgp'] = FALSE; $config['autodiscovery']['libvirt'] = FALSE; $config['autodiscovery']['snmpscan'] = FALSE; $config['enable_printers'] = 0; $config['enable_sla'] = 0; $config['enable_ports_junoseatmvp'] = 0; $config['enable_ports_adsl'] = 1; $config['ignore_mount_optical'] = 1; $config['poller_modules']['unix-agent'] = 1;
It's running on: [root@freshwater observium]# httpd -v Server version: Apache/2.4.6 (CentOS) Server built: May 12 2016 10:27:23 [root@freshwater observium]# php -v PHP 5.4.16 (cli) (built: May 12 2016 13:45:17) Copyright (c) 1997-2013 The PHP Group Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies [root@freshwater observium]#
And a restart to apache does not kill this process. I can't seem to find what is doing it, but the ONLY thing on this server is Observium and nothing else. It's possible it isnt, but it seems to be :( The only way to stop this is to restart the whole server for some reason
ps auxf from when the loop is happening and after an apache restart
root 6299 0.0 0.0 0 0 ? S 23:43 0:00 _ [kworker/3:1] root 1 0.0 0.1 190528 5564 ? Ss Jun14 1:00 /usr/lib/systemd/systemd --switched-root --system --deserialize 21 root 436 0.0 0.1 36820 4856 ? Ss Jun14 8:01 /usr/lib/systemd/systemd-journald root 458 0.0 0.0 118480 1260 ? Ss Jun14 0:00 /usr/sbin/lvmetad -f root 475 0.0 0.0 44868 3464 ? Ss Jun14 0:00 /usr/lib/systemd/systemd-udevd root 541 0.0 0.0 116724 1632 ? S<sl Jun14 0:02 /sbin/auditd -n root 564 0.0 0.0 19312 1260 ? Ss Jun14 0:27 /usr/sbin/irqbalance --foreground root 566 0.0 0.1 249796 7616 ? Ss Jun14 7:00 /usr/bin/vmtoolsd root 572 0.0 0.1 391788 5588 ? Ssl Jun14 0:39 /usr/sbin/rsyslogd -n root 573 0.0 0.0 26400 1744 ? Ss Jun14 0:18 /usr/lib/systemd/systemd-logind dbus 574 0.0 0.0 26724 1828 ? Ss Jun14 0:36 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation chrony 576 0.0 0.0 115844 1900 ? S Jun14 0:02 /usr/sbin/chronyd root 582 0.0 0.0 203368 1240 ? Ssl Jun14 0:00 /usr/sbin/gssproxy -D root 592 0.0 0.2 434960 8368 ? Ssl Jun14 0:12 /usr/sbin/NetworkManager --no-daemon polkitd 607 0.0 0.3 527584 12132 ? Ssl Jun14 0:07 /usr/lib/polkit-1/polkitd --no-debug root 824 0.5 0.7 251120 27696 ? Rs Jun14 48:26 /usr/sbin/snmptrapd -Lsd -f root 825 0.0 0.0 82560 3616 ? Ss Jun14 0:00 /usr/sbin/sshd -D root 23255 0.0 0.1 143416 5544 ? Ss 21:39 0:00 _ sshd: root@pts/0 root 23257 0.0 0.0 115384 2060 pts/0 Ss+ 21:39 0:00 | _ -bash root 25194 0.0 0.1 143808 5928 ? Ss 23:08 0:00 _ sshd: root@pts/1 root 25196 0.0 0.0 115384 2112 pts/1 Ss 23:08 0:00 _ -bash root 8186 0.0 0.0 151168 1960 pts/1 R+ 23:46 0:00 _ ps auxf root 829 0.0 0.0 29304 1004 ? Ss Jun14 0:00 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid root 831 0.0 0.2 222956 10612 ? Ss Jun14 4:36 /usr/sbin/snmpd -LS0-6d -f root 836 0.0 0.3 552452 15072 ? Ssl Jun14 0:00 /usr/sbin/libvirtd root 841 0.0 0.0 126332 1572 ? Ss Jun14 0:02 /usr/sbin/crond -n root 851 0.0 0.0 110036 852 tty1 Ss+ Jun14 0:00 /sbin/agetty --noclear tty1 linux mysql 1171 0.0 0.0 113256 1580 ? Ss Jun14 0:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr mysql 1600 0.4 5.9 1695560 232072 ? Sl Jun14 35:42 _ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/ root 2159 0.0 0.0 91140 2164 ? Ss Jun14 0:02 /usr/libexec/postfix/master -w postfix 2161 0.0 0.1 91420 4132 ? S Jun14 0:00 _ qmgr -l -t unix -u postfix 10160 0.0 0.1 91244 3924 ? S 22:30 0:00 _ pickup -l -t unix -u root 29025 0.0 0.3 396460 15080 ? Ss 23:16 0:00 /usr/sbin/httpd -DFOREGROUND apache 29027 0.0 0.3 397296 12784 ? S 23:16 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 29029 0.0 0.3 397556 12816 ? S 23:16 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 29031 0.0 0.2 396596 8644 ? S 23:16 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 29035 0.0 0.3 397296 12812 ? S 23:18 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 29039 0.0 0.3 397556 12816 ? S 23:18 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 29041 0.0 0.3 397300 12872 ? S 23:18 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 351 0.0 0.2 396596 8644 ? S 23:28 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 2450 0.0 0.2 396596 8644 ? S 23:33 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 4385 0.0 0.2 396596 8644 ? S 23:39 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 6301 0.0 0.1 396596 7652 ? S 23:44 0:00 _ /usr/sbin/httpd -DFOREGROUND [root@freshwater observium]#
Any help appreciated
Thanks!
You’re trying to issue a PTR (reverse lookup) query to a public (looks like google) server with an RFC1918 address… which is really a big no-no.
Options:
1) Create /etc/hosts entries for all your RFC1918-used IP space
2) Create an internal resolver that points to an internal authoritative server for the RFC1918 IP space.
Personally, as we use a LOT of RFC1918 (10.x.x.x/8, 172.16.x.x/12, 192.168.x.x/16) space, I have a pair of internal resolver servers (running unbound) that will answer queries for anywhere, but if they receive any requests for RFC1918 space, they redirect the request to a couple of internal authoritative servers (running nsd) which answer the queries for that RFC1918 space, both forwards and backwards. If you’re just running a single /24 of RFC1918 space, it may be easier to do the /etc/hosts entries.
…Ron
From: observium [mailto:observium-bounces@observium.org] On Behalf Of TAN Lists Sent: Monday, June 20, 2016 9:01 AM To: observium@observium.org Subject: [Observium] Looping PTR request
Hey all,
New to Observium and loving it so far.
I've done some reading, but I can not find an answer to my issue. I would really appreciate some pointers here.
After about 6 hours, i start seeing a looping PTR request:
23:14:37.602724 IP 192.168.155.13.39891 > 8.8.8.8.domain: 54851+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.604162 IP 192.168.155.13.53879 > 8.8.8.8.domain: 58763+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.605541 IP 192.168.155.13.52487 > 8.8.8.8.domain: 30717+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.606941 IP 192.168.155.13.34473 > 8.8.8.8.domain: 5645+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.608351 IP 192.168.155.13.45956 > 8.8.8.8.domain: 5585+ PTR? 30.151.168.192.in-addr.arpa. (45) 23:14:37.609741 IP 192.168.155.13.39834 > 8.8.8.8.domain: 4414+ PTR? 30.151.168.192.in-addr.arpa. (45) 23:14:37.611099 IP 192.168.155.13.35512 > 8.8.8.8.domain: 3031+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.612479 IP 192.168.155.13.49633 > 8.8.8.8.domain: 12185+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.613858 IP 192.168.155.13.59192 > 8.8.8.8.domain: 1926+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.615309 IP 192.168.155.13.51916 > 8.8.8.8.domain: 36203+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.616667 IP 192.168.155.13.49695 > 8.8.8.8.domain: 28584+ PTR? 30.151.168.192.in-addr.arpa. (45) 23:14:37.618245 IP 192.168.155.13.50620 > 8.8.8.8.domain: 13819+ PTR? 15.155.168.192.in-addr.arpa. (45) 23:14:37.619849 IP 192.168.155.13.42869 > 8.8.8.8.domain: 48052+ PTR? 15.155.168.192.in-addr.arpa. (45)
Manual lookup: root@freshwater ~]host 192.168.155.15 Host 15.155.168.192.in-addr.arpa. not found: 3(NXDOMAIN) [root@freshwater ~]# host 192.168.155.15 8.8.8.8 Using domain server: Name: 8.8.8.8 Address: 8.8.8.8#53 Aliases:
Host 15.155.168.192.in-addr.arpa. not found: 3(NXDOMAIN) [root@freshwater ~]#
I'm not running PTR on these IPs and have turned off discovery (i think) (i dont need it)
From my config: $config['autodiscovery']['ip_nets'] = array(" http://127.0.0.0/8 MailScanner warning: numerical links are often malicious: 127.0.0.0/8"); $config['autodiscovery']['xdp'] = FALSE; $config['autodiscovery']['ospf'] = FALSE; $config['autodiscovery']['bgp'] = FALSE; $config['autodiscovery']['libvirt'] = FALSE; $config['autodiscovery']['snmpscan'] = FALSE; $config['enable_printers'] = 0; $config['enable_sla'] = 0; $config['enable_ports_junoseatmvp'] = 0; $config['enable_ports_adsl'] = 1; $config['ignore_mount_optical'] = 1; $config['poller_modules']['unix-agent'] = 1;
It's running on: [root@freshwater observium]# httpd -v Server version: Apache/2.4.6 (CentOS) Server built: May 12 2016 10:27:23 [root@freshwater observium]# php -v PHP 5.4.16 (cli) (built: May 12 2016 13:45:17) Copyright (c) 1997-2013 The PHP Group Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies [root@freshwater observium]#
And a restart to apache does not kill this process. I can't seem to find what is doing it, but the ONLY thing on this server is Observium and nothing else. It's possible it isnt, but it seems to be :(
The only way to stop this is to restart the whole server for some reason
ps auxf from when the loop is happening and after an apache restart
root 6299 0.0 0.0 0 0 ? S 23:43 0:00 _ [kworker/3:1] root 1 0.0 0.1 190528 5564 ? Ss Jun14 1:00 /usr/lib/systemd/systemd --switched-root --system --deserialize 21 root 436 0.0 0.1 36820 4856 ? Ss Jun14 8:01 /usr/lib/systemd/systemd-journald root 458 0.0 0.0 118480 1260 ? Ss Jun14 0:00 /usr/sbin/lvmetad -f root 475 0.0 0.0 44868 3464 ? Ss Jun14 0:00 /usr/lib/systemd/systemd-udevd root 541 0.0 0.0 116724 1632 ? S<sl Jun14 0:02 /sbin/auditd -n root 564 0.0 0.0 19312 1260 ? Ss Jun14 0:27 /usr/sbin/irqbalance --foreground root 566 0.0 0.1 249796 7616 ? Ss Jun14 7:00 /usr/bin/vmtoolsd root 572 0.0 0.1 391788 5588 ? Ssl Jun14 0:39 /usr/sbin/rsyslogd -n root 573 0.0 0.0 26400 1744 ? Ss Jun14 0:18 /usr/lib/systemd/systemd-logind dbus 574 0.0 0.0 26724 1828 ? Ss Jun14 0:36 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation chrony 576 0.0 0.0 115844 1900 ? S Jun14 0:02 /usr/sbin/chronyd root 582 0.0 0.0 203368 1240 ? Ssl Jun14 0:00 /usr/sbin/gssproxy -D root 592 0.0 0.2 434960 8368 ? Ssl Jun14 0:12 /usr/sbin/NetworkManager --no-daemon polkitd 607 0.0 0.3 527584 12132 ? Ssl Jun14 0:07 /usr/lib/polkit-1/polkitd --no-debug root 824 0.5 0.7 251120 27696 ? Rs Jun14 48:26 /usr/sbin/snmptrapd -Lsd -f root 825 0.0 0.0 82560 3616 ? Ss Jun14 0:00 /usr/sbin/sshd -D root 23255 0.0 0.1 143416 5544 ? Ss 21:39 0:00 _ sshd: root@pts/0 root 23257 0.0 0.0 115384 2060 pts/0 Ss+ 21:39 0:00 | _ -bash root 25194 0.0 0.1 143808 5928 ? Ss 23:08 0:00 _ sshd: root@pts/1 root 25196 0.0 0.0 115384 2112 pts/1 Ss 23:08 0:00 _ -bash root 8186 0.0 0.0 151168 1960 pts/1 R+ 23:46 0:00 _ ps auxf root 829 0.0 0.0 29304 1004 ? Ss Jun14 0:00 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid root 831 0.0 0.2 222956 10612 ? Ss Jun14 4:36 /usr/sbin/snmpd -LS0-6d -f root 836 0.0 0.3 552452 15072 ? Ssl Jun14 0:00 /usr/sbin/libvirtd root 841 0.0 0.0 126332 1572 ? Ss Jun14 0:02 /usr/sbin/crond -n root 851 0.0 0.0 110036 852 tty1 Ss+ Jun14 0:00 /sbin/agetty --noclear tty1 linux mysql 1171 0.0 0.0 113256 1580 ? Ss Jun14 0:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr mysql 1600 0.4 5.9 1695560 232072 ? Sl Jun14 35:42 _ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/ root 2159 0.0 0.0 91140 2164 ? Ss Jun14 0:02 /usr/libexec/postfix/master -w postfix 2161 0.0 0.1 91420 4132 ? S Jun14 0:00 _ qmgr -l -t unix -u postfix 10160 0.0 0.1 91244 3924 ? S 22:30 0:00 _ pickup -l -t unix -u root 29025 0.0 0.3 396460 15080 ? Ss 23:16 0:00 /usr/sbin/httpd -DFOREGROUND apache 29027 0.0 0.3 397296 12784 ? S 23:16 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 29029 0.0 0.3 397556 12816 ? S 23:16 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 29031 0.0 0.2 396596 8644 ? S 23:16 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 29035 0.0 0.3 397296 12812 ? S 23:18 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 29039 0.0 0.3 397556 12816 ? S 23:18 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 29041 0.0 0.3 397300 12872 ? S 23:18 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 351 0.0 0.2 396596 8644 ? S 23:28 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 2450 0.0 0.2 396596 8644 ? S 23:33 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 4385 0.0 0.2 396596 8644 ? S 23:39 0:00 _ /usr/sbin/httpd -DFOREGROUND apache 6301 0.0 0.1 396596 7652 ? S 23:44 0:00 _ /usr/sbin/httpd -DFOREGROUND [root@freshwater observium]#
Any help appreciated
Thanks!
participants (2)
-
Ron Marosko -
TAN Lists