Thanks for clues in troubleshooting a problem with high DNS query rates from Observium.
I've been looking at Observium for network monitoring, and really like it so far, except that makes repeated DNS queries at a very high rate.
For example, an Observium box monitors an old Cisco 2651XM router in the same physical location. Running a capture on the DNS server, we see the Observium box made over 200 queries for the router's hostname in a 7-second interval. Also, the DNS server correctly responded to every one of these, usually within 20 ms.
What would cause so many DNS queries (and apparent non-caching of the responses)? This behavior is thrashing the logs on the DNS servers.
This is with Observium Community Edition 0.13.10.4586 running on Debian Wheezy.
Thanks again
dn
Are you running a caching service on the server where observium is installed?
Sent from my iPhone
On Oct 14, 2013, at 12:44, David Newman dnewman@networktest.com wrote:
Thanks for clues in troubleshooting a problem with high DNS query rates from Observium.
I've been looking at Observium for network monitoring, and really like it so far, except that makes repeated DNS queries at a very high rate.
For example, an Observium box monitors an old Cisco 2651XM router in the same physical location. Running a capture on the DNS server, we see the Observium box made over 200 queries for the router's hostname in a 7-second interval. Also, the DNS server correctly responded to every one of these, usually within 20 ms.
What would cause so many DNS queries (and apparent non-caching of the responses)? This behavior is thrashing the logs on the DNS servers.
This is with Observium Community Edition 0.13.10.4586 running on Debian Wheezy.
Thanks again
dn
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On 10/14/13 12:49 PM, Mehmet Akcin wrote:
Are you running a caching service on the server where observium is installed?
It's a plain-vanilla Debian Wheezy installation, which includes the standard Bind libraries:
$ dpkg -l | grep bind ii bind9-host 1:9.8.4.dfsg.P1-6+nmu2+deb7u1 amd64 Version of 'host' bundled with BIND 9.X ii libbind9-80 1:9.8.4.dfsg.P1-6+nmu2+deb7u1 amd64 BIND9 Shared Library used by BIND ii rpcbind 0.2.0-8 amd64 converts RPC program numbers into universal addresses
I'm hardly a Debian expert, but wouldn't bind cache responses?
Thanks!
dn
Does anyone monitoring juniper ex3300 DC power? Rodrigo Augusto Gestor de T.I. Grupo Connectoway http://www.connectoway.com.br http://www.connectoway.com.br/ http://www.1telecom.com.br http://www.1telecom.com.br/ * rodrigo@connectoway.com.br ( (81) 3366-7376 ( (81) 8184-3646 ( INOC-DBA 52965*100
On 14/10/13 16:50, "Nikolay Shopik" shopik@inblock.ru wrote:
Obviously because every snmpwalk have to resolve same hostname over and over again.
On 14.10.2013 23:44, David Newman wrote:
What would cause so many DNS queries (and apparent non-caching of the responses)? This behavior is thrashing the logs on the DNS servers.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Every time you make a call to get_snmp_*() you also make a calls to DNS via gethostbyaddr() and getaddrbyhost() though not directly by PHP. If you need to relieve stress on your little DNS server you should probably consider running a caching DNS server on your monitoring host. You should probably also consider putting in a DNS server built in this decade.
On 10/14/13 2:44 PM, "David Newman" dnewman@networktest.com wrote:
Thanks for clues in troubleshooting a problem with high DNS query rates from Observium.
I've been looking at Observium for network monitoring, and really like it so far, except that makes repeated DNS queries at a very high rate.
For example, an Observium box monitors an old Cisco 2651XM router in the same physical location. Running a capture on the DNS server, we see the Observium box made over 200 queries for the router's hostname in a 7-second interval. Also, the DNS server correctly responded to every one of these, usually within 20 ms.
What would cause so many DNS queries (and apparent non-caching of the responses)? This behavior is thrashing the logs on the DNS servers.
This is with Observium Community Edition 0.13.10.4586 running on Debian Wheezy.
Thanks again
dn
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On 2013-10-14 21:42, Tim Schuh wrote:
Every time you make a call to get_snmp_*() you also make a calls to DNS via gethostbyaddr() and getaddrbyhost() though not directly by PHP. If you need to relieve stress on your little DNS server you should probably consider running a caching DNS server on your monitoring host. You should probably also consider putting in a DNS server built in this decade.
We don't use the PHP SNMP functions, we fork to the snmp binaries instead.
Same thing though, every SNMP operation will cause a DNS lookup if you're not locally caching DNS queries.
adam.
On 10/14/13 2:06 PM, Adam Armstrong wrote:
On 2013-10-14 21:42, Tim Schuh wrote:
Every time you make a call to get_snmp_*() you also make a calls to DNS via gethostbyaddr() and getaddrbyhost() though not directly by PHP. If you need to relieve stress on your little DNS server you should probably consider running a caching DNS server on your monitoring host. You should probably also consider putting in a DNS server built in this decade.
We don't use the PHP SNMP functions, we fork to the snmp binaries instead.
Same thing though, every SNMP operation will cause a DNS lookup if you're not locally caching DNS queries.
That's really my question...what to change on the local resolver or Observium itself to cache responses?
Asking the same question hundreds of times in a few seconds is just silly.
Thanks!
dn
On 10/14/13 1:42 PM, Tim Schuh wrote:
Every time you make a call to get_snmp_*() you also make a calls to DNS via gethostbyaddr() and getaddrbyhost() though not directly by PHP. If you need to relieve stress on your little DNS server you should probably consider running a caching DNS server on your monitoring host. You should probably also consider putting in a DNS server built in this decade.
Not that you bothered to ask, but this is Bind 9.9.4, which is the latest release, and pre-production testing shows this DNS server can handle A queries at rates somewhere between 1000x and 10000x what Observium is doing.
In this particular case, the constraint is local log storage, and all those Observium queries are chewing through that pretty quickly.
dn
Hi David,
As explained, in the email trail : 1. Observium executes a binary called snmpwalk for each host you check. 2. Each time, snmpwalk is run separately (fork). Consequently it is not aware of any previous DNS requests. 3. Snmpwalk uses DNS lookups to resolve the name to IP(s).
Your solution, already mentioned, is to install a local DNS caching service like : apt-get install unscd then look at /etc/nscd.conf. As a result the system will locally cache any DNS lookups. It's a caching daemon. It avoids a lot of consecutive requests by caching the 1st answer but if you update a DNS entry you may have to restart nscd (aka empty the cache) if you cannot wait for the cache to expire.
I am not sure what you mean by 'local log storage'. If it logs too much in the logs then maybe decrease the log level (if acceptable) or *drumroll* modify the code. Good luck.
!google nscd !google linux dns caching
Thomas
On 10/14/13 8:07 PM, Thomas Guthmann wrote:
Hi David,
As explained, in the email trail :
- Observium executes a binary called snmpwalk for each host you check.
- Each time, snmpwalk is run separately (fork). Consequently it is not aware of any previous DNS requests.
- Snmpwalk uses DNS lookups to resolve the name to IP(s).
Your solution, already mentioned, is to install a local DNS caching service like : apt-get install unscd then look at /etc/nscd.conf. As a result the system will locally cache any DNS lookups.
It's a
caching daemon. It avoids a lot of consecutive requests by caching the 1st answer but if you update a DNS entry you may have to restart nscd (aka empty the cache) if you cannot wait for the cache to expire.
Thanks all who replied.
The root problem is me being relatively new to Debian and not understanding that a basic install does not include a caching resolver.
I am not sure what you mean by 'local log storage'. If it logs too much in the logs then maybe decrease the log level (if acceptable) or *drumroll* modify the code. Good luck.
In this case the DNS service runs on an embedded machine with limited local storage. I probably will change to a central syslog server.
Mostly I was just curious why Observium would produce something like 3 Mbytes of DNS log entries per hour for fewer than 20 devices using low-verbosity logging settings.
Thanks again.
dn
!google nscd !google linux dns caching
Thomas _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On 2013-10-15 18:24, David Newman wrote:
On 10/14/13 8:07 PM, Thomas Guthmann wrote: Hi David,
Mostly I was just curious why Observium would produce something like 3 Mbytes of DNS log entries per hour for fewer than 20 devices using low-verbosity logging settings.
Why on earth would anyone log DNS requests?
adam.
On 10/15/13 10:50 AM, Adam Armstrong wrote:
Why on earth would anyone log DNS requests?
Interesting question on a monitoring forum.
Reasons include capacity planning, troubleshooting (especially important with DNSSEC), tracking top domains, top RR types, top requesters, v4/v6 breakdown, and those are only the ones I can think of offhand.
There are certainly other ways to get some of this, such as DNS MIB extensions or the excellent dnstop tool, but logs can be useful too, especially where details are essential.
dn
On 2013-10-15 19:17, David Newman wrote:
On 10/15/13 10:50 AM, Adam Armstrong wrote:
Why on earth would anyone log DNS requests?
Interesting question on a monitoring forum.
Reasons include capacity planning, troubleshooting (especially important with DNSSEC), tracking top domains, top RR types, top requesters, v4/v6 breakdown, and those are only the ones I can think of offhand.
Why would someone who believes any of the above (rather than invoking them as excuses) to be important be running their DNS off a tiny embedded device with no space for logging?
There are certainly other ways to get some of this, such as DNS MIB extensions or the excellent dnstop tool, but logs can be useful too, especially where details are essential.
You seem to be trying to invent a problem and then trying to blame us for it.
Beware that I have no tolerance for such time wasting nonsense.
adam.
I never said it was a lightweight server did I. Quad core i7 with a boatload of RAM and raid storage for the actual BIND servers
Typographical errors courtesy of iPhone
On Oct 15, 2013, at 16:54, "Adam Armstrong" adama@memetic.org wrote:
On 2013-10-15 19:17, David Newman wrote:
On 10/15/13 10:50 AM, Adam Armstrong wrote: Why on earth would anyone log DNS requests? Interesting question on a monitoring forum. Reasons include capacity planning, troubleshooting (especially important with DNSSEC), tracking top domains, top RR types, top requesters, v4/v6 breakdown, and those are only the ones I can think of offhand.
Why would someone who believes any of the above (rather than invoking them as excuses) to be important be running their DNS off a tiny embedded device with no space for logging?
There are certainly other ways to get some of this, such as DNS MIB extensions or the excellent dnstop tool, but logs can be useful too, especially where details are essential.
You seem to be trying to invent a problem and then trying to blame us for it.
Beware that I have no tolerance for such time wasting nonsense.
adam. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On 10/15/13 1:37 PM, Adam Armstrong wrote:
On 2013-10-15 19:17, David Newman wrote:
On 10/15/13 10:50 AM, Adam Armstrong wrote:
Why on earth would anyone log DNS requests?
Interesting question on a monitoring forum.
Reasons include capacity planning, troubleshooting (especially important with DNSSEC), tracking top domains, top RR types, top requesters, v4/v6 breakdown, and those are only the ones I can think of offhand.
Why would someone who believes any of the above (rather than invoking them as excuses) to be important be running their DNS off a tiny embedded device with no space for logging?
Invoking excuses? No, I'm really not.
You asked a question, and I answered it.
There are certainly other ways to get some of this, such as DNS MIB extensions or the excellent dnstop tool, but logs can be useful too, especially where details are essential.
You seem to be trying to invent a problem and then trying to blame us for it.
Sorry that you perceive it that way. I'm not blaming Observium for anything. As I said in the OP, I *like* what I've seen of your "stinky turd" thus far.
Per an earlier email, the root problem was me being relatively new to Debian, and not understanding that (unlike the BSDs, where I have more experience) a caching resolver isn't part of the default install.
I'm a newcomer to this tool and am trying to understand how it works. I've got at least one other question I don't see covered in the docs or in the FAQ, but now am reluctant to ask because you seem rather hostile.
Again, though, I *like* what I've seen so far.
dn
Beware that I have no tolerance for such time wasting nonsense.
adam. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On 16/10/2013 23:06, David Newman wrote:
Sorry that you perceive it that way. I'm not blaming Observium for anything. As I said in the OP, I *like* what I've seen of your "stinky turd" thus far. Per an earlier email, the root problem was me being relatively new to Debian, and not understanding that (unlike the BSDs, where I have more experience) a caching resolver isn't part of the default install. I'm a newcomer to this tool and am trying to understand how it works. I've got at least one other question I don't see covered in the docs or in the FAQ, but now am reluctant to ask because you seem rather hostile. Again, though, I *like* what I've seen so far. dn
If you can show you looked in the docs, and it really isn't there, ask away!
Tom
On 17/10/13 6:19 PM, "Tom Laermans" tom.laermans@powersource.cx wrote:
If you can show you looked in the docs^H^H^H^H source, and it really isn't there, ask away!
Fixed that for you :)
'ello all,
Can't find this in the manual or faq page, the Deviecs page menu has changed, for example, we used to have (and now I cant fully remember) :
--- Page URL is : /devices/type=network Ports Basic Details Status Bits (etc). ---
If we want to set a default "Bits" for the graphs, which is how we used to have it, we currently have to click Graphs and scroll down to Bits, is there a default setting that can go into the config file or something else somewhere else I can tweak ?
I did find this in the wiki :
--- $config['ports_page_default'] = "details/"; ## eg "details/" "graphs/bits/" Defines the default view on the /device/*/ports/ page. ---
So the question is there a switch similar to the above for the /network/ page that I can switch on and set as a default ?
Kind regards,
Chris
Observium works fantastically with every piece of network hardware I've thrown at it. It works with my ESXi hosts. It works with my NAS4Free hosts. It works with our Arista switches, it even picks up BGP adjacency with no convoluted configuration. It works with my DD-WRT-based router. It event works with my crappy NetGear switch. I bet it will work just fine with the ZyXel that's on the way. Its *way* more polished than multi-million dollar software I used at previous employers and vastly superior. I just don't see Bind anywhere on the hardware compatibility list.
On 10/15/13 1:17 PM, "David Newman" dnewman@networktest.com wrote:
On 10/15/13 10:50 AM, Adam Armstrong wrote:
Why on earth would anyone log DNS requests?
Interesting question on a monitoring forum.
Reasons include capacity planning, troubleshooting (especially important with DNSSEC), tracking top domains, top RR types, top requesters, v4/v6 breakdown, and those are only the ones I can think of offhand.
There are certainly other ways to get some of this, such as DNS MIB extensions or the excellent dnstop tool, but logs can be useful too, especially where details are essential.
dn
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On 2013-10-15 22:48, Tim Schuh wrote:
fine with the ZyXel that's on the way. Its *way* more polished than multi-million dollar software I used at previous employers and vastly
I still think of Observium as a big giant stinky turd. It's so funny when you guys who've been forced to use even worse things think it's 'polished'.
:D
adam.
The MythBusters have already shown that you can polish a turd. :)
On 10/15/13 4:57 PM, "Adam Armstrong" adama@memetic.org wrote:
On 2013-10-15 22:48, Tim Schuh wrote:
fine with the ZyXel that's on the way. Its *way* more polished than multi-million dollar software I used at previous employers and vastly
I still think of Observium as a big giant stinky turd. It's so funny when you guys who've been forced to use even worse things think it's 'polished'.
:D
adam. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Tim,
Bind support is here... (you may ask for ISC Bind ?) on the observium agent.
Work like a charm (and very usefull to see some nasty thing happening on resolvers using bind or even unbound).
-- Xavier Beaudouin - Senior Network and System Administrator Infrastructure and network director Horizon Software - http://www.hsoftware.com/ 13 rue La Fayette - 75009 PARIS - France Phone: +33 (0)1 4260 9490 Fax: +33 (0)1 44 56 97 01
----- Mail original -----
De: "Tim Schuh" tschuh@vmware.com À: "Observium Network Observation System" observium@observium.org Envoyé: Mardi 15 Octobre 2013 23:48:53 Objet: Re: [Observium] why is the DNS query rate so high?
Observium works fantastically with every piece of network hardware I've thrown at it. It works with my ESXi hosts. It works with my NAS4Free hosts. It works with our Arista switches, it even picks up BGP adjacency with no convoluted configuration. It works with my DD-WRT-based router. It event works with my crappy NetGear switch. I bet it will work just fine with the ZyXel that's on the way. Its *way* more polished than multi-million dollar software I used at previous employers and vastly superior. I just don't see Bind anywhere on the hardware compatibility list.
On 10/15/13 1:17 PM, "David Newman" dnewman@networktest.com wrote:
On 10/15/13 10:50 AM, Adam Armstrong wrote:
Why on earth would anyone log DNS requests?
Interesting question on a monitoring forum.
Reasons include capacity planning, troubleshooting (especially important with DNSSEC), tracking top domains, top RR types, top requesters, v4/v6 breakdown, and those are only the ones I can think of offhand.
There are certainly other ways to get some of this, such as DNS MIB extensions or the excellent dnstop tool, but logs can be useful too, especially where details are essential.
dn
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Well. I used to keep about two gig of query logs per server when I did DNS
Used it for usage pattern analysis for the incoming anycast traffic and for DNS amplifier attack mitigation. Sometimes stuff inside the perimeter I a trouble maker as well
Alex
Typographical errors courtesy of iPhone
On Oct 15, 2013, at 14:09, "Adam Armstrong" adama@memetic.org wrote:
On 2013-10-15 18:24, David Newman wrote:
On 10/14/13 8:07 PM, Thomas Guthmann wrote: Hi David,
Mostly I was just curious why Observium would produce something like 3 Mbytes of DNS log entries per hour for fewer than 20 devices using low-verbosity logging settings.
Why on earth would anyone log DNS requests?
adam. _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
On 15/10/2013 19:50, Adam Armstrong wrote:
On 2013-10-15 18:24, David Newman wrote:
On 10/14/13 8:07 PM, Thomas Guthmann wrote: Hi David,
Mostly I was just curious why Observium would produce something like 3 Mbytes of DNS log entries per hour for fewer than 20 devices using low-verbosity logging settings.
Why on earth would anyone log DNS requests?
I used to log all queries for my AS112 node; for stats. Never fully logged a recursor though. Sounds like it chews through disk like Observium through iops... ;-)
Tom
participants (12)
-
Adam Armstrong -
Alex Latzko -
Chris Pelling -
David Newman -
Mehmet Akcin -
Nikolay Shopik -
Peter Childs -
Rodrigo Augusto -
Thomas Guthmann -
Tim Schuh -
Tom Laermans -
Xavier Beaudouin