Syslogging & Expansion
Hi Everyone,
We are considering the possibility of using Observium's syslog capability to handle the syslog output of 50 machines which will be spitting out roughly 300K lines of log per hour for a grand total in the ball park 360,000,000 log entries per 24 hours.
Has anyone used/attempted to handle this volume of logging with Observium in the past? If so, would you mind sharing your experience?
I am also looking for feedback on hardware suggestions for both the Observium machine as well for the standalone database server.
Any feedback is appreciated!
Thanks,
-Lane
You really want to use the “ELK” based logging with a scale-out infrastructure for this volume.
You can start here https://blog.devita.co/2014/09/04/monitoring-pfsense-firewall-logs-with-elk-...
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Lane Eckley Sent: Monday, November 10, 2014 03:04 PM To: Observium Network Observation System Subject: [Observium] Syslogging & Expansion
Hi Everyone,
We are considering the possibility of using Observium's syslog capability to handle the syslog output of 50 machines which will be spitting out roughly 300K lines of log per hour for a grand total in the ball park 360,000,000 log entries per 24 hours.
Has anyone used/attempted to handle this volume of logging with Observium in the past? If so, would you mind sharing your experience?
I am also looking for feedback on hardware suggestions for both the Observium machine as well for the standalone database server.
Any feedback is appreciated!
Thanks,
-Lane
Thanks!
I am not aware of a way to integrate it with Observium though which is rather unfortunate as that's the primary reason of not wanting to go with something like splunk,etc.
Maybe I am missing something? On Nov 10, 2014 3:59 PM, "Joseph L. Brunner" joe@affirmedsystems.com wrote:
You really want to use the “ELK” based logging with a scale-out infrastructure for this volume.
You can start here https://blog.devita.co/2014/09/04/monitoring-pfsense-firewall-logs-with-elk-...
*From:* observium [mailto:observium-bounces@observium.org] *On Behalf Of *Lane Eckley *Sent:* Monday, November 10, 2014 03:04 PM *To:* Observium Network Observation System *Subject:* [Observium] Syslogging & Expansion
Hi Everyone,
We are considering the possibility of using Observium's syslog capability to handle the syslog output of 50 machines which will be spitting out roughly 300K lines of log per hour for a grand total in the ball park 360,000,000 log entries per 24 hours.
Has anyone used/attempted to handle this volume of logging with Observium in the past? If so, would you mind sharing your experience?
I am also looking for feedback on hardware suggestions for both the Observium machine as well for the standalone database server.
Any feedback is appreciated!
Thanks,
-Lane
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
With that huge amount of logs, you might require a dedicated server (or more) for logging. I have been testing ELMA, which is a complete Linux distribution (based on Suse) that has very similar components to what Joseph described.
http://enterprise-log-management-appliance.org/
Cheers,
Tristan
*Tristan Rhodes* Network Engineer Weber State University 801.626.8549
On Mon, Nov 10, 2014 at 2:03 PM, Lane Eckley lane@staff.hypernia.com wrote:
Thanks!
I am not aware of a way to integrate it with Observium though which is rather unfortunate as that's the primary reason of not wanting to go with something like splunk,etc.
Maybe I am missing something? On Nov 10, 2014 3:59 PM, "Joseph L. Brunner" joe@affirmedsystems.com wrote:
You really want to use the “ELK” based logging with a scale-out infrastructure for this volume.
You can start here https://blog.devita.co/2014/09/04/monitoring-pfsense-firewall-logs-with-elk-...
*From:* observium [mailto:observium-bounces@observium.org] *On Behalf Of *Lane Eckley *Sent:* Monday, November 10, 2014 03:04 PM *To:* Observium Network Observation System *Subject:* [Observium] Syslogging & Expansion
Hi Everyone,
We are considering the possibility of using Observium's syslog capability to handle the syslog output of 50 machines which will be spitting out roughly 300K lines of log per hour for a grand total in the ball park 360,000,000 log entries per 24 hours.
Has anyone used/attempted to handle this volume of logging with Observium in the past? If so, would you mind sharing your experience?
I am also looking for feedback on hardware suggestions for both the Observium machine as well for the standalone database server.
Any feedback is appreciated!
Thanks,
-Lane
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
We are currently testing Graylog2 on dedicated hardware. Our Observium instance wasn’t able to take the extra load and IO.
Would be nice to integrate the Syslog and Observium servers…
Adriaan Smuts
Systems Administrator - Windows
________________________________
Direct Line:
+27 21 464 9565
Reception:
086 000 9500
www.webafrica.co.zahttp://www.webafrica.co.za/
[http://shared.webafrica.co.za/images/signature/signature_logo.png]
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Tristan Rhodes Sent: 10 November 2014 11:07 PM To: Observium Network Observation System Subject: Re: [Observium] Syslogging & Expansion
With that huge amount of logs, you might require a dedicated server (or more) for logging. I have been testing ELMA, which is a complete Linux distribution (based on Suse) that has very similar components to what Joseph described.
http://enterprise-log-management-appliance.org/
Cheers,
Tristan
Tristan Rhodes Network Engineer Weber State University 801.626.8549 [http://www.weber.edu/wsuimages/brand/templates/emailsig_sig1.jpg]
On Mon, Nov 10, 2014 at 2:03 PM, Lane Eckley <lane@staff.hypernia.commailto:lane@staff.hypernia.com> wrote:
Thanks!
I am not aware of a way to integrate it with Observium though which is rather unfortunate as that's the primary reason of not wanting to go with something like splunk,etc.
Maybe I am missing something? On Nov 10, 2014 3:59 PM, "Joseph L. Brunner" <joe@affirmedsystems.commailto:joe@affirmedsystems.com> wrote: You really want to use the “ELK” based logging with a scale-out infrastructure for this volume.
You can start here https://blog.devita.co/2014/09/04/monitoring-pfsense-firewall-logs-with-elk-...
From: observium [mailto:observium-bounces@observium.orgmailto:observium-bounces@observium.org] On Behalf Of Lane Eckley Sent: Monday, November 10, 2014 03:04 PM To: Observium Network Observation System Subject: [Observium] Syslogging & Expansion
Hi Everyone,
We are considering the possibility of using Observium's syslog capability to handle the syslog output of 50 machines which will be spitting out roughly 300K lines of log per hour for a grand total in the ball park 360,000,000 log entries per 24 hours.
Has anyone used/attempted to handle this volume of logging with Observium in the past? If so, would you mind sharing your experience?
I am also looking for feedback on hardware suggestions for both the Observium machine as well for the standalone database server.
Any feedback is appreciated!
Thanks,
-Lane
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
It would be, but it's kinda beyond my skill with MySQL to make it efficient, and beyond my skill to parse syslog in the way I'd want it to be done. :)
adam.
------ Original Message ------ From: "Adriaan Smuts" adriaan.smuts@webafrica.com To: "Observium Network Observation System" observium@observium.org Sent: 11/10/2014 3:11:43 PM Subject: Re: [Observium] Syslogging & Expansion
We are currently testing Graylog2 on dedicated hardware. Our Observium instance wasn’t able to take the extra load and IO.
Would be nice to integrate the Syslog and Observium servers…
Adriaan Smuts
Systems Administrator - Windows
Direct Line:
+27 21 464 9565
Reception:
086 000 9500
www.webafrica.co.za
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Tristan Rhodes Sent: 10 November 2014 11:07 PM To: Observium Network Observation System Subject: Re: [Observium] Syslogging & Expansion
With that huge amount of logs, you might require a dedicated server (or more) for logging. I have been testing ELMA, which is a complete Linux distribution (based on Suse) that has very similar components to what Joseph described.
http://enterprise-log-management-appliance.org/
Cheers,
Tristan
Tristan Rhodes Network Engineer
Weber State University
801.626.8549
On Mon, Nov 10, 2014 at 2:03 PM, Lane Eckley lane@staff.hypernia.com wrote:
Thanks!
I am not aware of a way to integrate it with Observium though which is rather unfortunate as that's the primary reason of not wanting to go with something like splunk,etc.
Maybe I am missing something?
On Nov 10, 2014 3:59 PM, "Joseph L. Brunner" joe@affirmedsystems.com wrote:
You really want to use the “ELK” based logging with a scale-out infrastructure for this volume.
You can start here https://blog.devita.co/2014/09/04/monitoring-pfsense-firewall-logs-with-elk-...
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Lane Eckley Sent: Monday, November 10, 2014 03:04 PM To: Observium Network Observation System Subject: [Observium] Syslogging & Expansion
Hi Everyone,
We are considering the possibility of using Observium's syslog capability to handle the syslog output of 50 machines which will be spitting out roughly 300K lines of log per hour for a grand total in the ball park 360,000,000 log entries per 24 hours.
Has anyone used/attempted to handle this volume of logging with Observium in the past? If so, would you mind sharing your experience?
I am also looking for feedback on hardware suggestions for both the Observium machine as well for the standalone database server.
Any feedback is appreciated!
Thanks,
-Lane
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Our syslog handling capability is a little bit brain-dead and scales poorly. There are probably better tools around for doing this... :)
adam.
------ Original Message ------ From: "Lane Eckley" lane@staff.hypernia.com To: "Observium Network Observation System" observium@observium.org Sent: 11/10/2014 2:03:44 PM Subject: [Observium] Syslogging & Expansion
Hi Everyone,
We are considering the possibility of using Observium's syslog capability to handle the syslog output of 50 machines which will be spitting out roughly 300K lines of log per hour for a grand total in the ball park 360,000,000 log entries per 24 hours.
Has anyone used/attempted to handle this volume of logging with Observium in the past? If so, would you mind sharing your experience?
I am also looking for feedback on hardware suggestions for both the Observium machine as well for the standalone database server.
Any feedback is appreciated!
Thanks,
-Lane
participants (5)
-
Adam Armstrong -
Adriaan Smuts -
Joseph L. Brunner -
Lane Eckley -
Tristan Rhodes