Recommendation to support 20K devices on observium
Good morning everyone,
We're evaluating to use observium as hardware monitoring devices based on snmp protocol for devices located all around the world.
We're talking about 20K devices and I believe that have one single host locate in SSF, getting all the info from all the devices, located on places like Shangai may not be the best scenario to get the best performance from the tool itself
Does observium support the concept of Master of masters?, I mean have local master observium hosts (EMEA, APAC, NA) and consolidate all the info in one single Master Observium Instance
If not what will be a recommended approach to cover an scenario like the one described
Thanks in advance for your help and collaboration
Kind Regards
It all depends very much on the types of devices. 20k 500 port switches isn't going to happen. 20k Linux services is probably pretty easy.
adam.
Sent from Mailbird [http://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm...] On 15/03/2016 16:41:13, Ruiz, Carlos carlos.ruiz.cr2@roche.com wrote: Good morning everyone,
We're evaluating to use observium as hardware monitoring devices based on snmp protocol for devices located all around the world.
We're talking about 20K devices and I believe that have one single host locate in SSF, getting all the info from all the devices, located on places like Shangai may not be the best scenario to get the best performance from the tool itself
Does observium support the concept of Master of masters?, I mean have local master observium hosts (EMEA, APAC, NA) and consolidate all the info in one single Master Observium Instance
If not what will be a recommended approach to cover an scenario like the one described
Thanks in advance for your help and collaboration
Kind Regards --
Carlos Ruiz Compute System Management Senior Engineer
Roche Farma, S.A.Josefa Valcarcel, 40. ES-28027 Madrid (Spain)
Phone: +34 91 305 99 25 Mobile: +34 662 10 46 14
* *
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi,
The problem is that we've a huge variety of device models worldwide and we'll discover most of the models when we start with the roll-out
Let me remake the question, is observium ready to work on a master of masters architecture design?
Thanks in advance for your help
Kind Regards.
2016-03-15 17:54 GMT+01:00 Adam Armstrong adama@memetic.org:
It all depends very much on the types of devices. 20k 500 port switches isn't going to happen. 20k Linux services is probably pretty easy.
adam.
Sent from Mailbird http://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm_campaign=sent-from-mailbird
On 15/03/2016 16:41:13, Ruiz, Carlos carlos.ruiz.cr2@roche.com wrote: Good morning everyone,
We're evaluating to use observium as hardware monitoring devices based on snmp protocol for devices located all around the world.
We're talking about 20K devices and I believe that have one single host locate in SSF, getting all the info from all the devices, located on places like Shangai may not be the best scenario to get the best performance from the tool itself
Does observium support the concept of Master of masters?, I mean have local master observium hosts (EMEA, APAC, NA) and consolidate all the info in one single Master Observium Instance
If not what will be a recommended approach to cover an scenario like the one described
Thanks in advance for your help and collaboration
Kind Regards
Carlos Ruiz Compute System Management Senior Engineer
Roche Farma, S.A.Josefa Valcarcel, 40. ES-28027 Madrid (Spain)
Phone: +34 91 305 99 25 Mobile: +34 662 10 46 14
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Maybe on the master node set up every device but disable polling; child nodes poll a certain pool of devices, then have a cron script that copies rrd files to the master?
Or is that not enough to trick Observium on the master to think it has gotten all that data by itself?
On Wed, Mar 16, 2016 at 7:56 AM, Ruiz, Carlos carlos.ruiz.cr2@roche.com wrote:
Hi,
The problem is that we've a huge variety of device models worldwide and we'll discover most of the models when we start with the roll-out
Let me remake the question, is observium ready to work on a master of masters architecture design?
Thanks in advance for your help
Kind Regards.
2016-03-15 17:54 GMT+01:00 Adam Armstrong adama@memetic.org:
It all depends very much on the types of devices. 20k 500 port switches isn't going to happen. 20k Linux services is probably pretty easy.
adam.
Sent from Mailbird http://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm_campaign=sent-from-mailbird
On 15/03/2016 16:41:13, Ruiz, Carlos carlos.ruiz.cr2@roche.com wrote: Good morning everyone,
We're evaluating to use observium as hardware monitoring devices based on snmp protocol for devices located all around the world.
We're talking about 20K devices and I believe that have one single host locate in SSF, getting all the info from all the devices, located on places like Shangai may not be the best scenario to get the best performance from the tool itself
Does observium support the concept of Master of masters?, I mean have local master observium hosts (EMEA, APAC, NA) and consolidate all the info in one single Master Observium Instance
If not what will be a recommended approach to cover an scenario like the one described
Thanks in advance for your help and collaboration
Kind Regards
Carlos Ruiz Compute System Management Senior Engineer
Roche Farma, S.A.Josefa Valcarcel, 40. ES-28027 Madrid (Spain)
Phone: +34 91 305 99 25 Mobile: +34 662 10 46 14
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Carlos Ruiz Compute System Management Senior Engineer
Roche Farma, S.A.Josefa Valcarcel, 40. ES-28027 Madrid (Spain)
Phone: +34 91 305 99 25 Mobile: +34 662 10 46 14
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
You need to keep the database up to date, too :)
Adam.
Sent from BlueMail
On Mar 16, 2016, 13:04, at 13:04, Jason Vanlandingham van.sjason@gmail.com wrote:
Maybe on the master node set up every device but disable polling; child nodes poll a certain pool of devices, then have a cron script that copies rrd files to the master?
Or is that not enough to trick Observium on the master to think it has gotten all that data by itself?
On Wed, Mar 16, 2016 at 7:56 AM, Ruiz, Carlos carlos.ruiz.cr2@roche.com wrote:
Hi,
The problem is that we've a huge variety of device models worldwide
and
we'll discover most of the models when we start with the roll-out
Let me remake the question, is observium ready to work on a master of masters architecture design?
Thanks in advance for your help
Kind Regards.
2016-03-15 17:54 GMT+01:00 Adam Armstrong adama@memetic.org:
It all depends very much on the types of devices. 20k 500 port
switches
isn't going to happen. 20k Linux services is probably pretty easy.
adam.
Sent from Mailbird
http://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm_campaign=sent-from-mailbird
On 15/03/2016 16:41:13, Ruiz, Carlos carlos.ruiz.cr2@roche.com
wrote:
Good morning everyone,
We're evaluating to use observium as hardware monitoring devices
based on
snmp protocol for devices located all around the world.
We're talking about 20K devices and I believe that have one single
host
locate in SSF, getting all the info from all the devices, located on
places
like Shangai may not be the best scenario to get the best
performance from
the tool itself
Does observium support the concept of Master of masters?, I mean
have
local master observium hosts (EMEA, APAC, NA) and consolidate all
the info
in one single Master Observium Instance
If not what will be a recommended approach to cover an scenario like
the
one described
Thanks in advance for your help and collaboration
Kind Regards
Carlos Ruiz Compute System Management Senior Engineer
Roche Farma, S.A.Josefa Valcarcel, 40. ES-28027 Madrid (Spain)
Phone: +34 91 305 99 25 Mobile: +34 662 10 46 14
_______________________________________________ observium mailing
list
observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Carlos Ruiz Compute System Management Senior Engineer
Roche Farma, S.A.Josefa Valcarcel, 40. ES-28027 Madrid (Spain)
Phone: +34 91 305 99 25 Mobile: +34 662 10 46 14
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Not to mention the fact that RRDs are pretty big, en masse.
adam.
Sent from Mailbird [http://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm...] On 16/03/2016 13:14:23, Adam Armstrong adama@memetic.org wrote: You need to keep the database up to date, too :) Adam. Sent from BlueMail [http://www.bluemail.me/r]
On Mar 16, 2016, at 13:04, Jason Vanlandingham <van.sjason@gmail.com [mailto:van.sjason@gmail.com]> wrote: Maybe on the master node set up every device but disable polling; child nodes poll a certain pool of devices, then have a cron script that copies rrd files to the master?
Or is that not enough to trick Observium on the master to think it has gotten all that data by itself?
On Wed, Mar 16, 2016 at 7:56 AM, Ruiz, Carlos <carlos.ruiz.cr2@roche.com [mailto:carlos.ruiz.cr2@roche.com]> wrote:
Hi,
The problem is that we've a huge variety of device models worldwide and we'll discover most of the models when we start with the roll-out
Let me remake the question, is observium ready to work on a master of masters architecture design?
Thanks in advance for your help
Kind Regards.
2016-03-15 17:54 GMT+01:00 Adam Armstrong <adama@memetic.org [mailto:adama@memetic.org]>:
It all depends very much on the types of devices. 20k 500 port switches isn't going to happen. 20k Linux services is probably pretty easy.
adam.
Sent from Mailbird [http://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm...] On 15/03/2016 16:41:13, Ruiz, Carlos <carlos.ruiz.cr2@roche.com [mailto:carlos.ruiz.cr2@roche.com]> wrote: Good morning everyone,
We're evaluating to use observium as hardware monitoring devices based on snmp protocol for devices located all around the world.
We're talking about 20K devices and I believe that have one single host locate in SSF, getting all the info from all the devices, located on places like Shangai may not be the best scenario to get the best performance from the tool itself
Does observium support the concept of Master of masters?, I mean have local master observium hosts (EMEA, APAC, NA) and consolidate all the info in one single Master Observium Instance
If not what will be a recommended approach to cover an scenario like the one described
Thanks in advance for your help and collaboration
Kind Regards --
Carlos Ruiz Compute System Management Senior Engineer Roche Farma, S.A.Josefa Valcarcel, 40. ES-28027 Madrid (Spain)
Phone: +34 91 305 99 25 [tel:%2B34%2091%20305%2099%2025] Mobile: +34 662 10 46 14 [tel:%2B34%20662%2010%2046%2014]
* *
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
--
Carlos Ruiz Compute System Management Senior Engineer Roche Farma, S.A.Josefa Valcarcel, 40. ES-28027 Madrid (Spain)
Phone: +34 91 305 99 25 [tel:%2B34%2091%20305%2099%2025] Mobile: +34 662 10 46 14 [tel:%2B34%20662%2010%2046%2014]
* *
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
Well I guess the hard part would be to have a distributed database? I have no experience in DB-clusters but if you could have a MySQL-server in each region that syncs with each other active/active-style (no idea if that is possible considering the delay) then it should not be to hard to just have a rsync-job syncing the RRD-folders(does not need to sync every 5min, the data will still be there) and modify the poller-wrapper to instead of launching poller.php instances locally make it launch them on remote servers.
maybe. /Markus
Ok, I though trying to write RRDs via NFS with like 200ms+ connections would be an even more terrible idea :p RRDs are fairly effective to compress, you could have rsync gzip them during transfer :)
/Markus
2016-03-16 15:55 GMT+01:00 Adam Armstrong adama@memetic.org:
Centralised mysql and rrd via NFS would be easier and lower bandwidth. Observium was originally built with this possibility in mind, though we rarely push it.
Syncing rrds is a terrible idea, look at the size of your rrd folder :)
Adam. On Mar 16, 2016 2:43 PM, Markus Klock markus@best-practice.se wrote:
Well I guess the hard part would be to have a distributed database? I have no experience in DB-clusters but if you could have a MySQL-server in each region that syncs with each other active/active-style (no idea if that is possible considering the delay) then it should not be to hard to just have a rsync-job syncing the RRD-folders(does not need to sync every 5min, the data will still be there) and modify the poller-wrapper to instead of launching poller.php instances locally make it launch them on remote servers.
maybe. /Markus
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
No. We currently have no mechanism to do this, since it would not be useful to 99% of users, and would vastly complicate the software.
Adam.
Sent from BlueMail
On Mar 16, 2016, 12:57, at 12:57, "Ruiz, Carlos" carlos.ruiz.cr2@roche.com wrote:
Hi,
The problem is that we've a huge variety of device models worldwide and we'll discover most of the models when we start with the roll-out
Let me remake the question, is observium ready to work on a master of masters architecture design?
Thanks in advance for your help
Kind Regards.
2016-03-15 17:54 GMT+01:00 Adam Armstrong adama@memetic.org:
It all depends very much on the types of devices. 20k 500 port
switches
isn't going to happen. 20k Linux services is probably pretty easy.
adam.
Sent from Mailbird
http://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm_campaign=sent-from-mailbird
On 15/03/2016 16:41:13, Ruiz, Carlos carlos.ruiz.cr2@roche.com
wrote:
Good morning everyone,
We're evaluating to use observium as hardware monitoring devices
based on
snmp protocol for devices located all around the world.
We're talking about 20K devices and I believe that have one single
host
locate in SSF, getting all the info from all the devices, located on
places
like Shangai may not be the best scenario to get the best performance
from
the tool itself
Does observium support the concept of Master of masters?, I mean have local master observium hosts (EMEA, APAC, NA) and consolidate all the
info
in one single Master Observium Instance
If not what will be a recommended approach to cover an scenario like
the
one described
Thanks in advance for your help and collaboration
Kind Regards
Carlos Ruiz Compute System Management Senior Engineer
Roche Farma, S.A.Josefa Valcarcel, 40. ES-28027 Madrid (Spain)
Phone: +34 91 305 99 25 Mobile: +34 662 10 46 14
_______________________________________________ observium mailing
list
observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Carlos Ruiz Compute System Management Senior Engineer
Roche Farma, S.A.Josefa Valcarcel, 40. ES-28027 Madrid (Spain)
Phone: +34 91 305 99 25 Mobile: +34 662 10 46 14
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (4)
-
Adam Armstrong -
Jason Vanlandingham -
Markus Klock -
Ruiz, Carlos