Syslog host matching with multiple devices and 1 public IP
Hi, We have recently started using the syslog part of Observium and for the most part it works well where we have multiple public IP's logs are coming from or for places where we have VPN's and can reach devices on their local IP's.
We are running into a problem however when we have multiple devices pushing their syslog's to us over a WAN link and they come from the same public IP. We're using the %fromhost% variable in the configuration and all logs seem to end up with the first device that we have set in /etc/hosts to match the public IP. We've tried changing the %fromhost% to %hostname% instead thinking it would use the hostname within the log entry but unfortunately that didn't work either.
I've had a look at the documentation on how to do manual matching but I don't think that would help here either as it's only going to match the public IP.
Has anyone come up against this before and know if there is a way to resolve it or will I need to create VPN's to each of those sites so we can push the logs over their private IP's instead?
Hmm. I'm not sure if we have an existing way to handle this. How does rsyslog usually allow you to separate these?
I think to handle these you need to trust the hostname the device itself sends. I don't think we currently pass that, so it might need modification of code and format.
adam.
mp--- via observium wrote on 21/01/2023 09:03:
Hi, We have recently started using the syslog part of Observium and for the most part it works well where we have multiple public IP's logs are coming from or for places where we have VPN's and can reach devices on their local IP's.
We are running into a problem however when we have multiple devices pushing their syslog's to us over a WAN link and they come from the same public IP. We're using the %fromhost% variable in the configuration and all logs seem to end up with the first device that we have set in /etc/hosts to match the public IP. We've tried changing the %fromhost% to %hostname% instead thinking it would use the hostname within the log entry but unfortunately that didn't work either.
I've had a look at the documentation on how to do manual matching but I don't think that would help here either as it's only going to match the public IP.
Has anyone come up against this before and know if there is a way to resolve it or will I need to create VPN's to each of those sites so we can push the logs over their private IP's instead? _______________________________________________ observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
That's what I've been trying to work out too, I think if the hostname could be used it would work which is why I tried changing the below template around so that the %fromhost% part was %hostname% instead as this is one of the variables from the rsyslog documentation and my understanding is that it should then use the device hostname from the message. I wasn't sure how observium would then pick it up as to whether it was looking at the position in the line and just taking the first part as to the host to match it to or if there was more to it.
# observium syslog template template(name="observium" type="string" string="%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg%||%programname%\n")
The below article shows how to store the messages by hostname in a different log file which I think is the standard rsyslog way however I've never tried this in the same scenario with multiple devices behind one public IP to be able to reliably say if this works or not.
https://www.rsyslog.com/storing-and-forwarding-remote-messages/
I was trying to run debug in observium to see what it picks up but after enabling it for syslog and also setting it to log unknown hosts I don't get any additional logging to see what the lines look like for these hosts.
I did run tcpdump instead and it looks like the hostname is sent ok in the syslog message so I think it's just a case of working out how to use that and get it into observium.
We have multiple ways for associate syslog hosts with devices in observium.
It seems you should use hostnames (%hostname%) from each device, but need sure that devices report correct hostnames in syslog message.
You can enable (temporary) store all syslog messages from all hosts, in config.php add: $config['syslog']['debug'] = TRUE;
be sure that observium logs dir have permissions for write syslog user (simplest way: chmod 777 /opt/observium/logs).
restart rsyslog service.
After that you will get files in logs dir: debug.<host>.syslog
Mainly you should know <host> here. Now you can manually map this hosts with devices: $config['syslog']['host_map']['<host>'] = '<observium_host>'; $config['syslog']['host_map']['<host>'] = '<device_id>';
P.S. Disable syslog debug in config.php and restart rsyslog again for prevent full disk with this debugs.
mp--- via observium wrote on 25.01.2023 00:30:
That's what I've been trying to work out too, I think if the hostname could be used it would work which is why I tried changing the below template around so that the %fromhost% part was %hostname% instead as this is one of the variables from the rsyslog documentation and my understanding is that it should then use the device hostname from the message. I wasn't sure how observium would then pick it up as to whether it was looking at the position in the line and just taking the first part as to the host to match it to or if there was more to it.
# observium syslog template template(name="observium" type="string" string="%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg%||%programname%\n")
The below article shows how to store the messages by hostname in a different log file which I think is the standard rsyslog way however I've never tried this in the same scenario with multiple devices behind one public IP to be able to reliably say if this works or not.
https://www.rsyslog.com/storing-and-forwarding-remote-messages/
I was trying to run debug in observium to see what it picks up but after enabling it for syslog and also setting it to log unknown hosts I don't get any additional logging to see what the lines look like for these hosts.
I did run tcpdump instead and it looks like the hostname is sent ok in the syslog message so I think it's just a case of working out how to use that and get it into observium. _______________________________________________ observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
Thanks. The debug is what I've been trying but unfortunately I don't get any new log files created for the host. If I look at tcpdump though I can see the syslog message coming in.
I've tried just chmod 777 as well just as a quick test but nothing got created still.
I do have other debug log files but just not for the host I'm expecting and I'm not seeing the event go to any other log either.
participants (3)
-
Adam Armstrong -
Mike Stupalov -
mp＠matt-parkinson.co.uk