I have an installation of Observium that has been up and running perfectly for about 18 months. Last Friday I ran updates and now syslog is no longer working. I have found that prior to the upgrade, rsyslog was version 7, now I am at Version 8. In searching I found that a few configuration file changes, I did those, and still a no go. Has anyone else come across this? I am a novice at Linux, and totally at a loss here. I can do a tcpdump and sure enough the syslog messages are hitting the server, but not displayed within Observium. Any help would be appreciated, but, please keep in mind I am a Centos novice.
Dave Rossi.
Have you tried restarting the rsyslog service?
I found, every time the PHP file changes (due to code changes or updates) I had to restart the service for it to start working again
Regards
Simon
On 31 Oct 2018, at 15:36, David Rossi via observium observium@observium.org wrote:
I have an installation of Observium that has been up and running perfectly for about 18 months.
Last Friday I ran updates and now syslog is no longer working.
I have found that prior to the upgrade, rsyslog was version 7, now I am at Version 8. In searching I found that a few configuration file changes, I did those, and still a no go.
Has anyone else come across this? I am a novice at Linux, and totally at a loss here. I can do a tcpdump and sure enough the syslog messages are hitting the server, but not displayed within Observium. Any help would be appreciated, but, please keep in mind I am a Centos novice.
Dave Rossi.
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Yes, I have restarted multiple times, even booted the server. I have uninstalled Rsyslog, reinstalled, reconfigured, still no go. I am thinking about removing rsyslog again, and going to syslog-ng as a test.
Dave Rossi.
From: observium observium-bounces@observium.org On Behalf Of Simon Mousey Smith via observium Sent: Wednesday, October 31, 2018 12:02 PM To: Observium observium@observium.org Cc: Simon Mousey Smith simonsmith5521@gmail.com Subject: Re: [Observium] Observium Syslog issues
Have you tried restarting the rsyslog service?
I found, every time the PHP file changes (due to code changes or updates) I had to restart the service for it to start working again
Regards
Simon
On 31 Oct 2018, at 15:36, David Rossi via observium <observium@observium.orgmailto:observium@observium.org> wrote:
I have an installation of Observium that has been up and running perfectly for about 18 months. Last Friday I ran updates and now syslog is no longer working. I have found that prior to the upgrade, rsyslog was version 7, now I am at Version 8. In searching I found that a few configuration file changes, I did those, and still a no go. Has anyone else come across this? I am a novice at Linux, and totally at a loss here. I can do a tcpdump and sure enough the syslog messages are hitting the server, but not displayed within Observium. Any help would be appreciated, but, please keep in mind I am a Centos novice.
Dave Rossi. _______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
syslog is one of those things you set up once and never ever touch ever again, so it becomes sort of voodoo.
My dev install is running 8.16, so here are by config parts that might be useful :
##############################
/etc/rsyslog.conf
module(load="imuxsock") # provides support for local system logging module(load="imudp") input(type="imudp" port="514")
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $RepeatedMsgReduction on
$FileOwner syslog $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 $PrivDropToUser syslog $PrivDropToGroup syslog
$WorkDirectory /var/spool/rsyslog $IncludeConfig /etc/rsyslog.d/*.conf
##################################
/etc/rsyslog.d/70-observium.conf
template(name="observium" type="string" string="%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg%||%programname%\n")
$ModLoad omprog
# rsyslog Input Modules input(type="imudp" port="514" ruleset="observium")
# rsyslog RuleSets ruleset(name="observium") { action(type="omprog" binary="/home/observium/dev/syslog.php" template="observium") }
*.* stop
#########################################
Hope that helps! :D
adam.
On 2018-10-31 16:07:37, David Rossi via observium observium@observium.org wrote: Yes, I have restarted multiple times, even booted the server. I have uninstalled Rsyslog, reinstalled, reconfigured, still no go. I am thinking about removing rsyslog again, and going to syslog-ng as a test. Dave Rossi. From: observium observium-bounces@observium.org On Behalf Of Simon Mousey Smith via observium Sent: Wednesday, October 31, 2018 12:02 PM To: Observium observium@observium.org Cc: Simon Mousey Smith simonsmith5521@gmail.com Subject: Re: [Observium] Observium Syslog issues Have you tried restarting the rsyslog service? I found, every time the PHP file changes (due to code changes or updates) I had to restart the service for it to start working again Regards Simon
On 31 Oct 2018, at 15:36, David Rossi via observium <observium@observium.org [mailto:observium@observium.org]> wrote: I have an installation of Observium that has been up and running perfectly for about 18 months. Last Friday I ran updates and now syslog is no longer working. I have found that prior to the upgrade, rsyslog was version 7, now I am at Version 8. In searching I found that a few configuration file changes, I did those, and still a no go. Has anyone else come across this? I am a novice at Linux, and totally at a loss here. I can do a tcpdump and sure enough the syslog messages are hitting the server, but not displayed within Observium. Any help would be appreciated, but, please keep in mind I am a Centos novice. Dave Rossi. _______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
I will give this a shot. I noticed your config file is 70-obsirvium.conf, mine is 30-……. I don’t think that should matter because *.conf is included for the configuration, is that correct? Also, I believe I need to change the path to syslog.php, is that also correct? If those assumptions are correct, I will try this and report back
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong via observium Sent: Wednesday, October 31, 2018 2:36 PM To: David Rossi via observium observium@observium.org Cc: Adam Armstrong adama@memetic.org Subject: Re: [Observium] Observium Syslog issues
syslog is one of those things you set up once and never ever touch ever again, so it becomes sort of voodoo.
My dev install is running 8.16, so here are by config parts that might be useful :
##############################
/etc/rsyslog.conf
module(load="imuxsock") # provides support for local system logging module(load="imudp") input(type="imudp" port="514")
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $RepeatedMsgReduction on
$FileOwner syslog $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 $PrivDropToUser syslog $PrivDropToGroup syslog
$WorkDirectory /var/spool/rsyslog $IncludeConfig /etc/rsyslog.d/*.conf
##################################
/etc/rsyslog.d/70-observium.conf
template(name="observium" type="string" string="%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timereported:8:25%||%msg%||%programname%\n")
$ModLoad omprog
# rsyslog Input Modules input(type="imudp" port="514" ruleset="observium")
# rsyslog RuleSets ruleset(name="observium") { action(type="omprog" binary="/home/observium/dev/syslog.php" template="observium") }
*.* stop
#########################################
Hope that helps! :D
adam.
On 2018-10-31 16:07:37, David Rossi via observium <observium@observium.orgmailto:observium@observium.org> wrote: Yes, I have restarted multiple times, even booted the server. I have uninstalled Rsyslog, reinstalled, reconfigured, still no go. I am thinking about removing rsyslog again, and going to syslog-ng as a test.
Dave Rossi.
From: observium <observium-bounces@observium.orgmailto:observium-bounces@observium.org> On Behalf Of Simon Mousey Smith via observium Sent: Wednesday, October 31, 2018 12:02 PM To: Observium <observium@observium.orgmailto:observium@observium.org> Cc: Simon Mousey Smith <simonsmith5521@gmail.commailto:simonsmith5521@gmail.com> Subject: Re: [Observium] Observium Syslog issues
Have you tried restarting the rsyslog service?
I found, every time the PHP file changes (due to code changes or updates) I had to restart the service for it to start working again
Regards
Simon
On 31 Oct 2018, at 15:36, David Rossi via observium <observium@observium.orgmailto:observium@observium.org> wrote:
I have an installation of Observium that has been up and running perfectly for about 18 months. Last Friday I ran updates and now syslog is no longer working. I have found that prior to the upgrade, rsyslog was version 7, now I am at Version 8. In searching I found that a few configuration file changes, I did those, and still a no go. Has anyone else come across this? I am a novice at Linux, and totally at a loss here. I can do a tcpdump and sure enough the syslog messages are hitting the server, but not displayed within Observium. Any help would be appreciated, but, please keep in mind I am a Centos novice.
Dave Rossi. _______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
It can also be helpful to simply run ~observium/syslog.php on the command line as your syslog user to confirm that there are no errors with permissions or config.php.
Your rsyslogd should fire up it's own copy of syslog.php and keep it running as long as itself is running; so you should see something like:
** root@observium ** /opt/observium ** Thu Nov 01 09:30:18 # ps auxww | grep syslog root 23509 0.0 0.0 14116 984 pts/5 R+ 09:30 0:00 grep syslog syslog 24688 0.0 0.0 432276 1252 ? Ssl Oct24 2:14 rsyslogd syslog 24708 0.0 0.6 274424 12456 ? S Oct24 4:59 php /opt/observium/syslog.php
We also have a block in rsyslog.conf that dumps all local7 traffic (our ciscos all use local7) to a regular log file just to confirm that rsyslogd is receiving and processing the logs.
If you create a ~observium/logs/debug file and make sure that your syslog user has write permissions, you can add:
$config['syslog']['debug'] = TRUE;
in your config.php and the syslog.php script will write out the raw string it's getting from rsyslogd for each entry. You can use that to make sure that the "hostname" part matches what you're using in observium (e.g. make sure it's an FQDN).
-rw-rw-r-- 1 syslog www-data 16339 Oct 23 16:24 debug.log
On 10/31/2018 12:01 PM, Simon Mousey Smith via observium wrote:
Have you tried restarting the rsyslog service?
I found, every time the PHP file changes (due to code changes or updates) I had to restart the service for it to start working again
Regards
Simon
On 31 Oct 2018, at 15:36, David Rossi via observium <observium@observium.org mailto:observium@observium.org> wrote:
I have an installation of Observium that has been up and running perfectly for about 18 months.
Last Friday I ran updates and now syslog is no longer working.
I have found that prior to the upgrade, rsyslog was version 7, now I am at Version 8. In searching I found that a few configuration file changes, I did those, and still a no go.
Has anyone else come across this? I am a novice at Linux, and totally at a loss here. I can do a tcpdump and sure enough the syslog messages are hitting the server, but not displayed within Observium. Any help would be appreciated, but, please keep in mind I am a Centos novice.
Dave Rossi.
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Being a novice some of what you mentioned went a bit over my head, but I seem to be learning, that is always a good thing.
I had thought that this was related to rsyslog not functioning, I believe I can now rule that out. I turned syslog off within Observium, and removed the configuration files and entries pointing to Observium. I then set rsyslog to dump to the messages file, all syslog messages are going there now, (I will shut that as it is growing quickly). I have 2 firewalls, 1 is sending with its FQDN, the 2nd is sending with its IP in the message. I also set up the debugging for the Observium syslog, and messages were going to the debug log, but not displayed from the web interface. Is it possibly a database issue? All Port information, and events are showing,just no syslogs. Running syslog.php, actually displayed no results what so ever....
-----Original Message----- From: observium observium-bounces@observium.org On Behalf Of Eric W. Bates via observium Sent: Thursday, November 1, 2018 9:40 AM To: Observium observium@observium.org Cc: Eric W. Bates ericx@whoi.edu Subject: Re: [Observium] Observium Syslog issues
It can also be helpful to simply run ~observium/syslog.php on the command line as your syslog user to confirm that there are no errors with permissions or config.php.
Your rsyslogd should fire up it's own copy of syslog.php and keep it running as long as itself is running; so you should see something like:
** root@observium ** /opt/observium ** Thu Nov 01 09:30:18 # ps auxww | grep syslog root 23509 0.0 0.0 14116 984 pts/5 R+ 09:30 0:00 grep syslog syslog 24688 0.0 0.0 432276 1252 ? Ssl Oct24 2:14 rsyslogd syslog 24708 0.0 0.6 274424 12456 ? S Oct24 4:59 php /opt/observium/syslog.php
We also have a block in rsyslog.conf that dumps all local7 traffic (our ciscos all use local7) to a regular log file just to confirm that rsyslogd is receiving and processing the logs.
If you create a ~observium/logs/debug file and make sure that your syslog user has write permissions, you can add:
$config['syslog']['debug'] = TRUE;
in your config.php and the syslog.php script will write out the raw string it's getting from rsyslogd for each entry. You can use that to make sure that the "hostname" part matches what you're using in observium (e.g. make sure it's an FQDN).
-rw-rw-r-- 1 syslog www-data 16339 Oct 23 16:24 debug.log
On 10/31/2018 12:01 PM, Simon Mousey Smith via observium wrote:
Have you tried restarting the rsyslog service?
I found, every time the PHP file changes (due to code changes or updates) I had to restart the service for it to start working again
Regards
Simon
On 31 Oct 2018, at 15:36, David Rossi via observium <observium@observium.org mailto:observium@observium.org> wrote:
I have an installation of Observium that has been up and running perfectly for about 18 months.
Last Friday I ran updates and now syslog is no longer working.
I have found that prior to the upgrade, rsyslog was version 7, now I am at Version 8. In searching I found that a few configuration file changes, I did those, and still a no go.
Has anyone else come across this? I am a novice at Linux, and totally at a loss here. I can do a tcpdump and sure enough the syslog messages are hitting the server, but not displayed within Observium. Any help would be appreciated, but, please keep in mind I am a Centos novice.
Dave Rossi.
observium mailing list observium@observium.org mailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Clark 159a, MS 46 508/289-3112
You may want to provide some of the output from the debug log... :D
Running syslog.php by hand should generate no output, you were just checking that it doesn't spit errors.
adam. On 2018-11-01 17:56:09, David Rossi via observium observium@observium.org wrote: Being a novice some of what you mentioned went a bit over my head, but I seem to be learning, that is always a good thing.
I had thought that this was related to rsyslog not functioning, I believe I can now rule that out. I turned syslog off within Observium, and removed the configuration files and entries pointing to Observium. I then set rsyslog to dump to the messages file, all syslog messages are going there now, (I will shut that as it is growing quickly). I have 2 firewalls, 1 is sending with its FQDN, the 2nd is sending with its IP in the message. I also set up the debugging for the Observium syslog, and messages were going to the debug log, but not displayed from the web interface. Is it possibly a database issue? All Port information, and events are showing,just no syslogs. Running syslog.php, actually displayed no results what so ever....
-----Original Message----- From: observium On Behalf Of Eric W. Bates via observium Sent: Thursday, November 1, 2018 9:40 AM To: Observium Cc: Eric W. Bates Subject: Re: [Observium] Observium Syslog issues
It can also be helpful to simply run ~observium/syslog.php on the command line as your syslog user to confirm that there are no errors with permissions or config.php.
Your rsyslogd should fire up it's own copy of syslog.php and keep it running as long as itself is running; so you should see something like:
** root@observium ** /opt/observium ** Thu Nov 01 09:30:18 # ps auxww | grep syslog root 23509 0.0 0.0 14116 984 pts/5 R+ 09:30 0:00 grep syslog syslog 24688 0.0 0.0 432276 1252 ? Ssl Oct24 2:14 rsyslogd syslog 24708 0.0 0.6 274424 12456 ? S Oct24 4:59 php /opt/observium/syslog.php
We also have a block in rsyslog.conf that dumps all local7 traffic (our ciscos all use local7) to a regular log file just to confirm that rsyslogd is receiving and processing the logs.
If you create a ~observium/logs/debug file and make sure that your syslog user has write permissions, you can add:
$config['syslog']['debug'] = TRUE;
in your config.php and the syslog.php script will write out the raw string it's getting from rsyslogd for each entry. You can use that to make sure that the "hostname" part matches what you're using in observium (e.g. make sure it's an FQDN).
-rw-rw-r-- 1 syslog www-data 16339 Oct 23 16:24 debug.log
On 10/31/2018 12:01 PM, Simon Mousey Smith via observium wrote:
Have you tried restarting the rsyslog service?
I found, every time the PHP file changes (due to code changes or updates) I had to restart the service for it to start working again
Regards
Simon
On 31 Oct 2018, at 15:36, David Rossi via observium
wrote:
I have an installation of Observium that has been up and running perfectly for about 18 months.
Last Friday I ran updates and now syslog is no longer working.
I have found that prior to the upgrade, rsyslog was version 7, now I am at Version 8. In searching I found that a few configuration file changes, I did those, and still a no go.
Has anyone else come across this? I am a novice at Linux, and totally at a loss here. I can do a tcpdump and sure enough the syslog messages are hitting the server, but not displayed within Observium. Any help would be appreciated, but, please keep in mind I am a Centos novice.
Dave Rossi.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Clark 159a, MS 46 508/289-3112
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
I want to thank you all for the help, but, this has consumed too much time, I am going to just scratch this, and start fresh…. Thanks for the suggestions and help.
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong via observium Sent: Thursday, November 1, 2018 3:57 PM To: David Rossi via observium observium@observium.org Cc: Adam Armstrong adama@memetic.org Subject: Re: [Observium] Observium Syslog issues
You may want to provide some of the output from the debug log... :D
Running syslog.php by hand should generate no output, you were just checking that it doesn't spit errors.
adam.
On 2018-11-01 17:56:09, David Rossi via observium <observium@observium.orgmailto:observium@observium.org> wrote: Being a novice some of what you mentioned went a bit over my head, but I seem to be learning, that is always a good thing.
I had thought that this was related to rsyslog not functioning, I believe I can now rule that out. I turned syslog off within Observium, and removed the configuration files and entries pointing to Observium. I then set rsyslog to dump to the messages file, all syslog messages are going there now, (I will shut that as it is growing quickly). I have 2 firewalls, 1 is sending with its FQDN, the 2nd is sending with its IP in the message. I also set up the debugging for the Observium syslog, and messages were going to the debug log, but not displayed from the web interface. Is it possibly a database issue? All Port information, and events are showing,just no syslogs. Running syslog.php, actually displayed no results what so ever....
-----Original Message----- From: observium On Behalf Of Eric W. Bates via observium Sent: Thursday, November 1, 2018 9:40 AM To: Observium Cc: Eric W. Bates Subject: Re: [Observium] Observium Syslog issues
It can also be helpful to simply run ~observium/syslog.php on the command line as your syslog user to confirm that there are no errors with permissions or config.php.
Your rsyslogd should fire up it's own copy of syslog.php and keep it running as long as itself is running; so you should see something like:
** root@observium ** /opt/observium ** Thu Nov 01 09:30:18 # ps auxww | grep syslog root 23509 0.0 0.0 14116 984 pts/5 R+ 09:30 0:00 grep syslog syslog 24688 0.0 0.0 432276 1252 ? Ssl Oct24 2:14 rsyslogd syslog 24708 0.0 0.6 274424 12456 ? S Oct24 4:59 php /opt/observium/syslog.php
We also have a block in rsyslog.conf that dumps all local7 traffic (our ciscos all use local7) to a regular log file just to confirm that rsyslogd is receiving and processing the logs.
If you create a ~observium/logs/debug file and make sure that your syslog user has write permissions, you can add:
$config['syslog']['debug'] = TRUE;
in your config.php and the syslog.php script will write out the raw string it's getting from rsyslogd for each entry. You can use that to make sure that the "hostname" part matches what you're using in observium (e.g. make sure it's an FQDN).
-rw-rw-r-- 1 syslog www-data 16339 Oct 23 16:24 debug.log
On 10/31/2018 12:01 PM, Simon Mousey Smith via observium wrote:
Have you tried restarting the rsyslog service?
I found, every time the PHP file changes (due to code changes or updates) I had to restart the service for it to start working again
Regards
Simon
On 31 Oct 2018, at 15:36, David Rossi via observium
wrote:
I have an installation of Observium that has been up and running perfectly for about 18 months.
Last Friday I ran updates and now syslog is no longer working.
I have found that prior to the upgrade, rsyslog was version 7, now I am at Version 8. In searching I found that a few configuration file changes, I did those, and still a no go.
Has anyone else come across this? I am a novice at Linux, and totally at a loss here. I can do a tcpdump and sure enough the syslog messages are hitting the server, but not displayed within Observium. Any help would be appreciated, but, please keep in mind I am a Centos novice.
Dave Rossi.
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Clark 159a, MS 46 508/289-3112
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Note that Observium will only accept syslog messages for IPs or hostnames that it knows in that form.
I.e. it won't accept syslog messages for "banana" if it only has "banana.foo.com" as hostname or sysname entries.
You can use config entries to rewrite these, but it's best to fix them at source.
This is why log entries are useful.
adam.
On 2018-11-01 21:54, David Rossi via observium wrote:
I want to thank you all for the help, but, this has consumed too much time, I am going to just scratch this, and start fresh…. Thanks for the suggestions and help.
FROM: observium [mailto:observium-bounces@observium.org] ON BEHALF OF Adam Armstrong via observium SENT: Thursday, November 1, 2018 3:57 PM TO: David Rossi via observium observium@observium.org CC: Adam Armstrong adama@memetic.org SUBJECT: Re: [Observium] Observium Syslog issues
You may want to provide some of the output from the debug log... :D
Running syslog.php by hand should generate no output, you were just checking that it doesn't spit errors.
adam.
On 2018-11-01 17:56:09, David Rossi via observium observium@observium.org wrote:
Being a novice some of what you mentioned went a bit over my head, but I seem to be learning, that is always a good thing.
I had thought that this was related to rsyslog not functioning, I believe I can now rule that out. I turned syslog off within Observium, and removed the configuration files and entries pointing to Observium. I then set rsyslog to dump to the messages file, all syslog messages are going there now, (I will shut that as it is growing quickly). I have 2 firewalls, 1 is sending with its FQDN, the 2nd is sending with its IP in the message. I also set up the debugging for the Observium syslog, and messages were going to the debug log, but not displayed from the web interface. Is it possibly a database issue? All Port information, and events are showing,just no syslogs. Running syslog.php, actually displayed no results what so ever....
-----Original Message----- From: observium On Behalf Of Eric W. Bates via observium Sent: Thursday, November 1, 2018 9:40 AM To: Observium Cc: Eric W. Bates Subject: Re: [Observium] Observium Syslog issues
It can also be helpful to simply run ~observium/syslog.php on the command line as your syslog user to confirm that there are no errors with permissions or config.php.
Your rsyslogd should fire up it's own copy of syslog.php and keep it running as long as itself is running; so you should see something like:
** root@observium ** /opt/observium ** Thu Nov 01 09:30:18 # ps auxww | grep syslog root 23509 0.0 0.0 14116 984 pts/5 R+ 09:30 0:00 grep syslog syslog 24688 0.0 0.0 432276 1252 ? Ssl Oct24 2:14 rsyslogd syslog 24708 0.0 0.6 274424 12456 ? S Oct24 4:59 php /opt/observium/syslog.php
We also have a block in rsyslog.conf that dumps all local7 traffic (our ciscos all use local7) to a regular log file just to confirm that rsyslogd is receiving and processing the logs.
If you create a ~observium/logs/debug file and make sure that your syslog user has write permissions, you can add:
$config['syslog']['debug'] = TRUE;
in your config.php and the syslog.php script will write out the raw string it's getting from rsyslogd for each entry. You can use that to make sure that the "hostname" part matches what you're using in observium (e.g. make sure it's an FQDN).
-rw-rw-r-- 1 syslog www-data 16339 Oct 23 16:24 debug.log
On 10/31/2018 12:01 PM, Simon Mousey Smith via observium wrote:
Have you tried restarting the rsyslog service?
I found, every time the PHP file changes (due to code changes or updates) I had to restart the service for it to start working
again
Regards
Simon
On 31 Oct 2018, at 15:36, David Rossi via observium
wrote:
I have an installation of Observium that has been up and running perfectly for about 18 months.
Last Friday I ran updates and now syslog is no longer working.
I have found that prior to the upgrade, rsyslog was version 7,
now I
am at Version 8. In searching I found that a few configuration
file
changes, I did those, and still a no go.
Has anyone else come across this? I am a novice at Linux, and
totally
at a loss here. I can do a tcpdump and sure enough the syslog messages are hitting the server, but not displayed within
Observium.
Any help would be appreciated, but, please keep in mind I am a
Centos
novice.
Dave Rossi.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Clark 159a, MS 46 508/289-3112
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Agreed, BUT, that does not explain why it just stopped working. I had this server in use for about 18 months, with absolutely no issues at all. It is not like I was setting this up for the first time.
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong via observium Sent: Thursday, November 1, 2018 5:57 PM To: Observium observium@observium.org Cc: Adam Armstrong adama@memetic.org Subject: Re: [Observium] Observium Syslog issues
Note that Observium will only accept syslog messages for IPs or hostnames that it knows in that form.
I.e. it won't accept syslog messages for "banana" if it only has "banana.foo.com" as hostname or sysname entries.
You can use config entries to rewrite these, but it's best to fix them at source.
This is why log entries are useful.
adam.
On 2018-11-01 21:54, David Rossi via observium wrote:
I want to thank you all for the help, but, this has consumed too much time, I am going to just scratch this, and start fresh…. Thanks for the suggestions and help.
FROM: observium [mailto:observium-bounces@observium.org] ON BEHALF OF Adam Armstrong via observium SENT: Thursday, November 1, 2018 3:57 PM TO: David Rossi via observium observium@observium.org CC: Adam Armstrong adama@memetic.org SUBJECT: Re: [Observium] Observium Syslog issues
You may want to provide some of the output from the debug log... :D
Running syslog.php by hand should generate no output, you were just checking that it doesn't spit errors.
adam.
On 2018-11-01 17:56:09, David Rossi via observium observium@observium.org wrote:
Being a novice some of what you mentioned went a bit over my head, but I seem to be learning, that is always a good thing.
I had thought that this was related to rsyslog not functioning, I believe I can now rule that out. I turned syslog off within Observium, and removed the configuration files and entries pointing to Observium. I then set rsyslog to dump to the messages file, all syslog messages are going there now, (I will shut that as it is growing quickly). I have 2 firewalls, 1 is sending with its FQDN, the 2nd is sending with its IP in the message. I also set up the debugging for the Observium syslog, and messages were going to the debug log, but not displayed from the web interface. Is it possibly a database issue? All Port information, and events are showing,just no syslogs. Running syslog.php, actually displayed no results what so ever....
-----Original Message----- From: observium On Behalf Of Eric W. Bates via observium Sent: Thursday, November 1, 2018 9:40 AM To: Observium Cc: Eric W. Bates Subject: Re: [Observium] Observium Syslog issues
It can also be helpful to simply run ~observium/syslog.php on the command line as your syslog user to confirm that there are no errors with permissions or config.php.
Your rsyslogd should fire up it's own copy of syslog.php and keep it running as long as itself is running; so you should see something like:
** root@observium ** /opt/observium ** Thu Nov 01 09:30:18 # ps auxww | grep syslog root 23509 0.0 0.0 14116 984 pts/5 R+ 09:30 0:00 grep syslog syslog 24688 0.0 0.0 432276 1252 ? Ssl Oct24 2:14 rsyslogd syslog 24708 0.0 0.6 274424 12456 ? S Oct24 4:59 php /opt/observium/syslog.php
We also have a block in rsyslog.conf that dumps all local7 traffic (our ciscos all use local7) to a regular log file just to confirm that rsyslogd is receiving and processing the logs.
If you create a ~observium/logs/debug file and make sure that your syslog user has write permissions, you can add:
$config['syslog']['debug'] = TRUE;
in your config.php and the syslog.php script will write out the raw string it's getting from rsyslogd for each entry. You can use that to make sure that the "hostname" part matches what you're using in observium (e.g. make sure it's an FQDN).
-rw-rw-r-- 1 syslog www-data 16339 Oct 23 16:24 debug.log
On 10/31/2018 12:01 PM, Simon Mousey Smith via observium wrote:
Have you tried restarting the rsyslog service?
I found, every time the PHP file changes (due to code changes or updates) I had to restart the service for it to start working
again
Regards
Simon
On 31 Oct 2018, at 15:36, David Rossi via observium
wrote:
I have an installation of Observium that has been up and running perfectly for about 18 months.
Last Friday I ran updates and now syslog is no longer working.
I have found that prior to the upgrade, rsyslog was version 7,
now I
am at Version 8. In searching I found that a few configuration
file
changes, I did those, and still a no go.
Has anyone else come across this? I am a novice at Linux, and
totally
at a loss here. I can do a tcpdump and sure enough the syslog messages are hitting the server, but not displayed within
Observium.
Any help would be appreciated, but, please keep in mind I am a
Centos
novice.
Dave Rossi.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Clark 159a, MS 46 508/289-3112
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hmm, that is odd.
I don't think anything has really changed regarding our syslog for quite a while except for log parsing.
It still seems that the only really important piece of data missing here are the hostnames of the log messages, since if they're actually ending up in the observium debugging log, they're making it to syslog.php, and the most likely reason for them not being accepted is that the hostname isn't being matched.
The syslog daemons have a lot of options which can modify the hostname being used.
adam. On 2018-11-01 21:59:42, David Rossi via observium observium@observium.org wrote: Agreed, BUT, that does not explain why it just stopped working. I had this server in use for about 18 months, with absolutely no issues at all. It is not like I was setting this up for the first time.
-----Original Message----- From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong via observium Sent: Thursday, November 1, 2018 5:57 PM To: Observium Cc: Adam Armstrong Subject: Re: [Observium] Observium Syslog issues
Note that Observium will only accept syslog messages for IPs or hostnames that it knows in that form.
I.e. it won't accept syslog messages for "banana" if it only has "banana.foo.com" as hostname or sysname entries.
You can use config entries to rewrite these, but it's best to fix them at source.
This is why log entries are useful.
adam.
On 2018-11-01 21:54, David Rossi via observium wrote:
I want to thank you all for the help, but, this has consumed too much time, I am going to just scratch this, and start fresh…. Thanks for the suggestions and help.
FROM: observium [mailto:observium-bounces@observium.org] ON BEHALF OF Adam Armstrong via observium SENT: Thursday, November 1, 2018 3:57 PM TO: David Rossi via observium CC: Adam Armstrong SUBJECT: Re: [Observium] Observium Syslog issues
You may want to provide some of the output from the debug log... :D
Running syslog.php by hand should generate no output, you were just checking that it doesn't spit errors.
adam.
On 2018-11-01 17:56:09, David Rossi via observium wrote:
Being a novice some of what you mentioned went a bit over my head, but I seem to be learning, that is always a good thing.
I had thought that this was related to rsyslog not functioning, I believe I can now rule that out. I turned syslog off within Observium, and removed the configuration files and entries pointing to Observium. I then set rsyslog to dump to the messages file, all syslog messages are going there now, (I will shut that as it is growing quickly). I have 2 firewalls, 1 is sending with its FQDN, the 2nd is sending with its IP in the message. I also set up the debugging for the Observium syslog, and messages were going to the debug log, but not displayed from the web interface. Is it possibly a database issue? All Port information, and events are showing,just no syslogs. Running syslog.php, actually displayed no results what so ever....
-----Original Message----- From: observium On Behalf Of Eric W. Bates via observium Sent: Thursday, November 1, 2018 9:40 AM To: Observium Cc: Eric W. Bates Subject: Re: [Observium] Observium Syslog issues
It can also be helpful to simply run ~observium/syslog.php on the command line as your syslog user to confirm that there are no errors with permissions or config.php.
Your rsyslogd should fire up it's own copy of syslog.php and keep it running as long as itself is running; so you should see something like:
** root@observium ** /opt/observium ** Thu Nov 01 09:30:18 # ps auxww | grep syslog root 23509 0.0 0.0 14116 984 pts/5 R+ 09:30 0:00 grep syslog syslog 24688 0.0 0.0 432276 1252 ? Ssl Oct24 2:14 rsyslogd syslog 24708 0.0 0.6 274424 12456 ? S Oct24 4:59 php /opt/observium/syslog.php
We also have a block in rsyslog.conf that dumps all local7 traffic (our ciscos all use local7) to a regular log file just to confirm that rsyslogd is receiving and processing the logs.
If you create a ~observium/logs/debug file and make sure that your syslog user has write permissions, you can add:
$config['syslog']['debug'] = TRUE;
in your config.php and the syslog.php script will write out the raw string it's getting from rsyslogd for each entry. You can use that to make sure that the "hostname" part matches what you're using in observium (e.g. make sure it's an FQDN).
-rw-rw-r-- 1 syslog www-data 16339 Oct 23 16:24 debug.log
On 10/31/2018 12:01 PM, Simon Mousey Smith via observium wrote:
Have you tried restarting the rsyslog service?
I found, every time the PHP file changes (due to code changes or updates) I had to restart the service for it to start working
again
Regards
Simon
On 31 Oct 2018, at 15:36, David Rossi via observium
wrote:
I have an installation of Observium that has been up and running perfectly for about 18 months.
Last Friday I ran updates and now syslog is no longer working.
I have found that prior to the upgrade, rsyslog was version 7,
now I
am at Version 8. In searching I found that a few configuration
file
changes, I did those, and still a no go.
Has anyone else come across this? I am a novice at Linux, and
totally
at a loss here. I can do a tcpdump and sure enough the syslog messages are hitting the server, but not displayed within
Observium.
Any help would be appreciated, but, please keep in mind I am a
Centos
novice.
Dave Rossi.
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Clark 159a, MS 46 508/289-3112
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (4)
-
Adam Armstrong -
David Rossi -
Eric W. Bates -
Simon Mousey Smith