Fortigate Ports Disappeared
Hi Everyone,
Has anyone else had this problem? All of a sudden some of the critical ports have disappeared, first though it was SNMP process crash on the firewall however simple snmpwalk proves that's not the case. I then deleted device including RRDs and re-added, now none of the ports are showing :(
[cid:image001.png@01D088DB.DC99D9B0]
I'm on 6404.
Thanks
PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL.
Kelway (UK) Ltd is a Company registered in England and Wales. This email is sent for and on behalf of Kelway Holdings Limited. Kelway Holdings Limited is the UK holding company for Kelway (UK) Limited trading as Kelway. The registered office for Kelway Holdings Limited (Company Number 05872067) and its UK group undertakings is 10 Fleet Place, London, EC4M 7RB. VAT Registration 902 1949 39. Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than Kelway or the sender of its existence.
For more information about Kelway please visit www.kelway.co.uk
snmpwalk doesn't prove anything, since we don't use snmpwalk. Checking the output from the poller module running in debug might, though.
./poller.php -h <host> -m ports -d
adam.
Sent from Mailbird [http://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm...] On 07/05/2015 15:38:44, Serghei Golipad s.golipad@kelway.com wrote: Hi Everyone, Has anyone else had this problem? All of a sudden some of the critical ports have disappeared, first though it was SNMP process crash on the firewall however simple snmpwalk proves that’s not the case. I then deleted device including RRDs and re-added, now none of the ports are showing L I’m on 6404. Thanks
PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL.
Kelway Ltd is a Company registered in England and Wales. This email is sent for and on behalf of Kelway Holdings Limited. Kelway Holdings Limited is the UK holding company for Kelway Limited trading as Kelway. The registered office for Kelway Holdings Limited (Company Number 05872067) and its UK group undertakings is 10 Fleet Place, London, EC4M 7RB. VAT Registration 902 1949 39. Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than Kelway or the sender of its existence.
For more information about Kelway please visit www.kelway.com
Hi Adam,
Anything I should be looking for in the debug? I can see ifNames correctly displayed during debug however those don’t make it into portal.
Also in event log I get below for all discovered ports:
Interface changed: [ifDescr] TEST_PORT -> NULL; [ifType] other -> NULL; [ifDuplex] -> 0
One of the things I’ve picked up from poller debug:
ignored (by empty ifDescr and ifName).
Thanks
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: 07 May 2015 15:45 To: observium@observium.org Subject: Re: [Observium] Fortigate Ports Disappeared
snmpwalk doesn't prove anything, since we don't use snmpwalk. Checking the output from the poller module running in debug might, though.
./poller.php -h <host> -m ports -d
adam.
Sent from Mailbirdhttp://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm_campaign=sent-from-mailbird
On 07/05/2015 15:38:44, Serghei Golipad <s.golipad@kelway.commailto:s.golipad@kelway.com> wrote: Hi Everyone,
Has anyone else had this problem? All of a sudden some of the critical ports have disappeared, first though it was SNMP process crash on the firewall however simple snmpwalk proves that’s not the case. I then deleted device including RRDs and re-added, now none of the ports are showing ☹
[cid:image001.png@01D088DB.DC99D9B0]
I’m on 6404.
Thanks
PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL.
Kelway Ltd is a Company registered in England and Wales. This email is sent for and on behalf of Kelway Holdings Limited. Kelway Holdings Limited is the UK holding company for Kelway Limited trading as Kelway. The registered office for Kelway Holdings Limited (Company Number 05872067) and its UK group undertakings is 10 Fleet Place, London, EC4M 7RB. VAT Registration 902 1949 39. Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than Kelway or the sender of its existence.
For more information about Kelway please visit www.kelway.comhttp://www.kelway.com
Hi Serghei, This is an issue we have run into several times. Every time it has been due to a SNMPD bug on the fortigates. There is an issue with several firmware versions that returns invalid values for interfaces that have been recently deleted (usually vlan subinterfaces or ipsec interfaces). When this has happened snmp on the fortigates will still return values for everything apart from anything referencing the deleted interface (which is still in snmp somewhere). I have logged a ticket with Fortinet about this (in Feb 2014) and it is supposed to be fixed in newer firmware versions (although I have seen it crop up again on ‘fixed’ versions).
When it is in this state snmpwalk actually fails the following query snmpwalk -v2c -c public fortigateIPAddress ifType Error in packet. Reason: (genError) A general failure occured Failed object: IF-MIB::ifType.45 (in this example 45 was snmp index of the deleted interface).
The fix for us has always been to kill and restart snmpd on the Fortigate (on all cluster members if it is a cluster). You’ll need to find the process ID of the snmpd process and then kill it; watchdog will automatically restart it and then it’ll be fine.
show running processes diag sys top 30 99
kill snmpd process by process id diag sys kill 11 <process ID>
I don’t remember if you have to do a discovery after this or if it will return to normal on the next polling cycle.
Interested to hear if this solves it for you – also interested to hear what firmware version you are seeing this on. Cheers, Nathan
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Serghei Golipad Sent: Friday, 8 May 2015 1:17 AM To: Observium Network Observation System Subject: Re: [Observium] Fortigate Ports Disappeared
Hi Adam,
Anything I should be looking for in the debug? I can see ifNames correctly displayed during debug however those don’t make it into portal.
Also in event log I get below for all discovered ports:
Interface changed: [ifDescr] TEST_PORT -> NULL; [ifType] other -> NULL; [ifDuplex] -> 0
One of the things I’ve picked up from poller debug:
ignored (by empty ifDescr and ifName).
Thanks
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: 07 May 2015 15:45 To: observium@observium.orgmailto:observium@observium.org Subject: Re: [Observium] Fortigate Ports Disappeared
snmpwalk doesn't prove anything, since we don't use snmpwalk. Checking the output from the poller module running in debug might, though.
./poller.php -h <host> -m ports -d
adam.
Sent from Mailbirdhttp://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm_campaign=sent-from-mailbird
On 07/05/2015 15:38:44, Serghei Golipad <s.golipad@kelway.commailto:s.golipad@kelway.com> wrote: Hi Everyone,
Has anyone else had this problem? All of a sudden some of the critical ports have disappeared, first though it was SNMP process crash on the firewall however simple snmpwalk proves that’s not the case. I then deleted device including RRDs and re-added, now none of the ports are showing ☹
[cid:image001.png@01D088DB.DC99D9B0]
I’m on 6404.
Thanks
PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL.
Kelway Ltd is a Company registered in England and Wales. This email is sent for and on behalf of Kelway Holdings Limited. Kelway Holdings Limited is the UK holding company for Kelway Limited trading as Kelway. The registered office for Kelway Holdings Limited (Company Number 05872067) and its UK group undertakings is 10 Fleet Place, London, EC4M 7RB. VAT Registration 902 1949 39. Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than Kelway or the sender of its existence.
For more information about Kelway please visit www.kelway.comhttp://www.kelway.com
Hi Nathan,
That was spot on :D thank you very much! This problem was been screwing our traffic accounting graphs and causing all sorts of aggravation ☺ We are on: v5.0,build3608,140409 (GA Patch 7) time to upgrade, I know.
P.S if only there was a fix to sessions being graphed ☹
Cheers
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Nathan Phelan Sent: 08 May 2015 05:58 To: Observium Network Observation System Subject: Re: [Observium] Fortigate Ports Disappeared
Hi Serghei, This is an issue we have run into several times. Every time it has been due to a SNMPD bug on the fortigates. There is an issue with several firmware versions that returns invalid values for interfaces that have been recently deleted (usually vlan subinterfaces or ipsec interfaces). When this has happened snmp on the fortigates will still return values for everything apart from anything referencing the deleted interface (which is still in snmp somewhere). I have logged a ticket with Fortinet about this (in Feb 2014) and it is supposed to be fixed in newer firmware versions (although I have seen it crop up again on ‘fixed’ versions).
When it is in this state snmpwalk actually fails the following query snmpwalk -v2c -c public fortigateIPAddress ifType Error in packet. Reason: (genError) A general failure occured Failed object: IF-MIB::ifType.45 (in this example 45 was snmp index of the deleted interface).
The fix for us has always been to kill and restart snmpd on the Fortigate (on all cluster members if it is a cluster). You’ll need to find the process ID of the snmpd process and then kill it; watchdog will automatically restart it and then it’ll be fine.
show running processes diag sys top 30 99
kill snmpd process by process id diag sys kill 11 <process ID>
I don’t remember if you have to do a discovery after this or if it will return to normal on the next polling cycle.
Interested to hear if this solves it for you – also interested to hear what firmware version you are seeing this on. Cheers, Nathan
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Serghei Golipad Sent: Friday, 8 May 2015 1:17 AM To: Observium Network Observation System Subject: Re: [Observium] Fortigate Ports Disappeared
Hi Adam,
Anything I should be looking for in the debug? I can see ifNames correctly displayed during debug however those don’t make it into portal.
Also in event log I get below for all discovered ports:
Interface changed: [ifDescr] TEST_PORT -> NULL; [ifType] other -> NULL; [ifDuplex] -> 0
One of the things I’ve picked up from poller debug:
ignored (by empty ifDescr and ifName).
Thanks
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: 07 May 2015 15:45 To: observium@observium.orgmailto:observium@observium.org Subject: Re: [Observium] Fortigate Ports Disappeared
snmpwalk doesn't prove anything, since we don't use snmpwalk. Checking the output from the poller module running in debug might, though.
./poller.php -h <host> -m ports -d
adam.
Sent from Mailbirdhttp://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm_campaign=sent-from-mailbird
On 07/05/2015 15:38:44, Serghei Golipad <s.golipad@kelway.commailto:s.golipad@kelway.com> wrote: Hi Everyone,
Has anyone else had this problem? All of a sudden some of the critical ports have disappeared, first though it was SNMP process crash on the firewall however simple snmpwalk proves that’s not the case. I then deleted device including RRDs and re-added, now none of the ports are showing ☹
[cid:image001.png@01D088DB.DC99D9B0]
I’m on 6404.
Thanks
PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL.
Kelway Ltd is a Company registered in England and Wales. This email is sent for and on behalf of Kelway Holdings Limited. Kelway Holdings Limited is the UK holding company for Kelway Limited trading as Kelway. The registered office for Kelway Holdings Limited (Company Number 05872067) and its UK group undertakings is 10 Fleet Place, London, EC4M 7RB. VAT Registration 902 1949 39. Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than Kelway or the sender of its existence.
For more information about Kelway please visit www.kelway.comhttp://www.kelway.com
Hi Serghei
there is a patch here: http://jira.observium.org/browse/OBSERVIUM-1278?jql=
It works for me... You can try it on your own risk :)
Kind regards Basile
2015-05-08 10:22 GMT+02:00 Serghei Golipad s.golipad@kelway.com:
Hi Nathan,
That was spot on :D thank you very much! This problem was been screwing our traffic accounting graphs and causing all sorts of aggravation J We are on: v5.0,build3608,140409 (GA Patch 7) time to upgrade, I know.
P.S if only there was a fix to sessions being graphed L
Cheers
*From:* observium [mailto:observium-bounces@observium.org] *On Behalf Of *Nathan Phelan *Sent:* 08 May 2015 05:58
*To:* Observium Network Observation System *Subject:* Re: [Observium] Fortigate Ports Disappeared
Hi Serghei,
This is an issue we have run into several times. Every time it has been due to a SNMPD bug on the fortigates.
There is an issue with several firmware versions that returns invalid values for interfaces that have been recently deleted (usually vlan subinterfaces or ipsec interfaces).
When this has happened snmp on the fortigates will still return values for everything apart from anything referencing the deleted interface (which is still in snmp somewhere).
I have logged a ticket with Fortinet about this (in Feb 2014) and it is supposed to be fixed in newer firmware versions (although I have seen it crop up again on ‘fixed’ versions).
When it is in this state snmpwalk actually fails the following query
snmpwalk -v2c -c public fortigateIPAddress ifType
Error in packet. Reason: (genError) A general failure occured Failed object: IF-MIB::ifType.45 (in this example 45 was snmp index of the deleted interface).
The fix for us has always been to kill and restart snmpd on the Fortigate (on all cluster members if it is a cluster).
You’ll need to find the process ID of the snmpd process and then kill it; watchdog will automatically restart it and then it’ll be fine.
*show running processes*
diag sys top 30 99
*kill snmpd process by process id*
diag sys kill 11 <process ID>
I don’t remember if you have to do a discovery after this or if it will return to normal on the next polling cycle.
Interested to hear if this solves it for you – also interested to hear what firmware version you are seeing this on.
Cheers,
Nathan
*From:* observium [mailto:observium-bounces@observium.org observium-bounces@observium.org] *On Behalf Of *Serghei Golipad *Sent:* Friday, 8 May 2015 1:17 AM *To:* Observium Network Observation System *Subject:* Re: [Observium] Fortigate Ports Disappeared
Hi Adam,
Anything I should be looking for in the debug? I can see ifNames correctly displayed during debug however those don’t make it into portal.
Also in event log I get below for all discovered ports:
Interface changed: [ifDescr] TEST_PORT -> NULL; [ifType] other -> NULL; [ifDuplex] -> 0
One of the things I’ve picked up from poller debug:
ignored (by empty ifDescr and ifName).
Thanks
*From:* observium [mailto:observium-bounces@observium.org observium-bounces@observium.org] *On Behalf Of *Adam Armstrong *Sent:* 07 May 2015 15:45 *To:* observium@observium.org *Subject:* Re: [Observium] Fortigate Ports Disappeared
snmpwalk doesn't prove anything, since we don't use snmpwalk. Checking the output from the poller module running in debug might, though.
./poller.php -h <host> -m ports -d
adam.
Sent from Mailbird http://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm_campaign=sent-from-mailbird
On 07/05/2015 15:38:44, Serghei Golipad s.golipad@kelway.com wrote:
Hi Everyone,
Has anyone else had this problem? All of a sudden some of the critical ports have disappeared, first though it was SNMP process crash on the firewall however simple snmpwalk proves that’s not the case. I then deleted device including RRDs and re-added, now none of the ports are showing L
[image: cid:image001.png@01D088DB.DC99D9B0]
I’m on 6404.
Thanks
PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL.
Kelway Ltd is a Company registered in England and Wales. This email is sent for and on behalf of Kelway Holdings Limited. Kelway Holdings Limited is the UK holding company for Kelway Limited trading as Kelway. The registered office for Kelway Holdings Limited (Company Number 05872067) and its UK group undertakings is 10 Fleet Place, London, EC4M 7RB. VAT Registration 902 1949 39. Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than Kelway or the sender of its existence.
For more information about Kelway please visit www.kelway.com
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Basile,
I’ve tried that one, patch fails. Are you able to send a copy of your fortigate.php.inc? P.S does it fix CPU graphs too? Also any reason why there’s CPU and Processor graphs?
Thanks
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Basile Bluntschli Sent: 08 May 2015 13:57 To: Observium Network Observation System Subject: Re: [Observium] Fortigate Ports Disappeared
Hi Serghei there is a patch here: http://jira.observium.org/browse/OBSERVIUM-1278?jql= It works for me... You can try it on your own risk :) Kind regards Basile
2015-05-08 10:22 GMT+02:00 Serghei Golipad <s.golipad@kelway.commailto:s.golipad@kelway.com>: Hi Nathan,
That was spot on :D thank you very much! This problem was been screwing our traffic accounting graphs and causing all sorts of aggravation ☺ We are on: v5.0,build3608,140409 (GA Patch 7) time to upgrade, I know.
P.S if only there was a fix to sessions being graphed ☹
Cheers
From: observium [mailto:observium-bounces@observium.orgmailto:observium-bounces@observium.org] On Behalf Of Nathan Phelan Sent: 08 May 2015 05:58
To: Observium Network Observation System Subject: Re: [Observium] Fortigate Ports Disappeared
Hi Serghei, This is an issue we have run into several times. Every time it has been due to a SNMPD bug on the fortigates. There is an issue with several firmware versions that returns invalid values for interfaces that have been recently deleted (usually vlan subinterfaces or ipsec interfaces). When this has happened snmp on the fortigates will still return values for everything apart from anything referencing the deleted interface (which is still in snmp somewhere). I have logged a ticket with Fortinet about this (in Feb 2014) and it is supposed to be fixed in newer firmware versions (although I have seen it crop up again on ‘fixed’ versions).
When it is in this state snmpwalk actually fails the following query snmpwalk -v2c -c public fortigateIPAddress ifType Error in packet. Reason: (genError) A general failure occured Failed object: IF-MIB::ifType.45 (in this example 45 was snmp index of the deleted interface).
The fix for us has always been to kill and restart snmpd on the Fortigate (on all cluster members if it is a cluster). You’ll need to find the process ID of the snmpd process and then kill it; watchdog will automatically restart it and then it’ll be fine.
show running processes diag sys top 30 99
kill snmpd process by process id diag sys kill 11 <process ID>
I don’t remember if you have to do a discovery after this or if it will return to normal on the next polling cycle.
Interested to hear if this solves it for you – also interested to hear what firmware version you are seeing this on. Cheers, Nathan
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Serghei Golipad Sent: Friday, 8 May 2015 1:17 AM To: Observium Network Observation System Subject: Re: [Observium] Fortigate Ports Disappeared
Hi Adam,
Anything I should be looking for in the debug? I can see ifNames correctly displayed during debug however those don’t make it into portal.
Also in event log I get below for all discovered ports:
Interface changed: [ifDescr] TEST_PORT -> NULL; [ifType] other -> NULL; [ifDuplex] -> 0
One of the things I’ve picked up from poller debug:
ignored (by empty ifDescr and ifName).
Thanks
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: 07 May 2015 15:45 To: observium@observium.orgmailto:observium@observium.org Subject: Re: [Observium] Fortigate Ports Disappeared
snmpwalk doesn't prove anything, since we don't use snmpwalk. Checking the output from the poller module running in debug might, though.
./poller.php -h <host> -m ports -d
adam.
Sent from Mailbirdhttp://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm_campaign=sent-from-mailbird
On 07/05/2015 15:38:44, Serghei Golipad <s.golipad@kelway.commailto:s.golipad@kelway.com> wrote: Hi Everyone,
Has anyone else had this problem? All of a sudden some of the critical ports have disappeared, first though it was SNMP process crash on the firewall however simple snmpwalk proves that’s not the case. I then deleted device including RRDs and re-added, now none of the ports are showing ☹
I’m on 6404.
Thanks
PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL.
Kelway Ltd is a Company registered in England and Wales. This email is sent for and on behalf of Kelway Holdings Limited. Kelway Holdings Limited is the UK holding company for Kelway Limited trading as Kelway. The registered office for Kelway Holdings Limited (Company Number 05872067) and its UK group undertakings is 10 Fleet Place, London, EC4M 7RB. VAT Registration 902 1949 39. Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than Kelway or the sender of its existence.
For more information about Kelway please visit www.kelway.comhttp://www.kelway.com
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Serghei
Fortigate CPU was fixed somewhere in August 2014. The same change broke back then the Firewall Session counter. You may want to update to the current svn version? And then reapply my patch? This should work.
Kind regards Basile
2015-05-08 15:23 GMT+02:00 Serghei Golipad s.golipad@kelway.com:
Hi Basile,
I’ve tried that one, patch fails. Are you able to send a copy of your fortigate.php.inc?
P.S does it fix CPU graphs too? Also any reason why there’s CPU and Processor graphs?
Thanks
*From:* observium [mailto:observium-bounces@observium.org] *On Behalf Of *Basile Bluntschli *Sent:* 08 May 2015 13:57
*To:* Observium Network Observation System *Subject:* Re: [Observium] Fortigate Ports Disappeared
Hi Serghei
there is a patch here: http://jira.observium.org/browse/OBSERVIUM-1278?jql=
It works for me... You can try it on your own risk :)
Kind regards
Basile
2015-05-08 10:22 GMT+02:00 Serghei Golipad s.golipad@kelway.com:
Hi Nathan,
That was spot on :D thank you very much! This problem was been screwing our traffic accounting graphs and causing all sorts of aggravation J We are on: v5.0,build3608,140409 (GA Patch 7) time to upgrade, I know.
P.S if only there was a fix to sessions being graphed L
Cheers
*From:* observium [mailto:observium-bounces@observium.org] *On Behalf Of *Nathan Phelan *Sent:* 08 May 2015 05:58
*To:* Observium Network Observation System *Subject:* Re: [Observium] Fortigate Ports Disappeared
Hi Serghei,
This is an issue we have run into several times. Every time it has been due to a SNMPD bug on the fortigates.
There is an issue with several firmware versions that returns invalid values for interfaces that have been recently deleted (usually vlan subinterfaces or ipsec interfaces).
When this has happened snmp on the fortigates will still return values for everything apart from anything referencing the deleted interface (which is still in snmp somewhere).
I have logged a ticket with Fortinet about this (in Feb 2014) and it is supposed to be fixed in newer firmware versions (although I have seen it crop up again on ‘fixed’ versions).
When it is in this state snmpwalk actually fails the following query
snmpwalk -v2c -c public fortigateIPAddress ifType
Error in packet. Reason: (genError) A general failure occured Failed object: IF-MIB::ifType.45 (in this example 45 was snmp index of the deleted interface).
The fix for us has always been to kill and restart snmpd on the Fortigate (on all cluster members if it is a cluster).
You’ll need to find the process ID of the snmpd process and then kill it; watchdog will automatically restart it and then it’ll be fine.
*show running processes*
diag sys top 30 99
*kill snmpd process by process id*
diag sys kill 11 <process ID>
I don’t remember if you have to do a discovery after this or if it will return to normal on the next polling cycle.
Interested to hear if this solves it for you – also interested to hear what firmware version you are seeing this on.
Cheers,
Nathan
*From:* observium [mailto:observium-bounces@observium.org observium-bounces@observium.org] *On Behalf Of *Serghei Golipad *Sent:* Friday, 8 May 2015 1:17 AM *To:* Observium Network Observation System *Subject:* Re: [Observium] Fortigate Ports Disappeared
Hi Adam,
Anything I should be looking for in the debug? I can see ifNames correctly displayed during debug however those don’t make it into portal.
Also in event log I get below for all discovered ports:
Interface changed: [ifDescr] TEST_PORT -> NULL; [ifType] other -> NULL; [ifDuplex] -> 0
One of the things I’ve picked up from poller debug:
ignored (by empty ifDescr and ifName).
Thanks
*From:* observium [mailto:observium-bounces@observium.org observium-bounces@observium.org] *On Behalf Of *Adam Armstrong *Sent:* 07 May 2015 15:45 *To:* observium@observium.org *Subject:* Re: [Observium] Fortigate Ports Disappeared
snmpwalk doesn't prove anything, since we don't use snmpwalk. Checking the output from the poller module running in debug might, though.
./poller.php -h <host> -m ports -d
adam.
Sent from Mailbird http://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm_campaign=sent-from-mailbird
On 07/05/2015 15:38:44, Serghei Golipad s.golipad@kelway.com wrote:
Hi Everyone,
Has anyone else had this problem? All of a sudden some of the critical ports have disappeared, first though it was SNMP process crash on the firewall however simple snmpwalk proves that’s not the case. I then deleted device including RRDs and re-added, now none of the ports are showing L
I’m on 6404.
Thanks
PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL.
Kelway Ltd is a Company registered in England and Wales. This email is sent for and on behalf of Kelway Holdings Limited. Kelway Holdings Limited is the UK holding company for Kelway Limited trading as Kelway. The registered office for Kelway Holdings Limited (Company Number 05872067) and its UK group undertakings is 10 Fleet Place, London, EC4M 7RB. VAT Registration 902 1949 39. Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than Kelway or the sender of its existence.
For more information about Kelway please visit www.kelway.com
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Basile,
Am I doing something wrong here?
observ01:/opt/observium/includes/polling/os# patch -p1 < r6384.patch can't find file to patch at input line 5 Perhaps you used the wrong -p or --strip option? The text leading up to this was: -------------------------- |Index: fortigate.inc.php |=================================================================== |--- fortigate.inc.php (revision 6384) |+++ fortigate.inc.php (working copy) -------------------------- File to patch: fortigate.inc.php patching file fortigate.inc.php Hunk #1 FAILED at 11. patch unexpectedly ends in middle of line Hunk #2 FAILED at 23. 2 out of 2 hunks FAILED -- saving rejects to file fortigate.inc.php.rej root@uk1soc-observ01:/opt/observium/includes/polling/os#
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Basile Bluntschli Sent: 08 May 2015 15:34 To: Observium Network Observation System Subject: Re: [Observium] Fortigate Ports Disappeared
Hi Serghei Fortigate CPU was fixed somewhere in August 2014. The same change broke back then the Firewall Session counter. You may want to update to the current svn version? And then reapply my patch? This should work. Kind regards Basile
2015-05-08 15:23 GMT+02:00 Serghei Golipad <s.golipad@kelway.commailto:s.golipad@kelway.com>: Hi Basile,
I’ve tried that one, patch fails. Are you able to send a copy of your fortigate.php.inc? P.S does it fix CPU graphs too? Also any reason why there’s CPU and Processor graphs?
Thanks
From: observium [mailto:observium-bounces@observium.orgmailto:observium-bounces@observium.org] On Behalf Of Basile Bluntschli Sent: 08 May 2015 13:57
To: Observium Network Observation System Subject: Re: [Observium] Fortigate Ports Disappeared
Hi Serghei there is a patch here: http://jira.observium.org/browse/OBSERVIUM-1278?jql= It works for me... You can try it on your own risk :) Kind regards Basile
2015-05-08 10:22 GMT+02:00 Serghei Golipad <s.golipad@kelway.commailto:s.golipad@kelway.com>: Hi Nathan,
That was spot on :D thank you very much! This problem was been screwing our traffic accounting graphs and causing all sorts of aggravation ☺ We are on: v5.0,build3608,140409 (GA Patch 7) time to upgrade, I know.
P.S if only there was a fix to sessions being graphed ☹
Cheers
From: observium [mailto:observium-bounces@observium.orgmailto:observium-bounces@observium.org] On Behalf Of Nathan Phelan Sent: 08 May 2015 05:58
To: Observium Network Observation System Subject: Re: [Observium] Fortigate Ports Disappeared
Hi Serghei, This is an issue we have run into several times. Every time it has been due to a SNMPD bug on the fortigates. There is an issue with several firmware versions that returns invalid values for interfaces that have been recently deleted (usually vlan subinterfaces or ipsec interfaces). When this has happened snmp on the fortigates will still return values for everything apart from anything referencing the deleted interface (which is still in snmp somewhere). I have logged a ticket with Fortinet about this (in Feb 2014) and it is supposed to be fixed in newer firmware versions (although I have seen it crop up again on ‘fixed’ versions).
When it is in this state snmpwalk actually fails the following query snmpwalk -v2c -c public fortigateIPAddress ifType Error in packet. Reason: (genError) A general failure occured Failed object: IF-MIB::ifType.45 (in this example 45 was snmp index of the deleted interface).
The fix for us has always been to kill and restart snmpd on the Fortigate (on all cluster members if it is a cluster). You’ll need to find the process ID of the snmpd process and then kill it; watchdog will automatically restart it and then it’ll be fine.
show running processes diag sys top 30 99
kill snmpd process by process id diag sys kill 11 <process ID>
I don’t remember if you have to do a discovery after this or if it will return to normal on the next polling cycle.
Interested to hear if this solves it for you – also interested to hear what firmware version you are seeing this on. Cheers, Nathan
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Serghei Golipad Sent: Friday, 8 May 2015 1:17 AM To: Observium Network Observation System Subject: Re: [Observium] Fortigate Ports Disappeared
Hi Adam,
Anything I should be looking for in the debug? I can see ifNames correctly displayed during debug however those don’t make it into portal.
Also in event log I get below for all discovered ports:
Interface changed: [ifDescr] TEST_PORT -> NULL; [ifType] other -> NULL; [ifDuplex] -> 0
One of the things I’ve picked up from poller debug:
ignored (by empty ifDescr and ifName).
Thanks
From: observium [mailto:observium-bounces@observium.org] On Behalf Of Adam Armstrong Sent: 07 May 2015 15:45 To: observium@observium.orgmailto:observium@observium.org Subject: Re: [Observium] Fortigate Ports Disappeared
snmpwalk doesn't prove anything, since we don't use snmpwalk. Checking the output from the poller module running in debug might, though.
./poller.php -h <host> -m ports -d
adam.
Sent from Mailbirdhttp://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm_campaign=sent-from-mailbird
On 07/05/2015 15:38:44, Serghei Golipad <s.golipad@kelway.commailto:s.golipad@kelway.com> wrote: Hi Everyone,
Has anyone else had this problem? All of a sudden some of the critical ports have disappeared, first though it was SNMP process crash on the firewall however simple snmpwalk proves that’s not the case. I then deleted device including RRDs and re-added, now none of the ports are showing ☹
I’m on 6404.
Thanks
PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL.
Kelway Ltd is a Company registered in England and Wales. This email is sent for and on behalf of Kelway Holdings Limited. Kelway Holdings Limited is the UK holding company for Kelway Limited trading as Kelway. The registered office for Kelway Holdings Limited (Company Number 05872067) and its UK group undertakings is 10 Fleet Place, London, EC4M 7RB. VAT Registration 902 1949 39. Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than Kelway or the sender of its existence.
For more information about Kelway please visit www.kelway.comhttp://www.kelway.com
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Serghei
I have no idea how to apply a svn diff file to an existing file. I just followed the instructions on the Observium website on how to submit a patch :) I would need to use google as well... Else you could manually edit the fortigate.inc.pcp file, the patch provides something like three or four lines of code change... That should be doable as well.
Kind regards Basile
2015-05-08 16:42 GMT+02:00 Serghei Golipad s.golipad@kelway.com:
Hi Basile,
Am I doing something wrong here?
observ01:/opt/observium/includes/polling/os# patch -p1 < r6384.patch
can't find file to patch at input line 5
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
|Index: fortigate.inc.php
|===================================================================
|--- fortigate.inc.php (revision 6384)
|+++ fortigate.inc.php (working copy)
File to patch: fortigate.inc.php
patching file fortigate.inc.php
Hunk #1 FAILED at 11.
patch unexpectedly ends in middle of line
Hunk #2 FAILED at 23.
2 out of 2 hunks FAILED -- saving rejects to file fortigate.inc.php.rej
root@uk1soc-observ01:/opt/observium/includes/polling/os#
*From:* observium [mailto:observium-bounces@observium.org] *On Behalf Of *Basile Bluntschli *Sent:* 08 May 2015 15:34
*To:* Observium Network Observation System *Subject:* Re: [Observium] Fortigate Ports Disappeared
Hi Serghei
Fortigate CPU was fixed somewhere in August 2014. The same change broke back then the Firewall Session counter.
You may want to update to the current svn version? And then reapply my patch?
This should work.
Kind regards
Basile
2015-05-08 15:23 GMT+02:00 Serghei Golipad s.golipad@kelway.com:
Hi Basile,
I’ve tried that one, patch fails. Are you able to send a copy of your fortigate.php.inc?
P.S does it fix CPU graphs too? Also any reason why there’s CPU and Processor graphs?
Thanks
*From:* observium [mailto:observium-bounces@observium.org] *On Behalf Of *Basile Bluntschli *Sent:* 08 May 2015 13:57
*To:* Observium Network Observation System *Subject:* Re: [Observium] Fortigate Ports Disappeared
Hi Serghei
there is a patch here: http://jira.observium.org/browse/OBSERVIUM-1278?jql=
It works for me... You can try it on your own risk :)
Kind regards
Basile
2015-05-08 10:22 GMT+02:00 Serghei Golipad s.golipad@kelway.com:
Hi Nathan,
That was spot on :D thank you very much! This problem was been screwing our traffic accounting graphs and causing all sorts of aggravation J We are on: v5.0,build3608,140409 (GA Patch 7) time to upgrade, I know.
P.S if only there was a fix to sessions being graphed L
Cheers
*From:* observium [mailto:observium-bounces@observium.org] *On Behalf Of *Nathan Phelan *Sent:* 08 May 2015 05:58
*To:* Observium Network Observation System *Subject:* Re: [Observium] Fortigate Ports Disappeared
Hi Serghei,
This is an issue we have run into several times. Every time it has been due to a SNMPD bug on the fortigates.
There is an issue with several firmware versions that returns invalid values for interfaces that have been recently deleted (usually vlan subinterfaces or ipsec interfaces).
When this has happened snmp on the fortigates will still return values for everything apart from anything referencing the deleted interface (which is still in snmp somewhere).
I have logged a ticket with Fortinet about this (in Feb 2014) and it is supposed to be fixed in newer firmware versions (although I have seen it crop up again on ‘fixed’ versions).
When it is in this state snmpwalk actually fails the following query
snmpwalk -v2c -c public fortigateIPAddress ifType
Error in packet. Reason: (genError) A general failure occured Failed object: IF-MIB::ifType.45 (in this example 45 was snmp index of the deleted interface).
The fix for us has always been to kill and restart snmpd on the Fortigate (on all cluster members if it is a cluster).
You’ll need to find the process ID of the snmpd process and then kill it; watchdog will automatically restart it and then it’ll be fine.
*show running processes*
diag sys top 30 99
*kill snmpd process by process id*
diag sys kill 11 <process ID>
I don’t remember if you have to do a discovery after this or if it will return to normal on the next polling cycle.
Interested to hear if this solves it for you – also interested to hear what firmware version you are seeing this on.
Cheers,
Nathan
*From:* observium [mailto:observium-bounces@observium.org observium-bounces@observium.org] *On Behalf Of *Serghei Golipad *Sent:* Friday, 8 May 2015 1:17 AM *To:* Observium Network Observation System *Subject:* Re: [Observium] Fortigate Ports Disappeared
Hi Adam,
Anything I should be looking for in the debug? I can see ifNames correctly displayed during debug however those don’t make it into portal.
Also in event log I get below for all discovered ports:
Interface changed: [ifDescr] TEST_PORT -> NULL; [ifType] other -> NULL; [ifDuplex] -> 0
One of the things I’ve picked up from poller debug:
ignored (by empty ifDescr and ifName).
Thanks
*From:* observium [mailto:observium-bounces@observium.org observium-bounces@observium.org] *On Behalf Of *Adam Armstrong *Sent:* 07 May 2015 15:45 *To:* observium@observium.org *Subject:* Re: [Observium] Fortigate Ports Disappeared
snmpwalk doesn't prove anything, since we don't use snmpwalk. Checking the output from the poller module running in debug might, though.
./poller.php -h <host> -m ports -d
adam.
Sent from Mailbird http://www.getmailbird.com/?utm_source=Mailbird&utm_medium=email&utm_campaign=sent-from-mailbird
On 07/05/2015 15:38:44, Serghei Golipad s.golipad@kelway.com wrote:
Hi Everyone,
Has anyone else had this problem? All of a sudden some of the critical ports have disappeared, first though it was SNMP process crash on the firewall however simple snmpwalk proves that’s not the case. I then deleted device including RRDs and re-added, now none of the ports are showing L
I’m on 6404.
Thanks
PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS EMAIL.
Kelway Ltd is a Company registered in England and Wales. This email is sent for and on behalf of Kelway Holdings Limited. Kelway Holdings Limited is the UK holding company for Kelway Limited trading as Kelway. The registered office for Kelway Holdings Limited (Company Number 05872067) and its UK group undertakings is 10 Fleet Place, London, EC4M 7RB. VAT Registration 902 1949 39. Confidentiality: This e-mail contains proprietary information, some or all of which may be confidential and/or legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the author by replying to this e-mail and then delete the original. If you are not the intended recipient you may not use, disclose, distribute, copy, print or rely on any information contained in this e-mail. You must not inform any other person other than Kelway or the sender of its existence.
For more information about Kelway please visit www.kelway.com
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
участники (4)
-
Adam Armstrong -
Basile Bluntschli -
Nathan Phelan -
Serghei Golipad