Re: ... subscription has been disabled from email nikto.barrada@gmail.com
I don't know why... But exactly the same for me. Last message I received on the mailing list was on feb 15.
/Tobias
Google blocking stuff, very annoying. Microsoft/O365/Live is just as annoying.
adam.
------ Original Message ------ From "Tobias Cederlund via observium" observium@lists.observium.org To observium@lists.observium.org Cc "Tobias Cederlund" tobias.cederlund@gmail.com Date 21/11/2024 22:36:47 Subject [Observium] Re: ... subscription has been disabled from email nikto.barrada@gmail.com
I don't know why... But exactly the same for me. Last message I received on the mailing list was on feb 15.
/Tobias
Hi all,
i have a problem with the quick search in my installation of observium professional. The results are displayed with a big delay. It varies from 3 to 10 seconds.
Everything else is very fast (in my opinion): [cid:image001.png@01DB3F50.B190C4F0]
Sometimes only a part of the string i entered is searched. My MariaDB is on another virtual server. The problem started when i upgraded the OS from 10 to 12 a month ago.
Does anybody have an idea how i can troubleshoot this issue?
Version Information Observium
24.10.13681 (20th October 2024)
OS
Linux 5.15.0-126-generic [amd64] (Ubuntu 22.04)
Apache
2.4.52 (Ubuntu)
PHP
8.1.2-1ubuntu2.19 (OPcache: ENABLED) (Memory: 1GB)
Python
3.10.12
MariaDB
10.11.6-MariaDB-0+deb12u1 (extension: mysqli 8.1.2-1ubuntu2.19)
SNMP
NET-SNMP 5.9.1
RRDtool
1.7.2 (rrdcached 1.7.2: unix:/var/run/rrdcached.sock)
Fping
5.1 (IPv4 and IPv6)
Fetch
cURL 7.81.0 (OpenSSL/3.0.2, LibZ 1.2.11, LibIDN 2.3.2)
Browser & Timezone Information User-Agent
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36
Browser
Chrome 130.0 (Windows)
Screen Resolution
3840x1600
Timezone
+01:00 (System), +01:00 (PHP), +01:00 (DB), +01:00 (User)
Statistics DB size
15.5GB
RRD size
116GB
Devices
779
Ports
41280
IPv4 Addresses
3271
IPv4 Networks
3497
IPv6 Addresses
287
IPv6 Networks
21
Services
Applications
5
Processors
1330
Memory pools
1055
Storage Entries
858
Disk I/O Entries
1996
HR-MIB Entries
875
Entity-MIB Entries
52783
Syslog Entries
11299175
Eventlog Entries
11656552
Sensors
135228
Printer Supplies
0
Netscaler VServers
641
Netscaler Services
503
Virtual Machines
0
IP SLAs
Kind regards, Florian
First truncate the syslog and eventlog tables, 11 million entries is unreasonable.
Make sure you have housekeeping configured to keep the size of those tables down, MySQL is really unsuitable for large log storage.
You need to truncate the tables first though, because housekeeping will take forever on those large tables.
After those tables are at a manageable size you can see if it’s any faster.
Adam.
Sent from my iPhone
On 25 Nov 2024, at 14:49, Meyer, Florian via observium observium@lists.observium.org wrote:
Hi all,
i have a problem with the quick search in my installation of observium professional. The results are displayed with a big delay. It varies from 3 to 10 seconds.
Everything else is very fast (in my opinion): <image001.png>
Sometimes only a part of the string i entered is searched. My MariaDB is on another virtual server. The problem started when i upgraded the OS from 10 to 12 a month ago.
Does anybody have an idea how i can troubleshoot this issue?
Version Information Observium 24.10.13681 (20th October 2024) OS Linux 5.15.0-126-generic [amd64] (Ubuntu 22.04) Apache 2.4.52 (Ubuntu) PHP 8.1.2-1ubuntu2.19 (OPcache: ENABLED) (Memory: 1GB) Python 3.10.12 MariaDB 10.11.6-MariaDB-0+deb12u1 (extension: mysqli 8.1.2-1ubuntu2.19) SNMP NET-SNMP 5.9.1 RRDtool 1.7.2 (rrdcached 1.7.2: unix:/var/run/rrdcached.sock) Fping 5.1 (IPv4 and IPv6) Fetch cURL 7.81.0 (OpenSSL/3.0.2, LibZ 1.2.11, LibIDN 2.3.2) Browser & Timezone Information User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Browser Chrome 130.0 (Windows) Screen Resolution 3840x1600 Timezone +01:00 (System), +01:00 (PHP), +01:00 (DB), +01:00 (User)
Statistics DB size 15.5GB RRD size 116GB Devices 779 Ports 41280 IPv4 Addresses 3271 IPv4 Networks 3497 IPv6 Addresses 287 IPv6 Networks 21 Services Applications 5 Processors 1330 Memory pools 1055 Storage Entries 858 Disk I/O Entries 1996 HR-MIB Entries 875 Entity-MIB Entries 52783 Syslog Entries 11299175 Eventlog Entries 11656552 Sensors 135228 Printer Supplies 0 Netscaler VServers 641 Netscaler Services 503 Virtual Machines 0 IP SLAs
Kind regards, Florian
observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
Hi Adam,
thanks for you reply. You are right 11 million entries look quite much. I had the housekeeping setting for syslog to keep 30 days and eventlogt o keep 365 days. I reduced both now to 7 days. Housekeeping script crunched like half an hour on those entries. I reduced syslog to 2.5 million and eventlog to 232k. +-------------------------------+-----------+ | Table | Size (MB) | +-------------------------------+-----------+ | eventlog | 5469 | | syslog | 3275 | | accesspoints-state | 2861 | | notifications_queue | 902 | | alert_log | 464 | | vlans_fdb | 248 | | bill_data | 131 |
But the problem still exist. The response times of the quick search is not faster. :(
Any other advise?
Kind regards, Florian
Von: Adam Armstrong via observium observium@lists.observium.org Gesendet: Montag, 25. November 2024 17:09 An: Observium observium@lists.observium.org Cc: Adam Armstrong adama@observium.org Betreff: [Observium] Re: Quick Search Slow - how to troubleshoot?
EXTERNER ABSENDER: Klicken Sie nur dann auf Links und Anhänge, wenn Sie dem Absender der Nachricht vertrauen. Diese Nachricht stammt von außerhalb der UMR! First truncate the syslog and eventlog tables, 11 million entries is unreasonable.
Make sure you have housekeeping configured to keep the size of those tables down, MySQL is really unsuitable for large log storage.
You need to truncate the tables first though, because housekeeping will take forever on those large tables.
After those tables are at a manageable size you can see if it’s any faster.
Adam.
Sent from my iPhone
On 25 Nov 2024, at 14:49, Meyer, Florian via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> wrote: Hi all,
i have a problem with the quick search in my installation of observium professional. The results are displayed with a big delay. It varies from 3 to 10 seconds.
Everything else is very fast (in my opinion): <image001.png>
Sometimes only a part of the string i entered is searched. My MariaDB is on another virtual server. The problem started when i upgraded the OS from 10 to 12 a month ago.
Does anybody have an idea how i can troubleshoot this issue?
Version Information Observium
24.10.13681 (20th October 2024)
OS
Linux 5.15.0-126-generic [amd64] (Ubuntu 22.04)
Apache
2.4.52 (Ubuntu)
PHP
8.1.2-1ubuntu2.19 (OPcache: ENABLED) (Memory: 1GB)
Python
3.10.12
MariaDB
10.11.6-MariaDB-0+deb12u1 (extension: mysqli 8.1.2-1ubuntu2.19)
SNMP
NET-SNMP 5.9.1
RRDtool
1.7.2 (rrdcached 1.7.2: unix:/var/run/rrdcached.sock)
Fping
5.1 (IPv4 and IPv6)
Fetch
cURL 7.81.0 (OpenSSL/3.0.2, LibZ 1.2.11, LibIDN 2.3.2)
Browser & Timezone Information User-Agent
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36
Browser
Chrome 130.0 (Windows)
Screen Resolution
3840x1600
Timezone
+01:00 (System), +01:00 (PHP), +01:00 (DB), +01:00 (User)
Statistics DB size
15.5GB
RRD size
116GB
Devices
779
Ports
41280
IPv4 Addresses
3271
IPv4 Networks
3497
IPv6 Addresses
287
IPv6 Networks
21
Services
Applications
5
Processors
1330
Memory pools
1055
Storage Entries
858
Disk I/O Entries
1996
HR-MIB Entries
875
Entity-MIB Entries
52783
Syslog Entries
11299175
Eventlog Entries
11656552
Sensors
135228
Printer Supplies
0
Netscaler VServers
641
Netscaler Services
503
Virtual Machines
0
IP SLAs
Kind regards, Florian
_______________________________________________ observium mailing list -- observium@lists.observium.orgmailto:observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.orgmailto:observium-leave@lists.observium.org
We don't really have any profiling on the search, but what it does is run a bunch of queries on all of the tables, which can be quite slow on some very large installs.
You /might/ want to export and recreate the database to free up the disk space from the eventlog and syslog tables, but if you aren't short of space you can leave it.
It's odd that the accesspoints-state and notifications_queue tables are so large. How many rows do they have?
I'm not sure if the AP table is used in the search query, if it's very large it might be the cause.
If that's not the cause we might be able to figure out which specific queries are slow.
Are you able to provide ssh/http access to the system so I can figure out which bits are slow and see if I can make them faster? (the alternative is a dump of the database, but that's usually difficult for the same reason!)
adam.
Meyer, Florian via observium wrote on 26/11/2024 17:02:
Hi Adam,
thanks for you reply. You are right 11 million entries look quite much. I had the housekeeping setting for syslog to keep 30 days and eventlogt o keep 365 days. I reduced both now to 7 days.
Housekeeping script crunched like half an hour on those entries.
I reduced syslog to 2.5 million and eventlog to 232k.
+-------------------------------+-----------+
| Table | Size (MB) |
+-------------------------------+-----------+
| eventlog | 5469 |
| syslog | 3275 |
| accesspoints-state | 2861 |
| notifications_queue | 902 |
| alert_log | 464 |
| vlans_fdb | 248 |
| bill_data | 131 |
But the problem still exist. The response times of the quick search is not faster. :(
Any other advise?
Kind regards,
Florian
*Von:*Adam Armstrong via observium observium@lists.observium.org *Gesendet:* Montag, 25. November 2024 17:09 *An:* Observium observium@lists.observium.org *Cc:* Adam Armstrong adama@observium.org *Betreff:* [Observium] Re: Quick Search Slow - how to troubleshoot?
*EXTERNER ABSENDER: Klicken Sie nur dann auf Links und Anhänge, wenn Sie dem Absender der Nachricht vertrauen. Diese Nachricht stammt von außerhalb der UMR!*
First truncate the syslog and eventlog tables, 11 million entries is unreasonable.
Make sure you have housekeeping configured to keep the size of those tables down, MySQL is really unsuitable for large log storage.
You need to truncate the tables first though, because housekeeping will take forever on those large tables.
After those tables are at a manageable size you can see if it’s any faster.
Adam.
Sent from my iPhone
On 25 Nov 2024, at 14:49, Meyer, Florian via observium <observium@lists.observium.org <mailto:observium@lists.observium.org>> wrote: Hi all, i have a problem with the quick search in my installation of observium professional. The results are displayed with a big delay. It varies from 3 to 10 seconds. Everything else is very fast (in my opinion): <image001.png> Sometimes only a part of the string i entered is searched. My MariaDB is on another virtual server. The problem started when i upgraded the OS from 10 to 12 a month ago. Does anybody have an idea how i can troubleshoot this issue? Version Information *Observium* 24.10.13681 (20th October 2024) *OS* Linux 5.15.0-126-generic [amd64] (Ubuntu 22.04) *Apache* 2.4.52 (Ubuntu) *PHP* 8.1.2-1ubuntu2.19 (OPcache: ENABLED) (Memory: 1GB) *Python* 3.10.12 *MariaDB* 10.11.6-MariaDB-0+deb12u1 (extension: mysqli 8.1.2-1ubuntu2.19) *SNMP* NET-SNMP 5.9.1 *RRDtool* 1.7.2 (rrdcached 1.7.2: unix:/var/run/rrdcached.sock) *Fping* 5.1 (IPv4 and IPv6) *Fetch* cURL 7.81.0 (OpenSSL/3.0.2, LibZ 1.2.11, LibIDN 2.3.2) Browser & Timezone Information *User-Agent* Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 *Browser* Chrome 130.0 (Windows) *Screen Resolution* 3840x1600 *Timezone* +01:00 (System), +01:00 (PHP), +01:00 (DB), +01:00 (User) Statistics *DB size* 15.5GB *RRD size* 116GB *Devices* 779 *Ports* 41280 *IPv4 Addresses* 3271 *IPv4 Networks* 3497 *IPv6 Addresses* 287 *IPv6 Networks* 21 *Services* *Applications* 5 *Processors* 1330 *Memory pools* 1055 *Storage Entries* 858 *Disk I/O Entries* 1996 *HR-MIB Entries* 875 *Entity-MIB Entries* 52783 *Syslog Entries* 11299175 *Eventlog Entries* 11656552 *Sensors* 135228 *Printer Supplies* 0 *Netscaler VServers* 641 *Netscaler Services* 503 *Virtual Machines* 0 *IP SLAs* Kind regards, Florian _______________________________________________ observium mailing list -- observium@lists.observium.org <mailto:observium@lists.observium.org> To unsubscribe send an email to observium-leave@lists.observium.org <mailto:observium-leave@lists.observium.org>
observium mailing list -- observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.org
Okay, i enabled mysql slow query (1 second :D) logging, but there is no slow query when using quick search. I gave mysqlcheck with –optimize a try and this seems to work. Results are shown in 2-4 seconds now.
To answer your question about the two tables;
mysql> select count(*) FROM `accesspoints-state`; +----------+ | count(*) | +----------+ | 57797349 | +----------+ 1 row in set (12.70 sec)
mysql> select count(*) FROM notifications_queue; +----------+ | count(*) | +----------+ | 9120 | +----------+ 1 row in set (0.01 sec)
Looks like notifications_queue has many old notifications – can i just empty this table? I looked after some entries – they are email notifications with ALERT_URLs that do not exist. There is just a message displayed: „Unfortunately, this alert entry id does not seem to exist in the database!“ So i think they dont need to be in the database anymore.
Sorry i cant grant access without having legal hassle.. this is bad for me, but i think the response times are ok for now. But thanks for your help so far.
Kind regards Florian
Von: Adam Armstrong via observium observium@lists.observium.org Gesendet: Dienstag, 26. November 2024 18:31 An: Meyer, Florian via observium observium@lists.observium.org Cc: Adam Armstrong adama@observium.org Betreff: [Observium] Re: Quick Search Slow - how to troubleshoot?
EXTERNER ABSENDER: Klicken Sie nur dann auf Links und Anhänge, wenn Sie dem Absender der Nachricht vertrauen. Diese Nachricht stammt von außerhalb der UMR! We don't really have any profiling on the search, but what it does is run a bunch of queries on all of the tables, which can be quite slow on some very large installs.
You /might/ want to export and recreate the database to free up the disk space from the eventlog and syslog tables, but if you aren't short of space you can leave it.
It's odd that the accesspoints-state and notifications_queue tables are so large. How many rows do they have?
I'm not sure if the AP table is used in the search query, if it's very large it might be the cause.
If that's not the cause we might be able to figure out which specific queries are slow.
Are you able to provide ssh/http access to the system so I can figure out which bits are slow and see if I can make them faster? (the alternative is a dump of the database, but that's usually difficult for the same reason!)
adam.
Meyer, Florian via observium wrote on 26/11/2024 17:02:
Hi Adam,
thanks for you reply. You are right 11 million entries look quite much. I had the housekeeping setting for syslog to keep 30 days and eventlogt o keep 365 days. I reduced both now to 7 days. Housekeeping script crunched like half an hour on those entries. I reduced syslog to 2.5 million and eventlog to 232k. +-------------------------------+-----------+ | Table | Size (MB) | +-------------------------------+-----------+ | eventlog | 5469 | | syslog | 3275 | | accesspoints-state | 2861 | | notifications_queue | 902 | | alert_log | 464 | | vlans_fdb | 248 | | bill_data | 131 |
But the problem still exist. The response times of the quick search is not faster. :(
Any other advise?
Kind regards, Florian
Von: Adam Armstrong via observium observium@lists.observium.orgmailto:observium@lists.observium.org Gesendet: Montag, 25. November 2024 17:09 An: Observium observium@lists.observium.orgmailto:observium@lists.observium.org Cc: Adam Armstrong adama@observium.orgmailto:adama@observium.org Betreff: [Observium] Re: Quick Search Slow - how to troubleshoot?
EXTERNER ABSENDER: Klicken Sie nur dann auf Links und Anhänge, wenn Sie dem Absender der Nachricht vertrauen. Diese Nachricht stammt von außerhalb der UMR! First truncate the syslog and eventlog tables, 11 million entries is unreasonable.
Make sure you have housekeeping configured to keep the size of those tables down, MySQL is really unsuitable for large log storage.
You need to truncate the tables first though, because housekeeping will take forever on those large tables.
After those tables are at a manageable size you can see if it’s any faster.
Adam.
Sent from my iPhone
On 25 Nov 2024, at 14:49, Meyer, Florian via observium <observium@lists.observium.orgmailto:observium@lists.observium.org> wrote: Hi all,
i have a problem with the quick search in my installation of observium professional. The results are displayed with a big delay. It varies from 3 to 10 seconds.
Everything else is very fast (in my opinion): <image001.png>
Sometimes only a part of the string i entered is searched. My MariaDB is on another virtual server. The problem started when i upgraded the OS from 10 to 12 a month ago.
Does anybody have an idea how i can troubleshoot this issue?
Version Information Observium
24.10.13681 (20th October 2024)
OS
Linux 5.15.0-126-generic [amd64] (Ubuntu 22.04)
Apache
2.4.52 (Ubuntu)
PHP
8.1.2-1ubuntu2.19 (OPcache: ENABLED) (Memory: 1GB)
Python
3.10.12
MariaDB
10.11.6-MariaDB-0+deb12u1 (extension: mysqli 8.1.2-1ubuntu2.19)
SNMP
NET-SNMP 5.9.1
RRDtool
1.7.2 (rrdcached 1.7.2: unix:/var/run/rrdcached.sock)
Fping
5.1 (IPv4 and IPv6)
Fetch
cURL 7.81.0 (OpenSSL/3.0.2, LibZ 1.2.11, LibIDN 2.3.2)
Browser & Timezone Information User-Agent
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36
Browser
Chrome 130.0 (Windows)
Screen Resolution
3840x1600
Timezone
+01:00 (System), +01:00 (PHP), +01:00 (DB), +01:00 (User)
Statistics DB size
15.5GB
RRD size
116GB
Devices
779
Ports
41280
IPv4 Addresses
3271
IPv4 Networks
3497
IPv6 Addresses
287
IPv6 Networks
21
Services
Applications
5
Processors
1330
Memory pools
1055
Storage Entries
858
Disk I/O Entries
1996
HR-MIB Entries
875
Entity-MIB Entries
52783
Syslog Entries
11299175
Eventlog Entries
11656552
Sensors
135228
Printer Supplies
0
Netscaler VServers
641
Netscaler Services
503
Virtual Machines
0
IP SLAs
Kind regards, Florian
_______________________________________________ observium mailing list -- observium@lists.observium.orgmailto:observium@lists.observium.org To unsubscribe send an email to observium-leave@lists.observium.orgmailto:observium-leave@lists.observium.org
_______________________________________________
observium mailing list -- observium@lists.observium.orgmailto:observium@lists.observium.org
To unsubscribe send an email to observium-leave@lists.observium.orgmailto:observium-leave@lists.observium.org
-- Sent from Postboxhttps://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach
participants (3)
-
Adam Armstrong -
Meyer, Florian -
Tobias Cederlund