Flush process causing slow performace
Since upgrading PHP to 5.4, Observium has loaded very slowly at times. Most of the time just a refresh of the homepage will take .150 seconds but occasionally it can take up to a minute to load. Running iotop shows the Flush process running taking up 99.99% IO which directly correlates to the slow page loads. Can anyone shed some light on a way to resolve this?
I’ve also noticed that in iotop after the following command runs like 5 or more iterations simultaneously, the dirty pages to be written gets to over 25000:
mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql ~var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
On Oct 26, 2015, at 8:36 AM, Johnson, Edward A. <eajohnson@prf.orgmailto:eajohnson@prf.org> wrote:
Since upgrading PHP to 5.4, Observium has loaded very slowly at times. Most of the time just a refresh of the homepage will take .150 seconds but occasionally it can take up to a minute to load. Running iotop shows the Flush process running taking up 99.99% IO which directly correlates to the slow page loads. Can anyone shed some light on a way to resolve this? _______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Turned out to be a balance issue between the number of pollers, the vm.dirty_background_ratio, and the vm.dirty_ratio.
On Oct 26, 2015, at 11:17 AM, Johnson, Edward A. <eajohnson@prf.orgmailto:eajohnson@prf.org> wrote:
I’ve also noticed that in iotop after the following command runs like 5 or more iterations simultaneously, the dirty pages to be written gets to over 25000:
mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql ~var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
On Oct 26, 2015, at 8:36 AM, Johnson, Edward A. <eajohnson@prf.orgmailto:eajohnson@prf.org> wrote:
Since upgrading PHP to 5.4, Observium has loaded very slowly at times. Most of the time just a refresh of the homepage will take .150 seconds but occasionally it can take up to a minute to load. Running iotop shows the Flush process running taking up 99.99% IO which directly correlates to the slow page loads. Can anyone shed some light on a way to resolve this? _______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
That sounds... complicated.
adam. On 26/10/2015 17:14:56, Johnson, Edward A. eajohnson@prf.org wrote: Turned out to be a balance issue between the number of pollers, the vm.dirty_background_ratio, and the vm.dirty_ratio.
On Oct 26, 2015, at 11:17 AM, Johnson, Edward A. <eajohnson@prf.org [mailto:eajohnson@prf.org]> wrote:
I’ve also noticed that in iotop after the following command runs like 5 or more iterations simultaneously, the dirty pages to be written gets to over 25000:
mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql ~var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
On Oct 26, 2015, at 8:36 AM, Johnson, Edward A. <eajohnson@prf.org [mailto:eajohnson@prf.org]> wrote:
Since upgrading PHP to 5.4, Observium has loaded very slowly at times. Most of the time just a refresh of the homepage will take .150 seconds but occasionally it can take up to a minute to load. Running iotop shows the Flush process running taking up 99.99% IO which directly correlates to the slow page loads. Can anyone shed some light on a way to resolve this? _______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium [http://postman.memetic.org/cgi-bin/mailman/listinfo/observium]
_______________________________________________ observium mailing list observium@observium.org [mailto:observium@observium.org] http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Yeah, there were 25 pollers (good lord) that triggered the mysqld jobs which in turn caused a logjam in the pages waiting to be written to disk. Lowering the ratios on the dirty pages and the poller number back down to 5 cleared it up. Lesson learned, increasing the RAM can sometimes negatively affect other ares.
On Oct 26, 2015, at 1:18 PM, Adam Armstrong <adama@memetic.orgmailto:adama@memetic.org> wrote:
That sounds... complicated.
adam.
On 26/10/2015 17:14:56, Johnson, Edward A. <eajohnson@prf.orgmailto:eajohnson@prf.org> wrote:
Turned out to be a balance issue between the number of pollers, the vm.dirty_background_ratio, and the vm.dirty_ratio.
On Oct 26, 2015, at 11:17 AM, Johnson, Edward A. <eajohnson@prf.orgmailto:eajohnson@prf.org> wrote:
I’ve also noticed that in iotop after the following command runs like 5 or more iterations simultaneously, the dirty pages to be written gets to over 25000:
mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql ~var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
On Oct 26, 2015, at 8:36 AM, Johnson, Edward A. <eajohnson@prf.orgmailto:eajohnson@prf.org> wrote:
Since upgrading PHP to 5.4, Observium has loaded very slowly at times. Most of the time just a refresh of the homepage will take .150 seconds but occasionally it can take up to a minute to load. Running iotop shows the Flush process running taking up 99.99% IO which directly correlates to the slow page loads. Can anyone shed some light on a way to resolve this? _______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Johnson, Edward A. : Turned out to be a balance issue between the number of pollers, the vm.dirty_background_ratio, and the vm.dirty_ratio.
If I recollect, many distro come "teaked" for end-users, so with both values too high with plenty of RAM servers.
Usually on servers that aren't file server, I set :
vm.dirty_background_bytes=67108864 vm.dirty_ratio=10
Edward, can you specify your sysctl values, kernel, distro that are not so good for that server ?
There are many different configs, I think it is useful to report the "guts" of the problem.
best regards, Sala
Linux version 2.6.32-573.7.1.el6.x86_64 (mockbuild@x86-031.build.eng.bos.redhat.commailto:mockbuild@x86-031.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP
vm.dirty_background_ratio = 10 vm.dirty_background_bytes = 0 vm.dirty_ratio = 20 vm.dirty_bytes = 0 vm.dirty_writeback_centisecs = 500 vm.dirty_expire_centisecs = 3000
On Oct 26, 2015, at 1:31 PM, Massimo.Sala@asl.bergamo.itmailto:Massimo.Sala@asl.bergamo.it wrote:
Johnson, Edward A. : Turned out to be a balance issue between the number of pollers, the vm.dirty_background_ratio, and the vm.dirty_ratio.
If I recollect, many distro come "teaked" for end-users, so with both values too high with plenty of RAM servers.
Usually on servers that aren't file server, I set :
vm.dirty_background_bytes=67108864 vm.dirty_ratio=10
Edward, can you specify your sysctl values, kernel, distro that are not so good for that server ?
There are many different configs, I think it is useful to report the "guts" of the problem.
best regards, Sala
_______________________________________________ observium mailing list observium@observium.orgmailto:observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Edward : vm.dirty_background_ratio = 10 vm.dirty_ratio = 20
Edward, your settings are pretty high, if you have more than 1 Gb of RAM.
They came from ancient times, and aren't tuned for today servers' big memories.
Usually on servers that aren't file server, I set : vm.dirty_background_bytes = 67108864 vm.dirty_ratio = 10
The background must be less than ( 5 % RAM ) and max 64 Mb, I prefer to set it not as percent.
Please let me know the results, by private Email, if you will try these settings.
Massimo
participants (3)
-
Adam Armstrong -
Johnson, Edward A. -
Massimo.Sala＠asl.bergamo.it