-------- Original Message -------- Subject: Linux kernel performance regressions Date: Wed, 11 Dec 2013 04:33:06 -0800 From: Howard Chu hyc@symas.com To: OpenLDAP-devel@openldap.org OpenLDAP-devel@openldap.org
We upgraded from kernel 3.5 to 3.12.3 to update some of our benchmark numbers and hit some major performance regressions, mainly because the kernel is throttling processes that use too much CPU. This is definitely a kernel bug, as the throttle mechanism belongs to the realtime scheduler and none of the processes being affected had realtime priority when the throttle kicked in.
I've posted a query to the linux kernel mailing list but haven't gotten any satisfactory answers yet. The same throttling behavior also occurs with 3.11.10, but there are no corresponding messages in the kernel log.
The email thread is here http://lkml.indiana.edu/hypermail/linux/kernel/1312.1/02313.html
A patch that may be related is described here http://lkml.org/lkml/2013/5/29/640 which explains part of the observed behavior, but not all of it (and indeed may be a red herring, unless it has some interaction with the realtime scheduler).
There appear to be other serious networking related regressions in 3.12 as well. http://lkml.indiana.edu/hypermail/linux/kernel/1312.1/02588.html
I recommend staying on 3.10 for production servers until this is sorted out.
Is this mean, that we can't use all of CPU's power with 3.12 kernel?
On 12/11/13, Howard Chu hyc@symas.com wrote:
-------- Original Message -------- Subject: Linux kernel performance regressions Date: Wed, 11 Dec 2013 04:33:06 -0800 From: Howard Chu hyc@symas.com To: OpenLDAP-devel@openldap.org OpenLDAP-devel@openldap.org
We upgraded from kernel 3.5 to 3.12.3 to update some of our benchmark numbers and hit some major performance regressions, mainly because the kernel is throttling processes that use too much CPU. This is definitely a kernel bug, as the throttle mechanism belongs to the realtime scheduler and none of the processes being affected had realtime priority when the throttle kicked in.
I've posted a query to the linux kernel mailing list but haven't gotten any satisfactory answers yet. The same throttling behavior also occurs with 3.11.10, but there are no corresponding messages in the kernel log.
The email thread is here http://lkml.indiana.edu/hypermail/linux/kernel/1312.1/02313.html
A patch that may be related is described here http://lkml.org/lkml/2013/5/29/640 which explains part of the observed behavior, but not all of it (and indeed may be a red herring, unless it has some interaction with the realtime scheduler).
There appear to be other serious networking related regressions in 3.12 as well. http://lkml.indiana.edu/hypermail/linux/kernel/1312.1/02588.html
I recommend staying on 3.10 for production servers until this is sorted out.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Maybe related to this bug/feature:
http://ubuntuforums.org/showthread.php?t=2179879 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1188647
On 12/13/13, Aris Setyawan aris.sety@gmail.com wrote:
Is this mean, that we can't use all of CPU's power with 3.12 kernel?
On 12/11/13, Howard Chu hyc@symas.com wrote:
-------- Original Message -------- Subject: Linux kernel performance regressions Date: Wed, 11 Dec 2013 04:33:06 -0800 From: Howard Chu hyc@symas.com To: OpenLDAP-devel@openldap.org OpenLDAP-devel@openldap.org
We upgraded from kernel 3.5 to 3.12.3 to update some of our benchmark numbers and hit some major performance regressions, mainly because the kernel is throttling processes that use too much CPU. This is definitely a kernel bug, as the throttle mechanism belongs to the realtime scheduler and none of the processes being affected had realtime priority when the throttle kicked in.
I've posted a query to the linux kernel mailing list but haven't gotten any satisfactory answers yet. The same throttling behavior also occurs with 3.11.10, but there are no corresponding messages in the kernel log.
The email thread is here http://lkml.indiana.edu/hypermail/linux/kernel/1312.1/02313.html
A patch that may be related is described here http://lkml.org/lkml/2013/5/29/640 which explains part of the observed behavior, but not all of it (and indeed may be a red herring, unless it has some interaction with the realtime scheduler).
There appear to be other serious networking related regressions in 3.12 as well. http://lkml.indiana.edu/hypermail/linux/kernel/1312.1/02588.html
I recommend staying on 3.10 for production servers until this is sorted out.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Aris Setyawan wrote:
Is this mean, that we can't use all of CPU's power with 3.12 kernel?
It looks like the bug may have been fixed in 3.12.4 or 3.12.5. At least, I haven't seen it happen again after upgrading.
On 12/11/13, Howard Chu hyc@symas.com wrote:
-------- Original Message -------- Subject: Linux kernel performance regressions Date: Wed, 11 Dec 2013 04:33:06 -0800 From: Howard Chu hyc@symas.com To: OpenLDAP-devel@openldap.org OpenLDAP-devel@openldap.org
We upgraded from kernel 3.5 to 3.12.3 to update some of our benchmark numbers and hit some major performance regressions, mainly because the kernel is throttling processes that use too much CPU. This is definitely a kernel bug, as the throttle mechanism belongs to the realtime scheduler and none of the processes being affected had realtime priority when the throttle kicked in.
I've posted a query to the linux kernel mailing list but haven't gotten any satisfactory answers yet. The same throttling behavior also occurs with 3.11.10, but there are no corresponding messages in the kernel log.
The email thread is here http://lkml.indiana.edu/hypermail/linux/kernel/1312.1/02313.html
A patch that may be related is described here http://lkml.org/lkml/2013/5/29/640 which explains part of the observed behavior, but not all of it (and indeed may be a red herring, unless it has some interaction with the realtime scheduler).
There appear to be other serious networking related regressions in 3.12 as well. http://lkml.indiana.edu/hypermail/linux/kernel/1312.1/02588.html
I recommend staying on 3.10 for production servers until this is sorted out.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
openldap-technical@openldap.org
-
Aris Setyawan -
Howard Chu