We upgraded from kernel 3.5 to 3.12.3 to update some of our benchmark numbers and hit some major performance regressions, mainly because the kernel is throttling processes that use too much CPU. This is definitely a kernel bug, as the throttle mechanism belongs to the realtime scheduler and none of the processes being affected had realtime priority when the throttle kicked in.

I've posted a query to the linux kernel mailing list but haven't gotten any satisfactory answers yet. The same throttling behavior also occurs with 3.11.10, but there are no corresponding messages in the kernel log.

The email thread is here http://lkml.indiana.edu/hypermail/linux/kernel/1312.1/02313.html

A patch that may be related is described here http://lkml.org/lkml/2013/5/29/640 which explains part of the observed behavior, but not all of it (and indeed may be a red herring, unless it has some interaction with the realtime scheduler).

There appear to be other serious networking related regressions in 3.12 as well. http://lkml.indiana.edu/hypermail/linux/kernel/1312.1/02588.html

I recommend staying on 3.10 for production servers until this is sorted out.