Bill MacAllister wrote:

--On Tuesday, June 25, 2013 03:10:17 PM -0700 Howard Chu hyc@symas.com wrote:

...

Probably bad default FS settings, and changed from your previous OS revision.

Also, you should watch vmstat while it runs to get a better idea of how much time the system is spending in I/O wait.

I have just re-mkfs'ed the new, slow system to make it look like the old, fast system. Just to make sure nothing else changed I have started a load on the older system. Things look fine.

I meant mount options, mkfs should have very little impact.

ext3 journaling is pretty awful. For ext4 you probably want data=writeback mode, and you probably should compare the commit= value, which defaults to 5 seconds and also barrier; I believe the barrier default changed between kernel revisions.

Now, comparing vmstat output, the new system is clearly badness incarnate.

Fast

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 6358656 303468 9301620 0 0 0 0 45 45 0 0 100 0 0 0 0 6358780 303468 9301620 0 0 0 0 47 41 0 0 100 0 0 0 0 6358780 303468 9301620 0 0 0 0 47 41 0 0 100 0 0 0 0 6358780 303468 9301620 0 0 0 0 93 43 0 1 99 0 0 0 0 6358532 303468 9301620 0 0 0 0 141 71 0 1 99 0 1 0 0 6358488 303468 9301620 0 0 0 14 116 48 0 1 99 0

Slow

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 4 0 13318088 36128 2759600 0 0 0 2134 379 83 0 0 88 12 0 4 0 13318308 36128 2759600 0 0 0 1044 277 70 0 0 88 12 0 4 0 13318508 36132 2759600 0 0 0 765 267 69 0 0 88 12 0 2 0 13318240 36152 2759604 0 0 0 818 593 104 0 0 88 12 0 2 0 13318332 36168 2759604 0 0 0 2611 1489 138 0 0 89 11

Lots of waiting, lots of blocking. What's the deal with all that free memory on the slow system?

I will interate on mkfs for a bit, but I thought I would send this off incase something jumps out.

Bill