On 2015-04-23 5:56 PM, Howard Chu wrote:
Geoff Swan wrote:
On 2015-04-23 4:43 PM, Howard Chu wrote:
Geoff Swan wrote:
On 2015-04-23 4:07 PM, Ulrich Windl wrote:
>> Geoff Swan gswan3@bigpond.net.au schrieb am 22.04.2015 um 23:18 >> in Nachricht
5538100A.3060301@bigpond.net.au:
[...]
Free stats look fine to me. No swap is being used, or has been used yet on this system.
total used free shared bufferscached Mem: 132005400 21928192 110077208 10612 363064 19230072 -/+ buffers/cache: 2335056 129670344 Swap: 8388604 0 8388604
The effect I am seeing is that despite a reasonably fast disc system, the kernel writing of dirty pages is painfully slow.
So disk I/O stats is also available via sar. May we see them? Finally there's blktrace where you can follow the timing and positions of each individual block being written, but that's not quite easy to run and analyze (unless I missed the easy way).
I suspect scattered writes that bring your I/O rate down.
Regards, Ulrich
sysstat (and sar) is not installed on this system, so I can't give you that output.
As I said in my previous email - use iostat and/or vmstat to monitor paging activity and system wait time. You can also install atop and get all of the relevant stats on one screen.
I find it indispensible these days.
BTW, just using nmon whilst running a test script that writes 20K small objects shows a write speed of around 3-5 MB/s to the disc (without dbnosync, so as to see the immediate effects). This is on a SAS disc with a 140MB/s capability, checked using regular tools such a dd (writing 1000 files of 384K each). I'm failing to understand why the write operations from the kernel page cache is so drastically slower. With dbnosync enabled, lmdb writes to the pages fast (as expected), however the pdflush that follows writes at the same slow speed, causing delays for further processes.
In normal (safe) operation, every transaction commit performs 2 fsyncs. Your 140MB/s throughput spec isn't relevant here, your disk's IOPS rate is what matters. You can use NOMETASYNC to do only 1 fsync per commit.
Put atop on the test machine. Nice utiltity. Ran the test script again and got these results around the memory and disk lines, which appear to confirm the low IOPS. Any clues on improving this? The dd test on the disc was just to confirm that the C600 SAS controller/driver and disc could achieve the throughput.
MEM | tot 31.4G | free 30.4G | cache 326.5M | buff 167.2M | slab 85.2M | shmem 10.0M | vmbal 0.0M | hptot 0.0M | hpuse 0.0M | SWP | tot 16.0G | free 16.0G | | | | | | vmcom 5.4G | vmlim 31.7G | DSK | sda | busy 75% | read 0 | write 3400 | KiB/r 0 | KiB/w 3 | MBr/s 0.00 | MBw/s 1.28 | avio 2.20 ms |