11:14 p.m.
Howard Chu wrote:
I've tested again with libhoard 3.5.1, and it's actually
superior to
libumem for both speed and fragmentation. Here are some results against
glibc 2.3.3, libhoard, and libumem:
I've retested with current HEAD, adding Google's tcmalloc to the mix.
glibc hoard umem tcmalloc
size time size time size time size time
initial 731 741 741 744
single 1368 2:03.02 1809 1:26.67 1778 1:43.49 1453 0:58.77
single 1369 2:23.61 1827 0:25.68 1825 0:18.40 1512 0:21.74
single 1368 1:48.41 1828 0:16.52 1826 0:21.07 1529 0:22.97
single 1368 1:48.59 1829 0:16.59 1827 0:16.95 1529 0:17.07
single 1368 1:48.72 1829 0:16.53 1827 0:16.61 1529 0:17.01
single 1368 1:48.39 1829 0:20.70 1827 0:16.56 1529 0:16.99
single 1368 1:48.63 1830 0:16.56 1828 0:17.48 1529 0:17.29
single 1384 1:48.14 1829 0:16.64 1828 0:22.17 1529 0:16.94
four 1967 1:20.21 1918 0:35.96 1891 0:29.95 1606 0:42.48
four 2002 1:10.58 1919 0:30.07 1911 0:29.00 1622 0:42.38
four 2009 1:33.45 1920 0:42.06 1911 0:40.01 1628 0:40.68
four 1998 1:32.94 1920 0:35.62 1911 0:39.11 1634 0:30.41
four 1995 1:35.47 1920 0:34.20 1911 0:28.40 1634 0:40.80
four 1986 1:34.38 1920 0:28.92 1911 0:31.16 1634 0:40.42
four 1989 1:33.23 1920 0:31.48 1911 0:33.73 1634 0:33.97
four 1999 1:33.04 1920 0:33.47 1911 0:38.33 1634 0:40.91
slapd CPU 26:31.56 8:34.78 8:33.39 9:19.87
The initial size is the size of the slapd process right after
startup,
with the process totally idle. The id2entry DB is 1.3GB with about
360,000 entries, BDB cache at 512M, entry cache at 70,000 entries,
cachefree 7000. The subsequent statistics are the size of the slapd
process after running a single search filtering on an unindexed
attribute, basically spanning the entire DB. The entries range in size
from a few K to a couple megabytes.
The BDB and slapd cache configurations are the same, but the machine now
has 4GB of RAM so the entire DB fits in the filesystem buffer cache. As
such there is no disk I/O during these tests.
After running the single ldapsearch 4 times, I then ran the same search
again with 4 jobs in parallel. There should of course be some process
growth for the resources for 3 additional threads (about 60MB is about
right since this is an x86_64 system).
This time I ran the tests 8 times each. Basically I was looking for the
slapd process size to stabilize at a constant number...
The machine only had 2GB of RAM, and you can see that with glibc malloc
the kswapd got really busy in the 4-way run. The times might improve
slightly after I add more RAM to the box. But clearly glibc malloc is
fragmenting the heap like crazy. The current version of libhoard looks
like the winner here.
There was no swap activity (or any other disk activity) this time, so
the glibc numbers don't explode like they did before. But even so, the
other allocators are 5-6 times faster. The Google folks claim tcmalloc
is the fastest allocator they have ever seen. These tests show it is
fast, but it is not the fastest. It definitely is the most
space-efficient multi-threaded allocator though. It's hard to judge just
by the execution times of the ldapsearch commands, so I also recorded
the amount of CPU time the slapd process consumed by the end of each
test. That gives a clearer picture of the performance differences for
each allocator.
These numbers aren't directly comparable to the ones I posted on August
30, because I was using a hacked up RE23 there and used HEAD here.
Going thru these tests is a little disturbing; I would expect the
single-threaded results to be 100% repeatable but they're not quite. In
one run I saw glibc use up 2.1GB during the 4-way test, and never shrink
back down. I later re-ran the same test (because I hadn't recorded the
CPU usage the first time around) and got these numbers instead. The
other thing that's disturbing is just how bad glibc's malloc really is,
even in the single-threaded case which is supposed to be its ideal
situation.
The other thing to note is that for this run with libhoard, I doubled
its SUPERBLOCK_SIZE from 64K to 128K. That's probably why it's less
space-efficient here than umem, while it was more efficient in the
August results. I guess I'll have to recompile that with the original
size to see what difference that makes. For now I'd say my preference
would be tcmalloc...
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/
3:23 p.m.
New subject: better malloc strategies
Howard Chu wrote:
Howard Chu wrote:
> I've tested again with libhoard 3.5.1, and it's actually superior to
> libumem for both speed and fragmentation. Here are some results
> against glibc 2.3.3, libhoard, and libumem:
I've retested with current HEAD, adding Google's tcmalloc to the mix.
> The initial size is the size of the slapd process right after
startup,
> with the process totally idle. The id2entry DB is 1.3GB with about
> 360,000 entries, BDB cache at 512M, entry cache at 70,000 entries,
> cachefree 7000. The subsequent statistics are the size of the slapd
> process after running a single search filtering on an unindexed
> attribute, basically spanning the entire DB. The entries range in size
> from a few K to a couple megabytes.
To be exact, there are 380836 entries in the DB, varying in size from 1K
to 10MB.
The other thing to note is that for this run with libhoard, I doubled
its SUPERBLOCK_SIZE from 64K to 128K. That's probably why it's less
space-efficient here than umem, while it was more efficient in the
August results. I guess I'll have to recompile that with the original
size to see what difference that makes. For now I'd say my preference
would be tcmalloc...
I reran the tests one more time with hoard in its default configuration
(64K SUPERBLOCK_SIZE), in single-user mode, with no network connected.
The results are here http://highlandsun.com/hyc/#Malloc
Kinda interesting - with hoard this shows us processing 23000 entries
per second in the single-threaded case, vs only 3521 per second with
glibc malloc. In the multi-threaded case we get peaks up to 51136
entries per second vs 19645/second with glibc. It's understandable that
the multi-threaded case may get greater than 2x boost over the
single-threaded numbers on this dual-core box, since some portion of
entries will be in-cache and won't take any malloc hit at all. But still
the difference against glibc is staggering. All of the other throughput
benchmarks we've published so far have just used the default libc malloc
and we're already faster than all other LDAP products. Re-running those
with hoard or umem will be a real eye-opener.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/
4:25 p.m.
New subject: better malloc strategies
Howard Chu wrote:
Kinda interesting - with hoard this shows us processing 23000 entries
per second in the single-threaded case, vs only 3521 per second with
glibc malloc.
It is possible that you are seeing the syndrome that I wrote about here:
http://www.bozemanpass.com/info/linux/malloc/Linux_Heap_Contention.html
AFAIK the poorly behaving code is still present in today's glibc malloc.
(the problem affects UMich-derived LDAP servers because entries in
the cache comprise a significant proportion of the heap traffic, and they
tend to get allocated by a different thread than frees them when the
cache fills up). malloc exhibits lock contention when blocks are freed
by a different thread than allocated them (more exactly when the threads
hash to different arenas).
5 p.m.
New subject: better malloc strategies
David Boreham wrote:
Howard Chu wrote:
> Kinda interesting - with hoard this shows us processing 23000 entries
> per second in the single-threaded case, vs only 3521 per second with
> glibc malloc.
It is possible that you are seeing the syndrome that I wrote about here:
http://www.bozemanpass.com/info/linux/malloc/Linux_Heap_Contention.html
AFAIK the poorly behaving code is still present in today's glibc malloc.
(the problem affects UMich-derived LDAP servers because entries in
the cache comprise a significant proportion of the heap traffic, and they
tend to get allocated by a different thread than frees them when the
cache fills up). malloc exhibits lock contention when blocks are freed
by a different thread than allocated them (more exactly when the threads
hash to different arenas).
Yes, that's a factor I had considered. One of the approaches I had
tested was tagging each entry with the threadID that allocated it, and
deferring the frees until the owning thread comes back to dispose of
them. I wasn't really happy with this approach because it meant the
entry cache size would not be strictly regulated, though on a busy
server it's likely that all of the frees would happen in reasonably
short time.
Still, free() contention alone doesn't explain the huge performance gap
in the single-threaded case. However, judging from the fact that the
very first glibc search often runs faster than the second and subsequent
ones, I'd say that there are other significant costs in glibc free().
Most likely would be that glibc returns memory back to the OS too
frequently. Hoard and Umem also return memory back to the OS, but they
also keep more extensive free lists. Tcmalloc never returns memory back
to the OS.
What's also interesting is that for Hoard, Umem, and Tcmalloc, the
multi-threaded query times are consistently about 2x slower than the
single-threaded case. The 2x slowdown makes sense since it's only a
dual-core CPU and it's doing 4x as much work. This kinda says that the
cost of malloc is overshadowed by the overhead of thread scheduling.
But for glibc the multi-threaded times are actually faster than the
single-threaded case. Since glibc's costs are being partially hidden it
seems that the multiple threads are actually getting some benefit from
the entry cache here. Still the difference between glibc and the other
allocators is so dramatic that this little difference is just academic.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/
6:41 p.m.
New subject: better malloc strategies
Howard Chu wrote:
What's also interesting is that for Hoard, Umem, and Tcmalloc,
the
multi-threaded query times are consistently about 2x slower than the
single-threaded case. The 2x slowdown makes sense since it's only a
dual-core CPU and it's doing 4x as much work. This kinda says that the
cost of malloc is overshadowed by the overhead of thread scheduling.
Is it possible that the block stride in the addresses returned by
malloc() is affecting
cache performance in the glibc case ?
If they are too close I think it is possible to thrash cache lines
between cores.
7 p.m.
New subject: better malloc strategies
David Boreham wrote:
Howard Chu wrote:
> What's also interesting is that for Hoard, Umem, and Tcmalloc, the
> multi-threaded query times are consistently about 2x slower than the
> single-threaded case. The 2x slowdown makes sense since it's only a
> dual-core CPU and it's doing 4x as much work. This kinda says that the
> cost of malloc is overshadowed by the overhead of thread scheduling.
Is it possible that the block stride in the addresses returned by
malloc() is affecting
cache performance in the glibc case ?
If they are too close I think it is possible to thrash cache lines
between cores.
That's a very good question, and I don't have an answer yet. I've been
working on some threading extensions for cachegrind so I can investigate
that. Unfortunately its existing infrastructure doesn't lend itself well
to tracking multiple caches, so that's been slow going.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/
11:49 a.m.
New subject: better malloc strategies
David Boreham wrote:
Howard Chu wrote:
> What's also interesting is that for Hoard, Umem, and Tcmalloc, the
> multi-threaded query times are consistently about 2x slower than the
> single-threaded case. The 2x slowdown makes sense since it's only a
> dual-core CPU and it's doing 4x as much work. This kinda says that the
> cost of malloc is overshadowed by the overhead of thread scheduling.
Is it possible that the block stride in the addresses returned by
malloc() is affecting
cache performance in the glibc case ?
If they are too close I think it is possible to thrash cache lines
between cores.
I've been tinkering with oprofile and some of the performance counters
etc... I see that with the current entry_free() that returns an entry to
the head of the free list, the same structs get re-used over and over.
This is cache-friendly on a single-core machine but causes cache
contention on a multi-core machine (because a just-freed entry tends to
get reused in a different thread). Putting freed entries at the tail of
the list avoids the contention in this case, but it sort of makes things
equally bad for all the cores. (I.e., everyone has to go out to main
memory for the structures, nobody gets any benefit from the cache.) For
the moment I'm going to leave it with entry_free() returning entries to
the head of the list.
Our current Entry structure is 80 bytes on a 64-bit machine. (Only 32
bytes on a 32 bit machine.) That's definitely not doing us any favors; I
may try padding it up to 128 bytes to see how that affects things.
Unfortunately while it may be more CPU cache-friendly, it will
definitely cost us as far as how many entries we can keep cached in RAM.
Another possibility would be to interleave the prealloc list. (E.g., 5
stripes of stride 8 would keep everything on 128 byte boundaries.)
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/
8:13 a.m.
New subject: better malloc strategies
Happy New Year!
Howard Chu wrote:
> I've tested again with libhoard 3.5.1, and it's actually
superior to
> libumem for both speed and fragmentation. Here are some results
> against glibc 2.3.3, libhoard, and libumem and tcmalloc:
With the latest code in HEAD the difference in speed between Hoard, Umem, and
TCmalloc pretty much disappears. Compared to the results from November,
they're about 5-15% faster in the single-threaded case, and significantly
faster in the multi-threaded case. Looking at glibc's performance it's pretty
clear that our new Entry and Attribute slabs helped a fair amount, but
weren't a complete cure. At this point I think the only cure for glibc malloc
is not to use it...
November 22 figures:
glibc hoard umem tcmalloc
size time size time size time size time
start 732 740 734 744
single 1368 01:06.04 1802 00:24.52 1781 00:38.46 1454 00:23.86
single 1369 01:47.15 1804 00:17.12 1808 00:18.93 1531 00:17.10
single 1384 01:48.22 1805 00:16.66 1825 00:24.56 1548 00:16.58
single 1385 01:48.65 1805 00:16.61 1825 00:16.17 1548 00:16.61
single 1384 01:48.87 1805 00:16.50 1825 00:16.37 1548 00:16.74
single 1384 01:48.63 1806 00:16.50 1825 00:16.22 1548 00:16.78
single 1385 02:31.95 1806 00:16.50 1825 00:16.30 1548 00:16.67
single 1384 02:43.56 1806 00:16.61 1825 00:16.20 1548 00:16.68
four 2015 02:00.42 1878 00:46.60 1883 00:28.70 1599 00:34.24
four 2055 01:17.54 1879 00:47.06 1883 00:39.45 1599 00:41.09
four 2053 01:21.53 1879 00:40.91 1883 00:37.90 1599 00:41.45
four 2045 01:20.48 1879 00:30.58 1883 00:39.59 1599 00:56.45
four 2064 01:26.11 1879 00:30.77 1890 00:47.71 1599 00:40.74
four 2071 01:29.01 1879 00:40.78 1890 00:44.53 1610 00:40.87
four 2053 01:30.59 1879 00:38.44 1890 00:39.31 1610 00:34.12
four 2056 01:28.11 1879 00:29.79 1890 00:39.53 1610 00:53.65
CPU1 15:23.00 02:20.00 02:43.00 02:21.00
CPU final 26:50.43 08:13.99 09:20.86 09:09.05
> The start size is the size of the slapd process right after
startup,
> with the process totally idle. The id2entry DB is 1.3GB with about
> 360,000 entries, BDB cache at 512M, entry cache at 70,000 entries,
> cachefree 7000. The subsequent statistics are the size of the slapd
> process after running a single search filtering on an unindexed
> attribute, basically spanning the entire DB. The entries range in size
> from a few K to a couple megabytes.
The process sizes are in megabytes. There are 380836 entries in the DB. The
CPU1 line is the amount of CPU time the slapd process accumulated after the
last single-threaded test. The multi-threaded test consists of starting the
identical search 4 times in the background of a shell script, using "wait" to
wait for all of them to complete, and timing the script. The CPU final line
is the total amount of CPU time the slapd process used at the end of all tests.
Here are the numbers for HEAD as of today:
glibc hoard umem tcmalloc
size time size time size time size time
start 649 655 651 660
single 1305 01:02.21 1746 00:32.73 1726 00:21.93 1364 00:19.97
single 1575 00:13.53 1786 00:12.57 1753 00:13.74 1396 00:12.92
single 1748 00:14.23 1797 00:13.34 1757 00:14.54 1455 00:13.84
single 1744 00:14.06 1797 00:13.45 1777 00:14.44 1473 00:13.92
single 1533 01:48.20 1798 00:13.45 1777 00:14.15 1473 00:13.90
single 1532 01:27.63 1797 00:13.44 1790 00:14.14 1473 00:13.89
single 1531 01:29.70 1798 00:13.42 1790 00:14.10 1473 00:13.87
single 1749 00:14.45 1798 00:13.41 1790 00:14.11 1473 00:13.87
four 2202 00:33.63 1863 00:23.11 1843 00:23.49 1551 00:23.37
four 2202 00:38.63 1880 00:23.23 1859 00:22.59 1551 00:23.71
four 2202 00:39.24 1880 00:23.34 1859 00:22.77 1564 00:23.57
four 2196 00:38.72 1880 00:23.23 1859 00:22.71 1564 00:23.65
four 2196 00:39.41 1881 00:23.40 1859 00:22.67 1564 00:23.96
four 2196 00:38.82 1880 00:23.13 1859 00:22.79 1564 00:23.41
four 2196 00:39.02 1881 00:23.18 1859 00:22.83 1564 00:23.27
four 2196 00:38.90 1880 00:23.12 1859 00:22.82 1564 00:23.48
CPU1 06:44.07 01:53.00 02:01.00 01:56.34
CPU final 12:56.51 05:48.56 05:52.21 05:47.77
Looking at the glibc numbers really makes you wonder what it's doing, running
"OK" for a while, then chewing up CPU, then coming back. Seems to be a pretty
expensive garbage collection pass. As before, the system was otherwise idle
and there was no paging/swapping/disk activity during the tests. The overall
improvements probably come mainly from the new cache-replacement code, and
splitting/rearranging some cache locks. (Oprofile has turned out to be quite
handy for identifying problem areas... Though it seems that playing with the
cacheline alignment only netted a 0.5-1% effect, pretty forgettable.)
It's worth noting that we pay a pretty high overhead cost for using BDB locks
here, but that cost seems to amortize out as the number of threads increases.
Put another way, this dual-core machine is scaling as if it were a quad - it
takes about a 4x increase in job load to see a 2x increase in execution time.
E.g., 16 concurrent searches complete in about 41 seconds for Hoard. 32
complete in only 57 seconds. That's starting to approach the speed of the
back-hdb entry cache - when all the entries are in the entry cache (as
opposed to the BDB cache) a single search completes in about .7 seconds,
which is hitting around the 500,000 entries/second mark.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/
4:21 p.m.
New subject: better malloc strategies
Oops. These November figures are for back-bdb. The new numbers in the
previous post were from back-hdb, so not comparable. I have new numbers for
back-bdb here. Interestingly, all of them are faster than back-hdb, except
for tcmalloc, which got slower. Nothing's ever straightforward...
Howard Chu wrote:
November 22 figures:
glibc hoard umem tcmalloc
size time size time size time size time
start 732 740 734 744
single 1368 01:06.04 1802 00:24.52 1781 00:38.46 1454 00:23.86
single 1369 01:47.15 1804 00:17.12 1808 00:18.93 1531 00:17.10
single 1384 01:48.22 1805 00:16.66 1825 00:24.56 1548 00:16.58
single 1385 01:48.65 1805 00:16.61 1825 00:16.17 1548 00:16.61
single 1384 01:48.87 1805 00:16.50 1825 00:16.37 1548 00:16.74
single 1384 01:48.63 1806 00:16.50 1825 00:16.22 1548 00:16.78
single 1385 02:31.95 1806 00:16.50 1825 00:16.30 1548 00:16.67
single 1384 02:43.56 1806 00:16.61 1825 00:16.20 1548 00:16.68
four 2015 02:00.42 1878 00:46.60 1883 00:28.70 1599 00:34.24
four 2055 01:17.54 1879 00:47.06 1883 00:39.45 1599 00:41.09
four 2053 01:21.53 1879 00:40.91 1883 00:37.90 1599 00:41.45
four 2045 01:20.48 1879 00:30.58 1883 00:39.59 1599 00:56.45
four 2064 01:26.11 1879 00:30.77 1890 00:47.71 1599 00:40.74
four 2071 01:29.01 1879 00:40.78 1890 00:44.53 1610 00:40.87
four 2053 01:30.59 1879 00:38.44 1890 00:39.31 1610 00:34.12
four 2056 01:28.11 1879 00:29.79 1890 00:39.53 1610 00:53.65
CPU1 15:23.00 02:20.00 02:43.00 02:21.00
CPU final 26:50.43 08:13.99 09:20.86 09:09.05
back-bdb numbers for HEAD as of today:
glibc hoard umem tcmalloc
size time size time size time size time
start 655 655 651 660
single 1703 00:20.50 1703 00:20.50 1696 00:22.24 1350 00:21.54
single 1708 00:11.79 1708 00:11.79 1720 00:12.26 1364 00:12.76
single 1715 00:12.86 1715 00:12.86 1743 00:13.61 1440 00:14.01
single 1729 00:12.98 1729 00:12.98 1743 00:13.07 1445 00:14.11
single 1729 00:12.91 1729 00:12.91 1743 00:13.05 1445 00:14.09
single 1746 00:12.93 1746 00:12.93 1743 00:13.08 1445 00:14.13
single 1747 00:12.92 1747 00:12.92 1743 00:13.06 1458 00:14.11
single 1747 00:12.92 1747 00:12.92 1743 00:13.06 1458 00:14.10
four 1814 00:41.54 1820 00:22.01 1791 00:22.97 1519 00:24.05
four 1858 00:44.89 1852 00:22.14 1791 00:22.43 1541 00:24.15
four 2035 00:33.69 1852 00:22.11 1791 00:22.36 1548 00:24.24
four 2060 00:33.89 1853 00:22.02 1791 00:22.35 1548 00:23.68
four 2075 00:33.67 1853 00:22.01 1791 00:22.30 1548 00:24.68
four 2066 00:35.08 1853 00:22.11 1799 00:22.33 1548 00:23.78
four 2056 00:32.59 1853 00:21.88 1815 00:22.24 1548 00:23.87
four 2055 00:41.03 1853 00:22.21 1815 00:22.11 1548 00:23.75
CPU1 06:16.19 01:49.00 01:54.14 01:59.01
CPU final 12:04.24 05:32.77 05:40.43 05:51.68
Here are the numbers for HEAD as of today: (back-hdb)
glibc hoard umem tcmalloc
size time size time size time size time
start 649 655 651 660
single 1305 01:02.21 1746 00:32.73 1726 00:21.93 1364 00:19.97
single 1575 00:13.53 1786 00:12.57 1753 00:13.74 1396 00:12.92
single 1748 00:14.23 1797 00:13.34 1757 00:14.54 1455 00:13.84
single 1744 00:14.06 1797 00:13.45 1777 00:14.44 1473 00:13.92
single 1533 01:48.20 1798 00:13.45 1777 00:14.15 1473 00:13.90
single 1532 01:27.63 1797 00:13.44 1790 00:14.14 1473 00:13.89
single 1531 01:29.70 1798 00:13.42 1790 00:14.10 1473 00:13.87
single 1749 00:14.45 1798 00:13.41 1790 00:14.11 1473 00:13.87
four 2202 00:33.63 1863 00:23.11 1843 00:23.49 1551 00:23.37
four 2202 00:38.63 1880 00:23.23 1859 00:22.59 1551 00:23.71
four 2202 00:39.24 1880 00:23.34 1859 00:22.77 1564 00:23.57
four 2196 00:38.72 1880 00:23.23 1859 00:22.71 1564 00:23.65
four 2196 00:39.41 1881 00:23.40 1859 00:22.67 1564 00:23.96
four 2196 00:38.82 1880 00:23.13 1859 00:22.79 1564 00:23.41
four 2196 00:39.02 1881 00:23.18 1859 00:22.83 1564 00:23.27
four 2196 00:38.90 1880 00:23.12 1859 00:22.82 1564 00:23.48
CPU1 06:44.07 01:53.00 02:01.00 01:56.34
CPU final 12:56.51 05:48.56 05:52.21 05:47.77
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/
11:15 p.m.
New subject: better malloc strategies
And another oops. The single-thread numbers for glibc in this post were
mis-copied (note they're identical to hoard's numbers). The multi-thread
numbers and the CPU times are all correct.
The actual single-thread stats for glibc in that test run were:
start 649
single 1281 01:02.25
single 1399 00:11.85
single 1404 01:43.01
single 1737 00:13.22
single 1748 00:12.80
single 1736 01:45.55
single 1764 00:12.79
single 1772 00:13.14
Still no idea why there are such erratic jumps in its behavior.
Howard Chu wrote:
back-bdb numbers for HEAD as of today:
glibc hoard umem tcmalloc
size time size time size time size time
start 655 655 651 660
single 1703 00:20.50 1703 00:20.50 1696 00:22.24 1350 00:21.54
single 1708 00:11.79 1708 00:11.79 1720 00:12.26 1364 00:12.76
single 1715 00:12.86 1715 00:12.86 1743 00:13.61 1440 00:14.01
single 1729 00:12.98 1729 00:12.98 1743 00:13.07 1445 00:14.11
single 1729 00:12.91 1729 00:12.91 1743 00:13.05 1445 00:14.09
single 1746 00:12.93 1746 00:12.93 1743 00:13.08 1445 00:14.13
single 1747 00:12.92 1747 00:12.92 1743 00:13.06 1458 00:14.11
single 1747 00:12.92 1747 00:12.92 1743 00:13.06 1458 00:14.10
four 1814 00:41.54 1820 00:22.01 1791 00:22.97 1519 00:24.05
four 1858 00:44.89 1852 00:22.14 1791 00:22.43 1541 00:24.15
four 2035 00:33.69 1852 00:22.11 1791 00:22.36 1548 00:24.24
four 2060 00:33.89 1853 00:22.02 1791 00:22.35 1548 00:23.68
four 2075 00:33.67 1853 00:22.01 1791 00:22.30 1548 00:24.68
four 2066 00:35.08 1853 00:22.11 1799 00:22.33 1548 00:23.78
four 2056 00:32.59 1853 00:21.88 1815 00:22.24 1548 00:23.87
four 2055 00:41.03 1853 00:22.21 1815 00:22.11 1548 00:23.75
CPU1 06:16.19 01:49.00 01:54.14 01:59.01
CPU final 12:04.24 05:32.77 05:40.43 05:51.68
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/
12:52 p.m.
New subject: better malloc strategies
Howard Chu wrote:
And another oops. The single-thread numbers for glibc in this post
were
mis-copied (note they're identical to hoard's numbers). The multi-thread
numbers and the CPU times are all correct.
The actual single-thread stats for glibc in that test run were:
Still no idea why there are such erratic jumps in its behavior.
Found the explanation for this
http://sourceware.org/ml/libc-alpha/2006-03/msg00033.html
Running the glibc test with MALLOC_MMAP_THRESHOLD_ env var set to 8MB
drastically improved its performance. Aside from some high startup overhead,
it's very nearly the fastest in single-threaded, which is more in line with
what other folks have written about it. Still the slowest for multi-threaded
though.
glibc hoard umem tmalloc
size time size time size time size time
start 649 655 653 660
single 1278 01:01.95 1708 00:20.23 1693 00:22.33 1353 00:20.91
single 1278 00:26.69 1747 00:11.73 1731 00:12.37 1374 00:12.26
single 1389 00:11.33 1747 00:11.09 1736 00:11.75 1374 00:11.58
single 1412 00:11.12 1748 00:11.06 1736 00:11.71 1374 00:11.58
single 1412 00:11.09 1748 00:11.05 1736 00:11.62 1374 00:11.57
single 1408 00:11.12 1748 00:11.07 1736 00:11.65 1374 00:11.58
single 1412 00:11.11 1748 00:11.06 1736 00:11.54 1374 00:11.56
single 1402 00:11.11 1748 00:11.07 1736 00:11.51 1374 00:11.52
four 1504 00:27.48 1801 00:19.37 1786 00:19.44 1435 00:19.64
four 1502 00:24.24 1801 00:19.22 1799 00:20.90 1451 00:20.44
four 1501 00:23.67 1829 00:19.18 1807 00:19.41 1451 00:19.89
four 1497 00:23.47 1832 00:21.12 1821 00:19.84 1451 00:19.92
four 1498 00:23.68 1832 00:19.50 1821 00:21.27 1451 00:19.91
four 1498 00:23.71 1832 00:19.35 1821 00:19.63 1451 00:19.97
four 1512 00:23.71 1832 00:19.40 1821 00:19.36 1451 00:19.88
four 1512 00:24.53 1832 00:19.51 1821 00:19.59 1451 00:19.98
CPU1 02:36.32 01:38.54 01:45.14 01:42.62
CPU final 06:36.88 05:01.90 05:11.78 05:02.91
This is with current back-bdb (HEAD as of yesterday), slightly faster overall
than the January 1 code. It still looks like a tradeoff between hoard for
best speed bs tcmalloc for best memory efficiency.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/
5925
days inactive
5969
days old
10 comments
2 participants
participants (2)
-
David Boreham -
Howard Chu