11:15 a.m.
I've got my two-node setup up and running, MMR w/ delta-syncrepl.
Node 1 is up and running with ~340k entries in the main DIT. It
consumes about 4G of VM on a Redhat AS6 box, and the data dir is around
2.3G on-disk (including __db.* and the one log.* file...), and another
100M in cn=accesslog.
I'm bringing up node 2 after nuking the data dir, and letting it
syncrepl from nothing. I was tracking a SIGBUS error, but that turned
out to be olcLogLevel=Stats+Sync generating a huge amount of logging and
filling up the disk. Fixed that, moved on.
Now, it's still crashing... but that's because slapd is bloating up
hugely and causing the machine to run out of VM and kill the process.
I've added about 8G of temporary swap, and it's still going. slapd on
node2 is 5.5G resident, and nearly 15G in size total now. On-disk the
hdb is only around 1.6G, so it still has a long ways to go.
Can I assume this is not the way it should be?
vmstat shows a lot of swap-out but almost no swap-in, so it's not
thrashing. pmap shows a large number of 64M "anon" segments. 140 of
those at one point, and 157 when I checked just now.
It looks a lot like a memory leak, though I can't tell offhand whether
the problem is in OpenLDAP (2.4.31) or in BerkeleyDB (5.3.15). When it
finishes, I'm planning on turning off delta-syncrepl and pave/rebuild
again, and see if it behaves the same. I could also give mdb a shot,
but since this is a mirror I'd have to rebuild both sides.
Any suggestions as to where I could start looking for the source of the
problem? Obviously I'm not planning on rebuilding this node on a
regular basis (and certainly not all via syncrepl) but I'm concerned
that over an extended period of time it'll leak memory even during
normal use.
I'm including my DB_CONFIG, just in case. I can supply more of the
config as necessary.
DB_CONFIG:
set_cachesize 0 536870912 0
set_lg_regionmax 10485760
set_lg_max 104857600
set_lg_bsize 26214400
set_lk_max_locks 4096
set_lk_max_objects 4096
set_flags DB_LOG_AUTOREMOVE
(If anything looks particularly stupid in here, even unrelated to the
leak, I'd love the advice...)
11:36 a.m.
--On Friday, May 18, 2012 3:15 PM -0300 Brandon Hume <hume-ol(a)bofh.ca>
wrote:
hdb is only around 1.6G, so it still has a long ways to go.
I'm including my DB_CONFIG, just in case. I can supply more of
the
config as necessary.
DB_CONFIG:
set_cachesize 0 536870912 0
Are you using shm keys with BDB?
I don't know anyone who is using BDB 5.3 at this point in time. I would
back down to something less recent and see if you see the same issue. I've
used BDB 4.7 for quite some time w/o issue.
Also, your BDB cachesize seems undersized, given the size of your DB. It
should be no less than dn2id.bdb + id2entry.bdb + 10% for growth.
Personally, I prefer to keep my BDB cachesize at <size of DB>, or in your
case, I'd set it to 2 GB, i.e.:
set_cachesize 2 0 0
You certainly could try switching just this one side to mdb, and seeing how
it does, and then switching the other side over if you like how it is
performing compared to BDB. Both nodes are not required to run the same DB
backend.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
9:36 a.m.
On 05/18/12 03:36 PM, Quanah Gibson-Mount wrote:
Are you using shm keys with BDB?
Not at this time.
I don't know anyone who is using BDB 5.3 at this point in time.
I
would back down to something less recent and see if you see the same
issue. I've used BDB 4.7 for quite some time w/o issue.
Well, with 4.8.30, the problem is still evident. Overall, using
delta-syncrepl seems to make a large difference. With my previous test
(still with bdb 5.3) and delta-syncrepl, the slapd process grew to 16G
before dying. With delta off (removing the 'syncdata=accesslog
logbase="cn=accesslog"
logfilter="(&(objectClass=auditWriteObject)(reqResult=0))"' portion of
the olcSyncrepl line) the node started and finished loading the entire
DB and grew to about 5.7G in total.
I don't have the bdb 4.7 tarball handy... but I'll being giving mdb a
shot next and see if it behaves the same.
Also, your BDB cachesize seems undersized, given the size of your DB.
It should be no less than dn2id.bdb + id2entry.bdb + 10% for growth.
Personally, I prefer to keep my BDB cachesize at <size of DB>, or in
your case, I'd set it to 2 GB, i.e.:
Thanks, if I stick with hdb I'll make that tweak. Those values are
pretty old, I've been shy about changing too much while changing
everything else. :)
9:59 a.m.
Brandon Hume wrote:
On 05/18/12 03:36 PM, Quanah Gibson-Mount wrote:
> Are you using shm keys with BDB?
Not at this time.
> I don't know anyone who is using BDB 5.3 at this point in time. I
> would back down to something less recent and see if you see the same
> issue. I've used BDB 4.7 for quite some time w/o issue.
Well, with 4.8.30, the problem is still evident. Overall, using
delta-syncrepl seems to make a large difference. With my previous test
(still with bdb 5.3) and delta-syncrepl, the slapd process grew to 16G
before dying. With delta off (removing the 'syncdata=accesslog
logbase="cn=accesslog"
logfilter="(&(objectClass=auditWriteObject)(reqResult=0))"' portion of
the olcSyncrepl line) the node started and finished loading the entire
DB and grew to about 5.7G in total.
The fact that switching off delta changes the behavior would point pretty
squarely at a bug in the delta-sync consumer code, not anything in the
underlying backend. If you can produce a small test case to demonstrate the
problem, it would be a good idea to post this to the ITS.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
2:42 p.m.
----- Howard Chu <hyc(a)symas.com> wrote:
Brandon Hume wrote:
> On 05/18/12 03:36 PM, Quanah Gibson-Mount wrote:
>> Are you using shm keys with BDB?
>
> Not at this time.
>
>> I don't know anyone who is using BDB 5.3 at this point in time. I
>> would back down to something less recent and see if you see the same
>> issue. I've used BDB 4.7 for quite some time w/o issue.
>
> Well, with 4.8.30, the problem is still evident. Overall, using
> delta-syncrepl seems to make a large difference. With my previous test
> (still with bdb 5.3) and delta-syncrepl, the slapd process grew to 16G
> before dying. With delta off (removing the 'syncdata=accesslog
> logbase="cn=accesslog"
> logfilter="(&(objectClass=auditWriteObject)(reqResult=0))"'
portion of
> the olcSyncrepl line) the node started and finished loading the entire
> DB and grew to about 5.7G in total.
The fact that switching off delta changes the behavior would point pretty
squarely at a bug in the delta-sync consumer code, not anything in the
underlying backend. If you can produce a small test case to demonstrate the
problem, it would be a good idea to post this to the ITS.
Or it just means that the accesslog is receiving a significant number of changes and that
the time configured for how often to delete the accesslog DB is too small. It will
certainly grow without bound up to the time deletion kicks in. For some of our larger
customers, I have delete run on all data older than 8 hours, every 2 hours, to keep the
accesslog DB size down. I.e., there is still not necessarily evidence of a memory leak.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
10:45 a.m.
On 05/23/12 06:42 PM, Quanah Gibson-Mount wrote:
Or it just means that the accesslog is receiving a significant number
of changes and that the time configured for how often to delete the
accesslog DB is too small. It will certainly grow
To the tune of 16+G of VM (only 4.7G resident) on a 512G accesslog DB
and 2G main DB, though? I've tried setting the logpurge to kill > 15min
entries every 30min, and it doesn't seem to be making a difference.
I'll try to see if I can generate a pile of entries not using our local
schemas and reproduce the problem.
I suppose I can also try switching accesslog over to mdb and see if it
still occurs.
8:42 a.m.
--On May 24, 2012 2:45:25 PM -0300 Brandon Hume <hume-ol(a)bofh.ca> wrote:
On 05/23/12 06:42 PM, Quanah Gibson-Mount wrote:
> Or it just means that the accesslog is receiving a significant number
> of changes and that the time configured for how often to delete the
> accesslog DB is too small. It will certainly grow
To the tune of 16+G of VM (only 4.7G resident) on a 512G accesslog DB and
2G main DB, though? I've tried setting the logpurge to kill > 15min
entries every 30min, and it doesn't seem to be making a difference.
Yeah, that sounds like a different issue then. ;)
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
2:43 p.m.
----- Howard Chu <hyc(a)symas.com> wrote:
Brandon Hume wrote:
> On 05/18/12 03:36 PM, Quanah Gibson-Mount wrote:
>> Are you using shm keys with BDB?
>
> Not at this time.
>
>> I don't know anyone who is using BDB 5.3 at this point in time. I
>> would back down to something less recent and see if you see the same
>> issue. I've used BDB 4.7 for quite some time w/o issue.
>
> Well, with 4.8.30, the problem is still evident. Overall, using
> delta-syncrepl seems to make a large difference. With my previous test
> (still with bdb 5.3) and delta-syncrepl, the slapd process grew to 16G
> before dying. With delta off (removing the 'syncdata=accesslog
> logbase="cn=accesslog"
> logfilter="(&(objectClass=auditWriteObject)(reqResult=0))"'
portion of
> the olcSyncrepl line) the node started and finished loading the entire
> DB and grew to about 5.7G in total.
The fact that switching off delta changes the behavior would point pretty
squarely at a bug in the delta-sync consumer code, not anything in the
underlying backend. If you can produce a small test case to demonstrate the
problem, it would be a good idea to post this to the ITS.
Or it just means that the accesslog is receiving a significant number of changes and that
the time configured for how often to delete the accesslog DB is too small. It will
certainly grow without bound up to the time deletion kicks in. For some of our larger
customers, I have delete run on all data older than 8 hours, every 2 hours, to keep the
accesslog DB size down. I.e., there is still not necessarily evidence of a memory leak.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
4213
days inactive
4220
days old
openldap-technical@openldap.org
7 comments
3 participants
participants (3)
-
Brandon Hume -
Howard Chu -
Quanah Gibson-Mount