Hi Frank,
On 01/07/16 14:15, Frank Swasey wrote:
Today at 7:15am, Mark Cairney wrote:
Another gotcha is w.r.t. rate limiting as both journald and rsyslog implement this independently of one another. Disabling this completely can make the bottleneck mentioned above even more apparent! Setting your OpenLDAP logging level appropriately can mitigate this (I log at Stats+ Sync)
I am also running with
loglevel stats sync
and my servers are busy enough that journald misses a lot of the messages even with ryslogd's rate limiting turned off. Since Red Hat's advice is to *not* remove the rate limiting in journald, I've not found a sweet spot yet where the sync logging does not get limited when the empty consumer is drinking from the fire hose. ... any experience you can help me with would be greatly appreciated.
My present /etc/systemd/journald.conf contains the following- these are the defaults from our config management system and I haven't found a need to adjust them yet (although they do look a bit looser than the RH defaults):
[Journal] Storage=persistent SplitMode=login RateLimitInterval=10s RateLimitBurst=3000
My corresponding rsyslog settings are: # Standard preamble
$ModLoad imuxsock $ModLoad imjournal $ModLoad immark
# Tuning $OmitLocalLogging on $IMJournalStateFile imjournal.state
$SystemLogRateLimitInterval 0
# Discard audit log entries user.info ~
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Configure imjournal rate limiting $imjournalRatelimitInterval 60 $imjournalRatelimitBurst 100000
# CAuth logs local4.* -/usr/local/authz/var/log/openldap.log
For day-to-day operations I'm not noticing any limiting kicking in. During operations that generate a lot of logs e.g. re-syncing a replica, SLAMD benchmarking logs do indeed get lost but this is preferable to the alternative where the logger becomes the bottleneck and slapd is essentially sitting there waiting for logs to be written before continuing.
RE: bringing up a replica I'd avoid using syncrepl from scratch- I find building the slave from a dump of the master is far quicker and more reliable. I tend to do (as a brief/trivial example): (master) slapcat -n 1 -o ldif-wrap=no -l /tmp/master
scp it across to the slave (slave) rm -f /var/openldap-data/database/*.mdb rm -f /var/openldap-data/accesslog/*.mdb slapadd -q -w -n1 -l /tmp/master.ldif
Importing this with slapadd takes me about 5 mins for a 1.5G ldif file using mdb- it took about 15-20 mins when I was using bdb. With any luck the replica will then pick up any changes that have been made to the master since the dump when you start slapd. The usual disclaimer that this is just what's working for me and not a definitive "this is how it should be done"