--On Thursday, June 30, 2016 12:27 PM -0400 Frank Swasey Frank.Swasey@uvm.edu wrote:

Folks,

I am working on moving my OpenLDAP 2.4.44 environment from RHEL6 to RHEL7. I want to double check an assumption I have made before I open a report with Red Hat about the quality of their product.

I am assuming that with "loglevel sync" and an empty consumer, that there should be one sincrepl_entry line that has "inserted UUID" in the syslog output for every entry created. Is that a valid assumption?

You're assuming RHEL7 logs to syslog by default. My experience so far is that it may or may not. I'm not sure why sometimes it does and sometimes it doesn't, and instead logs to journald.

--Quanah

--

Quanah Gibson-Mount Platform Architect Manager, Systems Team Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc

--On Thursday, June 30, 2016 1:10 PM -0400 Frank Swasey Frank.Swasey@uvm.edu wrote:

Today at 11:35am, Quanah Gibson-Mount wrote:

...

--On Thursday, June 30, 2016 12:27 PM -0400 Frank Swasey Frank.Swasey@uvm.edu wrote:

...

Folks,

I am working on moving my OpenLDAP 2.4.44 environment from RHEL6 to RHEL7. I want to double check an assumption I have made before I open a report with Red Hat about the quality of their product.

I am assuming that with "loglevel sync" and an empty consumer, that there should be one sincrepl_entry line that has "inserted UUID" in the syslog output for every entry created. Is that a valid assumption?

You're assuming RHEL7 logs to syslog by default. My experience so far is that it may or may not. I'm not sure why sometimes it does and sometimes it doesn't, and instead logs to journald.

I'm not assuming that completely. I have configured rsyslogd to send LOCAL6 to /var/log/slapd.log and am passing "-l LOCAL6" to slapd in the startup. I am assuming the system is actually going to do what I told it to (and that I've told it correctly). I'm asking the question to see if my assumption is faulty or not.

Right, we always configure things to use syslog. I'm simply telling you, in my experience, regardless of whether or not we had things configured to use syslog, RHEL7 *may or may not* honor that. There's some article I found before that details how to fix it so rsyslog works again under RHEL7 when you hit this case, that I'm trying to track down and send to you.

Again, I have no clue why RHEL7 does this. Literally I can install a series of VMs on RHEL7. Some correctly log out to rsyslog, some don't, without tweaking them. I.e., some default to only logging to journald and have to get reset to use rsyslog.

--Quanah

--

Quanah Gibson-Mount Platform Architect Manager, Systems Team Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc

Today at 12:28pm, Quanah Gibson-Mount wrote:

--On Thursday, June 30, 2016 10:13 AM -0700 Quanah Gibson-Mount quanah@zimbra.com wrote:

...

Again, I have no clue why RHEL7 does this. Literally I can install a series of VMs on RHEL7. Some correctly log out to rsyslog, some don't, without tweaking them. I.e., some default to only logging to journald and have to get reset to use rsyslog.

Here we go: https://bugzilla.zimbra.com/show_bug.cgi?id=104534

The first comment details what was done to fix the problem.

On a side note, we've been moving customers off of RHEL7 back to RHEL6, as we've simply found it too unstable for production use.

Quanah,

As always - thanks for helping me see the light!

Hi Toby/Quanah,

I'm currently using a RHEL7 derivative (SL7) on our production OpenLDAP servers i.e. Central Authorisation Service.

We use rsyslog partly out of familiarity but also because it allows sending logs to a central rsyslog server.

The way this setup appears to work with a system using systemd + rsyslog is that all the things are logged via systemd's journalctl.

The downside of this approach is that you are effectively running 2 logging daemons in parallel. As logging seems to be a potential bottleneck for OpenLDAP anyway it potentially exacerbates this even further.

Another gotcha is w.r.t. rate limiting as both journald and rsyslog implement this independently of one another. Disabling this completely can make the bottleneck mentioned above even more apparent! Setting your OpenLDAP logging level appropriately can mitigate this (I log at Stats+ Sync)

From my experience I'd say RHEL7 is a stable system to run OpenLDAP on. If you have a heavily-loaded system or don't need centralised logging though I'd try and get away with journalctl on it's own and only introduce rsyslog logging if you need it.

Kind regards,

Mark

On 01/07/16 10:21, Toby Blake wrote:

On Thu, 30 Jun 2016, Quanah Gibson-Mount wrote:

...

On a side note, we've been moving customers off of RHEL7 back to RHEL6, as we've simply found it too unstable for production use.

Hi Quanah,

This is concerning - can you provide a little more detail on this?

Cheers Toby Blake School of Informatics University of Edinburgh

Hi Frank,

On 01/07/16 14:15, Frank Swasey wrote:

Today at 7:15am, Mark Cairney wrote:

...

Another gotcha is w.r.t. rate limiting as both journald and rsyslog implement this independently of one another. Disabling this completely can make the bottleneck mentioned above even more apparent! Setting your OpenLDAP logging level appropriately can mitigate this (I log at Stats+ Sync)

I am also running with

loglevel stats sync

and my servers are busy enough that journald misses a lot of the messages even with ryslogd's rate limiting turned off. Since Red Hat's advice is to *not* remove the rate limiting in journald, I've not found a sweet spot yet where the sync logging does not get limited when the empty consumer is drinking from the fire hose. ... any experience you can help me with would be greatly appreciated.

My present /etc/systemd/journald.conf contains the following- these are the defaults from our config management system and I haven't found a need to adjust them yet (although they do look a bit looser than the RH defaults):

[Journal] Storage=persistent SplitMode=login RateLimitInterval=10s RateLimitBurst=3000

My corresponding rsyslog settings are: # Standard preamble

$ModLoad imuxsock $ModLoad imjournal $ModLoad immark

# Tuning $OmitLocalLogging on $IMJournalStateFile imjournal.state

$SystemLogRateLimitInterval 0

# Discard audit log entries user.info ~

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Configure imjournal rate limiting $imjournalRatelimitInterval 60 $imjournalRatelimitBurst 100000

# CAuth logs local4.* -/usr/local/authz/var/log/openldap.log

For day-to-day operations I'm not noticing any limiting kicking in. During operations that generate a lot of logs e.g. re-syncing a replica, SLAMD benchmarking logs do indeed get lost but this is preferable to the alternative where the logger becomes the bottleneck and slapd is essentially sitting there waiting for logs to be written before continuing.

RE: bringing up a replica I'd avoid using syncrepl from scratch- I find building the slave from a dump of the master is far quicker and more reliable. I tend to do (as a brief/trivial example): (master) slapcat -n 1 -o ldif-wrap=no -l /tmp/master

scp it across to the slave (slave) rm -f /var/openldap-data/database/*.mdb rm -f /var/openldap-data/accesslog/*.mdb slapadd -q -w -n1 -l /tmp/master.ldif

Importing this with slapadd takes me about 5 mins for a 1.5G ldif file using mdb- it took about 15-20 mins when I was using bdb. With any luck the replica will then pick up any changes that have been made to the master since the dump when you start slapd. The usual disclaimer that this is just what's working for me and not a definitive "this is how it should be done"