GitLab

at 2020-11-17T17:15:40+00:00

lloadd ahoy

Backend configuration

Startup adjustment

Update connection init

Upstream connection setup

Set up connections in the worker threads

Connection write support

Operation parsing

This is a proxy now

Update maximum number or parameters for backend

Authenticate the upstream connection if configured

Rename backend mutex

Response handling, exploit optional bervals

Logging improvements

Option for response handling

Client authentication

Make features global

Proxyauthz support

Switch bindkey to use Backend instead of bindconf

Add connection number config

Maintain the configured amount of connections per backend

Configuration part for retry timeouts

Update backend progress tracking

Implement backend retry timeouts

Fix proxyauthz handling

There might be errors before we save the operation in c_ops

Report initial bind errors to client

Reject operations when binding

Save connection ids on operation for logging purposes

Pending operation tracking and limiting

Do not leak addrinfos

Do not leak BerElements

connection reference counting

Do not clear c_pendingber on short write

On a failed bind, stop the callback from firing again

Not a problem but causes a slew of calls to upstream_bind_cb that will
all fail in the same way.

Rework client_read_cb along the lines of upstream

Shutdown handling

Operation reference counting

Destroy the unbind operation when acted upon

Do not leak responses to abandoned ops

Retain a reference around for handle_responses

Reset c_*ber after freeing and check c_pendingber race

Do not bother to write to a dying connection

Stop the read callback on a dead connection.

The connection might be ready to read (close) but if we can't destroy it
yet, we don't want the callback to trigger all the time or process new
data.

Clean up events properly

Revert connection/operation mutex order.

There was still a race where the connection could be freed as the
operation was still being used.

Convert backend and upstream management to use CIRCLEQ.

This alone doesn't make the server do a round robin.

Round-robin for upstream connections

Round robin for backends.

Several threads calling backend_select might reset current_backend to a
different place, there are two options to deal with that:
- just let the last rotation win (the current approach)
- detect whether first == current_backend and only replace then

Not sure which one is more useful, going with the simpler.

Fix refcounting for all code paths

Fix error handling wrt. its callers

Initialise listeners after all workers have been

Tweak connection error logging.

Do not log when receiving the last bytes on a connection. Log failed
writes.

Lookup operations by saved connid.

We reset the connection pointer on a destruction attempt, avoid the
spurious asserts.

Free all pending operations on shutdown

Keep a list of active clients for shutdown purposes.

Potentially for timeout detection purposes in the future.

Improve logging

Another attempt at operation/connection destroy interaction.

Only enable verifycredentials if libldap does

lloadd documentation

First test for load balancer

Second test

Rename listener-threads to reflect the option

Split client/upstream PDU size limits

Documentation updates

Be consistent with bind responses on no upstream

Handle upstream connection shutdown properly

Forward controls correctly in the face of proxyauth

Borrow liblber code to get abandon processing to work

Split bind configuration from backends

Rework proxyauthz handling

Document new bind configuration

Handle backend unsolicited response properly

Unify logging output

Remember and clear bind status correctly

Manage connection refcnt better

On connection shutdown, free op from the correct side

Simplify abandon processing

Fix issues in bind response handling

Make sure operation stays alive when we process it

Update tests to match latest configuration layout

Handle asynchronous connect properly

Adjust backend operation counting

Unblock the client when we can't find an upstream

If we can't find an upstream, we keep the client around, so it needs to
be unblocked.

Only one bind at a time

Realign logging levels.

Stats now logs all operations, stats2 additionally intermediate messages
(search entries).

Fix emfile handling

More logging improvements

Do not read on the last iteration.

When the pdu processing limit is hit, we still attempt to read another
PDU. If we succeed, the ber_get_next call in the read callback will
abort since a full PDU is already present.

Update docs and defaults

Avoid a deadlock with client

More logging improvements

Make ready only when still alive

TENTATIVE: communicate more for op destroy race

Close up the race

Do not back off until we get a failure

CLOSING is another potential state we could be in

Handle a client connection disconnected from op

Reset connection state on abandon

Stop processing if we freed the client

Shorten time operation_mutex is locked

Do not crash when closing both client and upstream

Do not unlock client unless we are destroying it

Switch from a global mutex

Set binding state after we have dropped all ops

Add a load test

Unify connection locking and I/O

lload_libevent_init can fail and wants to log

Refactor operation_send_reject

Rework upstream conn setup

Improve spec conformance, logging

Exop support

At the moment, no exops are processed internally, all are passed on
unchanged.

Rename macros and symbols to lloadd

Rename bind handlers

Move client related functions to client.c

Move bind handling to bind.c

Update write timeout to timeval

Respond to timeout events properly

Upstream TLS support

Client TLS support

Set up TLS context for backends

We should just be able to call backend_retry

Make timeouts common and redo connection read timeouts

Upstream queues ordered by c_connid

In preparation for operation timeout events.

Record operation activity times

Factor out abandon message preparation

Introduce operation timeout machinery

Protect currently impossible branch

Log timed out connections more clearly

request_abandon RFC4511 conformance

Update test suite

Clean ups and renames to coexist with slapd

Enable compilation of the load balancer as a module

To compile the balancer as a slapd module, pass --enable-balancer=mod to ./configure
Use --enable-balancer(=yes) to compile as standalone server.

Do not copy files from slapd, just link them

Initial extension to upstream selection

RFC4511 says Binds do not abandon, send a "reset" bind instead

Passing the client directly will allow clearing it from op

Refactor request parsing and sending.

We have to do most of out processing before we send the request over to
the upstream. If we don't, we might be too late and the response might
have arrived already.

SASL bind support

Introduces pinned operations. When SASL bind finishes, we might still
have to maintain a link between the client an an upstream for future
bind operations if we got a SASL Bind in Progress result code. We zero
out the msgids and remember a server-unique identifer on the client and
the relevant operation that lets us retrieve that link again. This
operation is reclaimed just like anything else when connections drop.

Hopefully, this should work for LDAP TXN and VC Exop support with SASL
later as well since it allows for many-to-many links to exist.

Handle EXTERNAL mechanism

Will only try to extract the TLS client certificate name if used during
the last handshake.

Do not compare c_auth when NULL

Clear c_auth on every bind request

For a new bind request, this is obvious, for SASL bind requests, we do
not know the final identity until we have finished handling it, make
sure it stays empty until then.

Manage c_sasl_bind_mech on upstream

Operation abandon related fixes

Fix pinned operation forwarding

Reset pin on simple bind

Add SASL documentation on SASL handling

Resolve authzid after a successful auth

Update tests to support lloadd as a module

Lload cn=monitor initial implementation

Use slapd's config.h

Convert the load balancer into a backend

Rework monitor startup

Takes care of dealing with monitor not present/not configured and fix a
monitor startup issue.

Module shutdown

More meaningful connection type reporting

Implement pause handlers

Fix a race on bind response processing.

During response processing, an upstream connection could be marked ready
after a different bind had already been allocated to it, thus allowing
two binds to be in progress on the same connection.

Cleanup sasl_bind_mech resets

Move op counting to operation_init

Fix operation counts

Trying to abandon an operation does not automatically make it completed,
it might have failed already but we're just racing to reach the client
to record that.

Introduce a new connection status - gentle shutdown

Split backend destruction from resetting it

expose task functions for invalidation

Introduce infra to handle config changes

Fix operation status tracking.

An operation is rejected iff it has to be dropped before we can find an
upstream for it (unless we handle it ourselves, that is). At that point
it is failed unless completed successfully.

This makes a difference for multi-stage binds which alternate between
'failed' (we are waiting on a server response) and 'completed' (server
did what we asked them to, waiting on client to continue).

Change log level for unsolicited response

Fix interaction of graceful connection closing and SASL bind support

Enable dynamic configuration

Reflect backend URI change in cn=monitor

Record the backend name

Deal with no backends being configured

Rework lloadd startup

Module shutdown support

Do not unlock upstream without referencing its dying ops

Implement runtime monitor (un)registration

Unregistration is a hack and we shoould either make the subsystems into
an entry (if monitor allows subentry generation) or implement subsystem
unregistration in back-monitor.

Protect operation when abandoning

Work around libevent base not waking up on shutdown

Keep a reference around for the bind task

Record pending DNS resolution to be able to cancel

Record connect task to allow canceling it

Make io-threads modification startup-only

Switch to ldap_parse_url_ext

This simplifies port parsing in the end. Also pass the url to
ldap_open_listener in anticipation of incremental listener config.

Introduce lload_open_new_listener

Reorganise listener support in cn=config and module startup

Document config behaviour

libevent 2.0 support

Fix a new backend not being operational if added via cn=config

Fix a race in managing b_dns_req

Clean up backend_retry and its callers.

Tighten checks on retry management

Refactor backend reset

Reuse the connection walking facility in timeout management.

Reuse connection_walk for client matters

Reuse connection walking in monitor for upstreams too

Initial implementation of cn=config testing script

Permit lloadd to share slapd TLS context

Simplify pause handling

Gets rid of a race where unpause+pause fired in a quick succession would
miss the event_base_loopbreak() call.

Fix backend starttls= setting being ignored

Do not enforce a valid ld in lutil_sasl_interact

Introduce SASL support for upstream connections

A failed cn=config ADD needs to be handled

Tests for monitoring support

Add TLS tests

SASL and proxyauthz tests

Fix races with backend_retry

Switch to LDAP_OTHER when handling a lost upstream.

LDAP_UNAVAILABLE signals "the server is shutting down or a subsystem
necessary to complete the operation is offline", so intelligent clients
tend to infer the connection will not be usable any more, which is not
the case here.

Epoch based memory reclamation

Similar to the algorithm presented in
https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-579.pdf

Not completely lock-free at the moment. Also the problems with epoch
based memory reclamation are still present - a thread actively observing
an epoch getting stuck will prevent LloadConnections and LloadOperations
being freed, potentially running out of memory.

Straighten up client pending op tracking

Implement client pending operation limits

Shorten to one epoch per PDU

A full read cycle can take a very long time if the limits are set too
high.

Make sure read event is not enabled while upstream_bind is scheduled

Fix race between unlinking a client and processing incoming data

Fix a SASL channel-binding leak

Thread error checking

Introduce mutex checks

Switched off unless thread debugging is on, but still useful for static
analysis.

Implement read throttling when writes backlog

Reject operations in such a case with LDAP_BUSY. If read_event feature
is on, just stop reading from the connection. However this could still
result in deadlocks in reasonable situations. Need to figure out better
ways to make it safe and still protect ourselves.

Introduce the notion of experimental features

Let the last thread dispose of pending references

If we're idle, there might be objects pending cleanup for the last two
epochs. Unless another thread comes in and checks into a new epoch or we
shut down, they will linger forever.

If one of the objects was a connection, it wouldn't get closed and be
stuck in CLOSE_WAIT state, potentially refusing another ligitimate
connection if its socket address were to match the one we're yet to
close.

Honour keepalive settings for upstreams

Do not accept requests with msgid == 0

It is used internally to identify pinned operations and should not be
encountered over the wire.

Flush cache before calling dispose()

This needs to be confirmed:
Location based atomics do not imply a full fence of the same level. So
to get the code in dispose() read the actual data, it seems we need to
initiate a fence.

Handle upstream rejecting a StartTLS exop

Add lloadd into our testing regime

Quanah Gibson-Mount pushed to branch master at openldap / OpenLDAP

Commits:

18 changed files: