We currently run our openLDAP service on our campus behind an F5 load balancer which preserves the IP address of the connecting client through to the backend servers, which we rely on for a small amount of IP address based authorization differentiating between on-campus and off-campus access.
However, management is strongly pushing us to migrate the service to the Amazon cloud, using Amazon's load balancer. Unfortunately, Amazon's load balancer only supports client NAT for directing connections to the back end servers, so they have no idea who the actual client is, it just appears to be the load balancer itself.
Amazon's solution for that is to support HAProxy's proxy protocol in their load balancer:
https://www.haproxy.com/blog/haproxy/proxy-protocol/
Basically, this is an in band signaling mechanism that inserts an additional header in the initial connection data containing the original client IP address/source port and destination IP address/source port, allowing the server to utilize that information for the connection rather than the actual details of the network connection from the proxy itself.
This requires support from the application running on the server, as it must remove and process that proxy header from the connection data before moving on with whatever data would normally be passed on the connection.
There are some fair number of services that support this proxy, including of course HAProxy itself, such as the apache web server and the postfix mail server.
openLDAP does not support the protocol, and I was unable to find any past discussion of it.
I was wondering if this feature would be something acceptable for inclusion to openLDAP, or if from an architectural perspective it would be considered undesirable.
In general, I believe applications listening on a specific port are either expecting the proxy protocol header, or not, I do not think it is dynamically determined. As such, from an implementation perspective, my initial thought is that it would be implemented in terms of configuration as an additional URL specified via the -h option, something like "ldapp://" or "ldap_p://", "ldapsp://" or "ldaps_p://" or whatever seems most desirable. A server might listen on the standard ports accepting only proxied connections, or it might listen for normal connections on the standard ports and for proxy connections on alternative ports.
When a connection is accepted on a port marked as requiring the proxy protocol, it would read and process the proxy header to populate the appropriate data structures regarding connection, and then move on as it normally would to deal with the connection.
If this feature is of interest, I will probably spend a little time poking at it and seeing how much trouble it will be to implement.
Thanks…
On 11/19/20 2:49 AM, Paul B. Henson wrote:
Amazon's solution for that is to support HAProxy's proxy protocol in their load balancer:
https://www.haproxy.com/blog/haproxy/proxy-protocol/
Basically, this is an in band signaling mechanism that inserts an additional header in the initial connection data containing the original client IP address/source port and destination IP address/source port,
AFAICS this only works with HTTP and SMTP.
openLDAP does not support the protocol, and I was unable to find any past discussion of it.
LDAP uses BER-encoded ASN.1, not ASCII.
The LDAP session tracking extended control [1] can be used to pass the client's IP address of a proxied connection to the LDAP server. Currently slapd only logs the content of this control.
But it would have to be implemented in the proxy, here the AWS load-balancer. *And* slapd's ACLs would have to be extended to evaluate this.
Would be a nice feature for lloadd [2].
[1] https://tools.ietf.org/html/draft-wahl-ldap-session-03
[2] https://bugs.openldap.org/show_bug.cgi?id=8747
Ciao, Michael.
On 11/19/20 9:52 AM, Michael Ströder wrote:
On 11/19/20 2:49 AM, Paul B. Henson wrote:
Amazon's solution for that is to support HAProxy's proxy protocol in their load balancer:
https://www.haproxy.com/blog/haproxy/proxy-protocol/
Basically, this is an in band signaling mechanism that inserts an additional header in the initial connection data containing the original client IP address/source port and destination IP address/source port,
AFAICS this only works with HTTP and SMTP.
Aaargh! I've missed the binary header part. So forget my former comments.
Ciao, Michael.
Paul B. Henson wrote:
We currently run our openLDAP service on our campus behind an F5 load balancer which preserves the IP address of the connecting client through to the backend servers, which we rely on for a small amount of IP address based authorization differentiating between on-campus and off-campus access.
However, management is strongly pushing us to migrate the service to the Amazon cloud, using Amazon's load balancer. Unfortunately, Amazon's load balancer only supports client NAT for directing connections to the back end servers, so they have no idea who the actual client is, it just appears to be the load balancer itself.
Amazon's solution for that is to support HAProxy's proxy protocol in their load balancer:
https://www.haproxy.com/blog/haproxy/proxy-protocol/
Basically, this is an in band signaling mechanism that inserts an additional header in the initial connection data containing the original client IP address/source port and destination IP address/source port, allowing the server to utilize that information for the connection rather than the actual details of the network connection from the proxy itself.
This requires support from the application running on the server, as it must remove and process that proxy header from the connection data before moving on with whatever data would normally be passed on the connection.
There are some fair number of services that support this proxy, including of course HAProxy itself, such as the apache web server and the postfix mail server.
openLDAP does not support the protocol, and I was unable to find any past discussion of it.
I was wondering if this feature would be something acceptable for inclusion to openLDAP, or if from an architectural perspective it would be considered undesirable.
In general, I believe applications listening on a specific port are either expecting the proxy protocol header, or not, I do not think it is dynamically determined. As such, from an implementation perspective, my initial thought is that it would be implemented in terms of configuration as an additional URL specified via the -h option, something like "ldapp://" or "ldap_p://", "ldapsp://" or "ldaps_p://" or whatever seems most desirable. A server might listen on the standard ports accepting only proxied connections, or it might listen for normal connections on the standard ports and for proxy connections on alternative ports.
Yeah, that agrees with my read of the document. I think "ldapp://" and "ldapsp://" would be usable.
When a connection is accepted on a port marked as requiring the proxy protocol, it would read and process the proxy header to populate the appropriate data structures regarding connection, and then move on as it normally would to deal with the connection.
If this feature is of interest, I will probably spend a little time poking at it and seeing how much trouble it will be to implement.
Doesn't seem too problematic. I would only support the version 2 (binary) header, seems silly to implement the version 1 support for such an old spec.
On 11/19/20 5:04 PM, Howard Chu wrote:
Paul B. Henson wrote:
In general, I believe applications listening on a specific port are either expecting the proxy protocol header, or not, I do not think it is dynamically determined. As such, from an implementation perspective, my initial thought is that it would be implemented in terms of configuration as an additional URL specified via the -h option, something like "ldapp://" or "ldap_p://", "ldapsp://" or "ldaps_p://" or whatever seems most desirable. A server might listen on the standard ports accepting only proxied connections, or it might listen for normal connections on the standard ports and for proxy connections on alternative ports.
Yeah, that agrees with my read of the document. I think "ldapp://" and "ldapsp://" would be usable.
My suggestions:
1. Config directives for specifying IP address(es) and network(s) expected and trusted to send proxy protocol header.
2. Separate who peernamestyle for explicitly using the proxied IP addresses in ACLs. This would allow to specify ACLs with client-proxy relationship.
3. Log the proxied peername separately, similar how session tracking control is logged.
Ciao, Michael.
Michael Ströder wrote:
On 11/19/20 5:04 PM, Howard Chu wrote:
Paul B. Henson wrote:
In general, I believe applications listening on a specific port are either expecting the proxy protocol header, or not, I do not think it is dynamically determined. As such, from an implementation perspective, my initial thought is that it would be implemented in terms of configuration as an additional URL specified via the -h option, something like "ldapp://" or "ldap_p://", "ldapsp://" or "ldaps_p://" or whatever seems most desirable. A server might listen on the standard ports accepting only proxied connections, or it might listen for normal connections on the standard ports and for proxy connections on alternative ports.
Yeah, that agrees with my read of the document. I think "ldapp://" and "ldapsp://" would be usable.
My suggestions:
- Config directives for specifying IP address(es) and network(s)
expected and trusted to send proxy protocol header.
Sounds like unnecessary work. Just use an ACL.
- Separate who peernamestyle for explicitly using the proxied IP
addresses in ACLs. This would allow to specify ACLs with client-proxy relationship.
Yeah, maybe. Although I see this adding extra burden: if you have an existing deployment with peer-based ACLs, you will have to rewrite all of them after the proxy server is in place. I thought the entire point of adopting HAproxy protocol was so that you could continue operating with the client's addresses *transparently*. If you have to rewrite all of the rules regardless, I don't see any reason to bother with HAproxy.
- Log the proxied peername separately, similar how session tracking
control is logged.
Again, kind of defeats the purpose of transparently relaying the client's address.
On 11/19/2020 12:55 AM, Michael Ströder wrote:
Aaargh! I've missed the binary header part. So forget my former comments.
Version 1 of the protocol is ASCII, version 2 is binary. However, in both cases the proxy protocol data is removed and processed before the connection is handed down to the regular application logic, so it should be compatible with any server protocol.
On 11/19/2020 8:04 AM, Howard Chu wrote:
Yeah, that agrees with my read of the document. I think "ldapp://" and "ldapsp://" would be usable.
Cool.
Doesn't seem too problematic. I would only support the version 2 (binary) header, seems silly to implement the version 1 support for such an old spec.
I agree. I will put something together and come back when I have an initial diff for review, thanks…
On 11/19/2020 10:02 AM, Howard Chu wrote:
- Config directives for specifying IP address(es) and network(s)
expected and trusted to send proxy protocol header.
Sounds like unnecessary work. Just use an ACL.
I don't think an ldap level ACL would work for what he means? I think he wants to control what source IP addresses are allowed to connect to the proxy protocol port and arbitrarily say what the client IP address is which is passed down to the underlying ldap access control to be processed by ACLs. If you are using IP address based access control, you wouldn't want arbitrary clients to be able to connect and send a proxy header claiming to be somebody else.
However, given that the proxy protocol and regular access will be on different ports, this seems better handled at the network firewall level.
- Log the proxied peername separately, similar how session
tracking control is logged.
Again, kind of defeats the purpose of transparently relaying the client's address.
I'm not completely sure what he meant by this, but I was planning on modifying the initial connection log to indicate it was proxied, including both the address of the proxy and the proxied address.
so for a normal connection something like:
Nov 15 19:21:18 themis slapd[22846]: conn=482386 fd=89 ACCEPT from IP=134.71.247.3:54401 (IP=0.0.0.0:389)
and for a proxied connection something like:
Nov 15 19:21:18 themis slapd[22846]: conn=482386 fd=89 ACCEPT from IP=134.71.247.3:54401 proxyIP=10.128.1.0:34323 (IP=0.0.0.0:389)
Paul B. Henson wrote:
On 11/19/2020 10:02 AM, Howard Chu wrote:
- Config directives for specifying IP address(es) and network(s) expected and trusted to send proxy protocol header.
Sounds like unnecessary work. Just use an ACL.
I don't think an ldap level ACL would work for what he means? I think he wants to control what source IP addresses are allowed to connect to the proxy protocol port and arbitrarily say what the client IP address is which is passed down to the underlying ldap access control to be processed by ACLs. If you are using IP address based access control, you wouldn't want arbitrary clients to be able to connect and send a proxy header claiming to be somebody else.
However, given that the proxy protocol and regular access will be on different ports, this seems better handled at the network firewall level.
It sounded to me like he is not passing the client address transparently, which means that the actual peer address is still known, and would be governed by the current peer-based ACL mechanisms already, while introducing a new term to represent the proxied client address.
Anyway yes, controlling this at the firewall level would be better too.
- Log the proxied peername separately, similar how session
tracking control is logged.
Again, kind of defeats the purpose of transparently relaying the client's address.
I'm not completely sure what he meant by this, but I was planning on modifying the initial connection log to indicate it was proxied, including both the address of the proxy and the proxied address.
so for a normal connection something like:
Nov 15 19:21:18 themis slapd[22846]: conn=482386 fd=89 ACCEPT from IP=134.71.247.3:54401 (IP=0.0.0.0:389)
and for a proxied connection something like:
Nov 15 19:21:18 themis slapd[22846]: conn=482386 fd=89 ACCEPT from IP=134.71.247.3:54401 proxyIP=10.128.1.0:34323 (IP=0.0.0.0:389)
This would require that you actually read and process the proxy header immediately after the accept call. It strikes me that this is the wrong thing to do, if you also want to support TLS. I.e., that means the proxy header travels in the clear before you do a TLS handshake, which is probably not what one would want if they're using TLS. I suppose you would need to check what the proxy server actually supports, if it actually supports connecting to the target server using TLS.
On 11/19/2020 1:37 PM, Howard Chu wrote:
This would require that you actually read and process the proxy header immediately after the accept call. It strikes me that this is the wrong thing to do, if you also want to support TLS.
Unless I'm misunderstanding the specification, that is the only way it would work. The TLS negotiation, barring TLS interception by the proxy, is between the client and the backend server, not between the proxy and the backend server. The proxy would have no way to inject encrypted data into that communication. The "proxy" protocol is also used not just for application level proxies, but also for basic network level load-balancing. Those systems have no idea if TLS is in use are not, they are just shuffling packets back and forth between two endpoints, so again would have no way to insert encrypted data into the TLS layer.
The proxy protocol data is always shimmed in at the very beginning of the connection before the initial data supplied by the client, the server sucks it out and then passes the client data and the rest of the connection traffic to the protocol level.
This does seem to make it susceptible to man in the middle attacks where someone could swap out the proxy protocol data, but I think the general assumption is that the connection between the proxy/load balancer and the backend server is within a trusted network where such an attack is not a concern.
Paul B. Henson wrote:
On 11/19/2020 1:37 PM, Howard Chu wrote:
This would require that you actually read and process the proxy header immediately after the accept call. It strikes me that this is the wrong thing to do, if you also want to support TLS.
Unless I'm misunderstanding the specification, that is the only way it would work. The TLS negotiation, barring TLS interception by the proxy, is between the client and the backend server, not between the proxy and the backend server.
Yes, I understand that any TLS session initiated by the client is only between the client and the proxy server. But nobody says the proxy server can't talk to the backend server using its own TLS session. Unless you can point out anywhere in the HAproxy spec that explicitly forbids this.
This does seem to make it susceptible to man in the middle attacks where someone could swap out the proxy protocol data, but I think the general assumption is that the connection between the proxy/load balancer and the backend server is within a trusted network where such an attack is not a concern.
This assumption is not wise.
On 11/20/20 1:52 PM, Howard Chu wrote:
Paul B. Henson wrote:
On 11/19/2020 1:37 PM, Howard Chu wrote:
This would require that you actually read and process the proxy header immediately after the accept call. It strikes me that this is the wrong thing to do, if you also want to support TLS.
Unless I'm misunderstanding the specification, that is the only way it would work. The TLS negotiation, barring TLS interception by the proxy, is between the client and the backend server, not between the proxy and the backend server.
Yes, I understand that any TLS session initiated by the client is only between the client and the proxy server.
No, this is not necessarily the case. HA proxy can act as application-level proxy for some protocols (IIRC HTTP and SMTP) or as a TCP relay.
Paul mentioned the latter case where slapd is the TLS server end-point also from the client's perspective and HA proxy does *not* break up TLS connection.
Ciao, Michael.
On 11/20/2020 4:52 AM, Howard Chu wrote:
client and the proxy server. But nobody says the proxy server can't talk to the backend server using its own TLS session. Unless you can point out anywhere in the HAproxy spec that explicitly forbids this.
Eeehhhh.... I don't think it explicitly forbids it, but that's just not how it says the protocol works :).
"Another approach consists in prepending each connection with a header reporting the characteristics of the other side's connection. This method is simpler to implement, does not require any protocol-specific knowledge on either side, and completely fits the purpose since what is desired precisely is to know the other side's connection endpoints. It is easy to perform for the sender (just send a short header once the connection is established) and to parse for the receiver (simply perform one read() on the incoming connection to fill in addresses after an accept). The protocol used to carry connection information across proxies was thus called the PROXY protocol."
The intention is to be "simple", not require any "protocol specific knowledge" (such as whether or not the connection being proxied uses TLS), and is implemented as "just send a short header", which doesn't explicitly forbid doing a bunch of other stuff, but that's not what is specified.
Another excerpt:
"In both cases, the protocol simply consists in an easily parsable header placed by the connection initiator at the beginning of each connection. The protocol is intentionally stateless in that it does not expect the sender to wait for the receiver before sending the header, nor the receiver to send anything back."
Negotiating its own TLS session would not be stateless…
I've attached my first pass at adding proxy protocol support to slapd. I haven't updated any documentation/man pages yet, I'll start taking a look at that while you all eviscerate my code and let me know what needs to be fixed before merging :).
I'd like to backport this to OPENLDAP_REL_ENG_2_4 if/when it's accepted, hopefully that will be ok.
Thanks...
--On Friday, December 4, 2020 5:08 PM -0800 "Paul B. Henson" henson@acm.org wrote:
I've attached my first pass at adding proxy protocol support to slapd. I haven't updated any documentation/man pages yet, I'll start taking a look at that while you all eviscerate my code and let me know what needs to be fixed before merging :).
I'd like to backport this to OPENLDAP_REL_ENG_2_4 if/when it's accepted, hopefully that will be ok.
You should:
a) sign up for an account in gitlab if you haven't already: https://git.openldap.org
b) fork the master OpenLDAP repo
c) Create a development branch in your fork, generally named after the ITS. I.e.: git clone -b its# master
d) Apply your patch
e) Submit a merge request for review. You can mark it WIP if you feel like it may need additional work and is not yet ready for integration.
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
--On Saturday, December 5, 2020 2:40 PM -0800 Quanah Gibson-Mount quanah@symas.com wrote:
I'd like to backport this to OPENLDAP_REL_ENG_2_4 if/when it's accepted, hopefully that will be ok.
Also looks like I need to make further edits to the devel page on submissions, since this info is at the very bottom, and outdated info preceeds it.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
On Sat, Dec 05, 2020 at 02:44:12PM -0800, Quanah Gibson-Mount wrote:
Also looks like I need to make further edits to the devel page on submissions, since this info is at the very bottom, and outdated info preceeds it.
Ah, yah; I just saw the part about submitting patches in git format-patch format :).
Merge request submitted, thanks...
Haven't heard anything on my pull request since my last update:
https://git.openldap.org/openldap/openldap/-/merge_requests/209
Just wanted to touch base and see if there was anything else in need of fixing/changing with it.
Thanks...
On Mon, Dec 21, 2020 at 08:41:26PM -0800, Paul B. Henson wrote:
Haven't heard anything on my pull request since my last update:
https://git.openldap.org/openldap/openldap/-/merge_requests/209
Just wanted to touch base and see if there was anything else in need of fixing/changing with it.
Hi Paul, Thanks so much for working on this and sorry haven't had much time to give it a proper look yet. Will test a bit and review the load balancer side properly in the new year. A brief look found only some formatting (whitespace) inconsistencies so far.
Thanks again and wishing you nice holidays,
On 12/22/2020 2:19 AM, Ondřej Kuzník wrote:
Thanks so much for working on this and sorry haven't had much time to give it a proper look yet. Will test a bit and review the load balancer side properly in the new year. A brief look found only some formatting (whitespace) inconsistencies so far.
Cool, thanks for the update. No rush, don't mean to be pushy :), just wanted to make sure it didn't fall through the cracks. Happy holidays as well, and looking forward to following up with you next year, thanks…
-
Howard Chu -
Michael Ströder -
Ondřej Kuzník -
Ozgur -
Paul B. Henson -
Quanah Gibson-Mount