Hi,
I will probably also post this to the Postfix mailing list but it is fundamentally an (Open)LDAP question so here goes:
Short version: What is a recommended way to set up virtual mail hosting based on OpenLDAP? I.e. providing mail and authentication services, like SMTP and IMAP, using Postfix and Dovecot, for multiple *independent domains* such as example.net, example.org, example.com?
I am looking for RTFMs, HOWTOs, blogs, or any experience and anecdotes users on this list can provide. I myself no experience designing a DIT in LDAP (I am more at home in Postgres) and have much learning to do.
Long version: I know such setups exist and I have found many references in the archives of this list but there was never a completely straigt- forward answer that didn't say "it depends on your requirements" or involve frontends/add-ons like Jamm or Phamm, which I have no interest in.
So the requirements are basically: * Independent domains and users, i.e. john.doe@example.org is completely different/distinct from john.doe@example.net, even though both may be the same physical human being. * Thus accounts in different domains must have separate passwords fields * Groups and aliases must be possible * Performance should not be terrible, obviously * Applications such as Apache, Ejabberd, Wikis and Webmail clients (to name a few) which support LDAP authentication should be able to query the DIT or DITs without needing any hacks or ugly constructs (this is a vague requirement, I know).
Now I believe the question basically boils down to this:
Should we use multiple independent backend databases (DITs) or one large "hosting" database as described in [1,2]? Which of the two is the better approach? Which is more flexible, which has less administrative or functional overhead?
If we use multiple DITs we probably will have to glue them together somehow, won't we? How would queries against multiple independent DITs look?
Our current setup is more like [1,2], one big hosting database. It is a PostgreSQL database with a few tables for virtual domains (virtual_mailbox_domains in Postfix), virtual users (virtual_mailbox_maps), virtual aliases (virtual_alias_maps) and a few others. It is based loosely on [3].
This works fine but now the need has arisen to see if we can migrate that setup to an LDAP-based one, mainly for flexibility and compatibility with various authentication needs: many applications and services provide some kind of LDAP-based authentication but are hopelessly overwhelmed with SQL backends, especially when the queries are a bit complex.
Thanks in advance!
Andreas
PS: I gathered much from the article in [1] but by now it is over 7 years old and many things have changed so I can't follow it to the letter.
[1] http://www.linuxjournal.com/article/5917 [2] http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/059... [3] http://workaround.org/ispmail/lenny
On 04/10/10 20:47 +0200, Andreas Ntaflos wrote:
Hi,
I will probably also post this to the Postfix mailing list but it is fundamentally an (Open)LDAP question so here goes:
Short version: What is a recommended way to set up virtual mail hosting based on OpenLDAP? I.e. providing mail and authentication services, like SMTP and IMAP, using Postfix and Dovecot, for multiple *independent domains* such as example.net, example.org, example.com?
A very flexible is to implement a pam/nss layer on top of your DIT that presents your users as fully qualified to your system software.
For instance, on my system performing a 'getent passwd' lists all of my user accounts as fully qualified accounts (e.g. jsmith@example.net).
Assuming that dovecot and your other server software do not strip domains, or at least strip them in predictable ways, then you can use pam/nss to export your users as system level users.
Of course, there are many reasons for implementing direct LDAP support in your software, but you can build up from NSS accounts into something more flexible, rather than spiral downward into trying to work around different LDAP implementations within your software.
On Monday 04 October 2010 21:19:42 Dan White wrote:
On 04/10/10 20:47 +0200, Andreas Ntaflos wrote:
Short version: What is a recommended way to set up virtual mail hosting based on OpenLDAP? I.e. providing mail and authentication services, like SMTP and IMAP, using Postfix and Dovecot, for multiple *independent domains* such as example.net, example.org, example.com?
A very flexible is to implement a pam/nss layer on top of your DIT that presents your users as fully qualified to your system software.
[...]
Assuming that dovecot and your other server software do not strip domains, or at least strip them in predictable ways, then you can use pam/nss to export your users as system level users.
Dan, thank you for the reply and ideas! Essentially making all virtual users look like system users to Postfix and Dovecot (and other services) certainly sounds interesting but I am not sure if this won't make things more complex than they need to be. And wouldn't this approach require any services and applications to know how to handle PAM/NSS? I will keep it in mind, however, could come in handy in the future.
Our current setup using Postgres and virtual users, while complex enough, is quite adequate for our ISP needs. We just need to evaluate if and how it is feasible to model this setup using LDAP as a backend.
So I guess my question is really more about how to properly design a DIT that holds multiple independent domains and for each domain possibly hundreds of users and groups.
The problem is roughly equivalent to designing a proper relational database schema to manage and query user information only that a relational database schema is generally not designed with a single root or base node like the typical LDAP tree. This makes finding the information I require difficult.
Thanks again for your reply!
Andreas
On 05/10/10 04:17 +0200, Andreas Ntaflos wrote:
Dan, thank you for the reply and ideas! Essentially making all virtual users look like system users to Postfix and Dovecot (and other services) certainly sounds interesting but I am not sure if this won't make things more complex than they need to be. And wouldn't this approach require any services and applications to know how to handle PAM/NSS? I will keep it in mind, however, could come in handy in the future.
Most modern unixes make use of nss in a way that's transparent to applications - that is, any application which retrieves users and groups via the getpwent and getgrent system calls will make use of an nss plugin transparently.
An application does explicitly compile against PAM to make use of it, but shouldn't need to know any details about which PAM modules are used, and implementing a PAM LDAP module would not require a recompile of your applications.
Our current setup using Postgres and virtual users, while complex enough, is quite adequate for our ISP needs. We just need to evaluate if and how it is feasible to model this setup using LDAP as a backend.
Postfix, as you probably already know, has LDAP support for looking up most tables, which is how I implement virtual domain lookups.
So I guess my question is really more about how to properly design a DIT that holds multiple independent domains and for each domain possibly hundreds of users and groups.
I went with a flat design, which has worked well. That is:
uid=jsmith@example.net,ou=people,dc=example,dc=com uid=jsmith@custdom.com,ou=people,dc=example,dc=com
and
cn=jsmith@example.net,ou=groups,dc=example,dc=com cn=jsmith@custdom.com,ou=groups,dc=example,dc=com
The problem is roughly equivalent to designing a proper relational database schema to manage and query user information only that a relational database schema is generally not designed with a single root or base node like the typical LDAP tree. This makes finding the information I require difficult.
On Tuesday 05 October 2010 05:57:16 Dan White wrote:
Postfix, as you probably already know, has LDAP support for looking up most tables, which is how I implement virtual domain lookups.
I've asked this in another message as well but how would an LDAP query for Postfix's virtual_mailbox_domains look like, typically? Would the DIT need an additional ou, such as ou=VirtualDomains,dc=example,dc=com? I can't seem to get my head around that.
So I guess my question is really more about how to properly design a DIT that holds multiple independent domains and for each domain possibly hundreds of users and groups.
I went with a flat design, which has worked well. That is:
uid=jsmith@example.net,ou=people,dc=example,dc=com uid=jsmith@custdom.com,ou=people,dc=example,dc=com
and
cn=jsmith@example.net,ou=groups,dc=example,dc=com cn=jsmith@custdom.com,ou=groups,dc=example,dc=com
So example.com is the suffix for the whole of the hosting database and the uids and cns are fully qualified e-mail addresses? Sounds practical enough. I'll be experimenting with that starting next week.
Thanks again!
Andreas
On 07/10/10 22:57 +0200, Andreas Ntaflos wrote:
On Tuesday 05 October 2010 05:57:16 Dan White wrote:
Postfix, as you probably already know, has LDAP support for looking up most tables, which is how I implement virtual domain lookups.
I've asked this in another message as well but how would an LDAP query for Postfix's virtual_mailbox_domains look like, typically? Would the DIT need an additional ou, such as ou=VirtualDomains,dc=example,dc=com? I can't seem to get my head around that.
I hard code the list of domains within my main.cf, with virtual_mailbox_domains, since its a small list.
You could replace that static list with a call to LDAP, like:
virtual_mailbox_domains = ldap:/etc/postfix/list-of-domains.cf
with some kind of one off LDAP entry in your tree, but that's more trouble that I'm willing to deal with. If your list of domains changes fairly often, then that makes more since (but might incur a slight performance penalty).
It could look something like:
dn: ou=domainlist,dc=example,dc=com objectClass: top objectClass: organizationalUnit ou: domainlist description: custdom.com description: example.net description: example.com description: ...
So I guess my question is really more about how to properly design a DIT that holds multiple independent domains and for each domain possibly hundreds of users and groups.
I went with a flat design, which has worked well. That is:
uid=jsmith@example.net,ou=people,dc=example,dc=com uid=jsmith@custdom.com,ou=people,dc=example,dc=com
and
cn=jsmith@example.net,ou=groups,dc=example,dc=com cn=jsmith@custdom.com,ou=groups,dc=example,dc=com
So example.com is the suffix for the whole of the hosting database and the uids and cns are fully qualified e-mail addresses? Sounds practical enough. I'll be experimenting with that starting next week.
Right.
--On Monday, October 04, 2010 8:47 PM +0200 Andreas Ntaflos daff@pseudoterminal.org wrote:
Hi,
I will probably also post this to the Postfix mailing list but it is fundamentally an (Open)LDAP question so here goes:
Short version: What is a recommended way to set up virtual mail hosting based on OpenLDAP? I.e. providing mail and authentication services, like SMTP and IMAP, using Postfix and Dovecot, for multiple *independent domains* such as example.net, example.org, example.com?
Use the empty base "", and create each domain (example.net, example.org, example.com), etc with users in each. They will all be distinct from each other.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On Monday 04 October 2010 21:27:58 Quanah Gibson-Mount wrote:
--On Monday, October 04, 2010 8:47 PM +0200 Andreas Ntaflos
Short version: What is a recommended way to set up virtual mail hosting based on OpenLDAP? I.e. providing mail and authentication services, like SMTP and IMAP, using Postfix and Dovecot, for multiple *independent domains* such as example.net, example.org, example.com?
Use the empty base "", and create each domain (example.net, example.org, example.com), etc with users in each. They will all be distinct from each other.
Hi Quanah, thank you too for your reply!
Could you elaborate on using an empty base/suffix? The concept seems strange to me and I don't really understand the implications (but I have very little experience with LDAP as of yet). However, googling around most posts seem to describe only problems with empty suffixes. Do you know of any examples or documentation that describe this approach more detailed?
Thanks!
Andreas
--On Tuesday, October 05, 2010 4:22 AM +0200 Andreas Ntaflos daff@pseudoterminal.org wrote:
Hi Quanah, thank you too for your reply!
Could you elaborate on using an empty base/suffix? The concept seems strange to me and I don't really understand the implications (but I have very little experience with LDAP as of yet). However, googling around most posts seem to describe only problems with empty suffixes. Do you know of any examples or documentation that describe this approach more detailed?
Download and install Zimbra, it uses exactly this approach. And I've been using the empty suffix for years, it works just fine. There have been issues unique to it in the past.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On Tuesday 05 October 2010 04:35:59 Quanah Gibson-Mount wrote:
--On Tuesday, October 05, 2010 4:22 AM +0200 Andreas Ntaflos
daff@pseudoterminal.org wrote:
Hi Quanah, thank you too for your reply!
Could you elaborate on using an empty base/suffix? The concept seems strange to me and I don't really understand the implications (but I have very little experience with LDAP as of yet). However, googling around most posts seem to describe only problems with empty suffixes. Do you know of any examples or documentation that describe this approach more detailed?
Download and install Zimbra, it uses exactly this approach. And I've been using the empty suffix for years, it works just fine. There have been issues unique to it in the past.
I've always wanted to have a look at Zimbra and to see how and if we could integrate it in our hosting setup. Now I have a second reason, thanks. It's quite a beast, though.
Andreas
On Monday, 4 October 2010 19:47:16 Andreas Ntaflos wrote:
Hi,
I will probably also post this to the Postfix mailing list but it is fundamentally an (Open)LDAP question so here goes:
Short version: What is a recommended way to set up virtual mail hosting based on OpenLDAP? I.e. providing mail and authentication services, like SMTP and IMAP, using Postfix and Dovecot, for multiple *independent domains* such as example.net, example.org, example.com?
I am looking for RTFMs, HOWTOs, blogs, or any experience and anecdotes users on this list can provide. I myself no experience designing a DIT in LDAP (I am more at home in Postgres) and have much learning to do.
Long version: I know such setups exist and I have found many references in the archives of this list but there was never a completely straigt- forward answer that didn't say "it depends on your requirements" or involve frontends/add-ons like Jamm or Phamm, which I have no interest in.
So the requirements are basically:
- Independent domains and users, i.e. john.doe@example.org is
completely different/distinct from john.doe@example.net, even though both may be the same physical human being.
- Thus accounts in different domains must have separate passwords
fields
- Groups and aliases must be possible
What specifically do you mean by groups?
- Performance should not be terrible, obviously
How many accounts, and what are the specs (memory, CPU speed, disk speed) of the hardware you will be running the LDAP server on?
- Applications such as Apache, Ejabberd, Wikis and Webmail clients (to
name a few) which support LDAP authentication should be able to query the DIT or DITs without needing any hacks or ugly constructs (this is a vague requirement, I know).
Now I believe the question basically boils down to this:
Should we use multiple independent backend databases (DITs) or one large "hosting" database as described in [1,2]?
There are limits on the number of backend databases you can have, so for any hosting application, individual databases per domain will probably not be feasible. I would recommend a single database in most cases. What the suffix is is largely irrelevant IMHO. For example, there is almost no difference between uid=joe@example.com,dc=example,dc=com (IOW, empty suffix) vs uid=joe@example.com,dc=example,dc=com,o=mysitename. However, if accounts will have aliases on different domains, it does not make much sense to over- complicate the structure for no benefit, uid=joe@example.com,o=mysitename would be fine.
Which of the two is the better approach? Which is more flexible, which has less administrative or functional overhead?
Obviously multiple databases gives you more flexibility for more administrative overhead.
The advantages of multiple databases are: -Different index settings -Independent replication configuration -Different access controls
If these will not provide significant benefits, the additional overhead of managing multiple databases (and their tuning) is not justified IMHO.
If we use multiple DITs we probably will have to glue them together somehow, won't we? How would queries against multiple independent DITs look?
Our current setup is more like [1,2], one big hosting database. It is a PostgreSQL database with a few tables for virtual domains (virtual_mailbox_domains in Postfix), virtual users (virtual_mailbox_maps), virtual aliases (virtual_alias_maps) and a few others. It is based loosely on [3].
This works fine but now the need has arisen to see if we can migrate that setup to an LDAP-based one, mainly for flexibility and compatibility with various authentication needs: many applications and services provide some kind of LDAP-based authentication but are hopelessly overwhelmed with SQL backends, especially when the queries are a bit complex.
Thanks in advance!
While NSS was mentioned, I would avoid stuffing everthing through NSS, when all the applications you have listed so far already have direct LDAP support. However, if you are going to use LDAP for any "system" accounts, you may want to consider the DIT structure a bit more carefully than if you aren't.
Regards, Buchan
On Tuesday 05 October 2010 10:19:37 Buchan Milne wrote:
On Monday, 4 October 2010 19:47:16 Andreas Ntaflos wrote:
So the requirements are basically:
- Independent domains and users, i.e. john.doe@example.org is
completely different/distinct from john.doe@example.net, even though both may be the same physical human being.
- Thus accounts in different domains must have separate passwords
fields
- Groups and aliases must be possible
What specifically do you mean by groups?
Thank you too for your reply! It is most helpful.
By group support I just mean the usual way to group users together, as in ou=groups,dc=example,dc=com. Users on different hosted domains won't/can't be in the same group, if that matters any.
- Performance should not be terrible, obviously
How many accounts, and what are the specs (memory, CPU speed, disk speed) of the hardware you will be running the LDAP server on?
This was more of a requirement for the design, i.e. avoiding unnecessary lookups and things like that.
There are about 10 hosted domains and the most active one has about 300 accounts, which is not very much. The others are much less active and have much less users. But it is quite possible those numbers will grow in the next year. The hardware is a dual quad core Xeon 5520 with 12GB RAM and four SATA disks RAID-10ed together. Up until now the server hasn't broken a sweat.
- Applications such as Apache, Ejabberd, Wikis and Webmail clients
(to
name a few) which support LDAP authentication should be able to query the DIT or DITs without needing any hacks or ugly constructs (this is a vague requirement, I know).
Now I believe the question basically boils down to this:
Should we use multiple independent backend databases (DITs) or one large "hosting" database as described in [1,2]?
There are limits on the number of backend databases you can have, so for any hosting application, individual databases per domain will probably not be feasible. I would recommend a single database in most cases. What the suffix is is largely irrelevant IMHO. For example, there is almost no difference between uid=joe@example.com,dc=example,dc=com (IOW, empty suffix) vs uid=joe@example.com,dc=example,dc=com,o=mysitename. However, if accounts will have aliases on different domains, it does not make much sense to over- complicate the structure for no benefit, uid=joe@example.com,o=mysitename would be fine.
Ok, so a single hosting database is clearly the better approach (others have suggested similarly). It also seems that a DIT for such a hosting database doesn't have to be as complicated as I had initially thought, which is great.
However, when using uid=joe@example.com,o=mysitename, how would Postfix (or anything else) look up which virtual domains it is supposed to serve, i.e. virtual_mailbox_domains?
That's the one think about which I am not yet quite clear. In our relational database there's a simple table "virtual_domains" with a name and an ID for the hosted domains. Postfix only wants the key (the domain name) to exist when doing a lookup so a query is just a simple "SELECT 1 FROM virtual_domains WHERE name='%s'", %s being the domain name of course. How would a "corresponding" LDAP query look like?
Which of the two is the better approach? Which is more flexible, which has less administrative or functional overhead?
Obviously multiple databases gives you more flexibility for more administrative overhead.
The advantages of multiple databases are: -Different index settings -Independent replication configuration -Different access controls
If these will not provide significant benefits, the additional overhead of managing multiple databases (and their tuning) is not justified IMHO.
Sounds reasonable.
While NSS was mentioned, I would avoid stuffing everthing through NSS, when all the applications you have listed so far already have direct LDAP support. However, if you are going to use LDAP for any "system" accounts, you may want to consider the DIT structure a bit more carefully than if you aren't.
Right, I thought as much as well. Real users (i.e. not accounts in virtual domains) are no concern in our simple ISP-like setup so while basing everyting on NSS is a very elegant and transparent solution, as Dan has explained, I think it's too much additional complexity for us.
Thanks again for your reply, much appreciated.
Andreas
On Thursday, 7 October 2010 21:37:30 Andreas Ntaflos wrote:
On Tuesday 05 October 2010 10:19:37 Buchan Milne wrote:
On Monday, 4 October 2010 19:47:16 Andreas Ntaflos wrote:
So the requirements are basically:
- Independent domains and users, i.e. john.doe@example.org is
completely different/distinct from john.doe@example.net, even though both may be the same physical human being.
- Thus accounts in different domains must have separate passwords
fields
- Groups and aliases must be possible
What specifically do you mean by groups?
Thank you too for your reply! It is most helpful.
By group support I just mean the usual way to group users together, as in ou=groups,dc=example,dc=com. Users on different hosted domains won't/can't be in the same group, if that matters any.
But, what would you use the groups for? Authorization to specific services only available to specific groups?
[...]
- Applications such as Apache, Ejabberd, Wikis and Webmail clients
(to
name a few) which support LDAP authentication should be able to query the DIT or DITs without needing any hacks or ugly constructs (this is a vague requirement, I know).
Now I believe the question basically boils down to this:
Should we use multiple independent backend databases (DITs) or one large "hosting" database as described in [1,2]?
There are limits on the number of backend databases you can have, so for any hosting application, individual databases per domain will probably not be feasible. I would recommend a single database in most cases. What the suffix is is largely irrelevant IMHO. For example, there is almost no difference between uid=joe@example.com,dc=example,dc=com (IOW, empty suffix) vs uid=joe@example.com,dc=example,dc=com,o=mysitename. However, if accounts will have aliases on different domains, it does not make much sense to over- complicate the structure for no benefit, uid=joe@example.com,o=mysitename would be fine.
Ok, so a single hosting database is clearly the better approach (others have suggested similarly). It also seems that a DIT for such a hosting database doesn't have to be as complicated as I had initially thought, which is great.
However, when using uid=joe@example.com,o=mysitename, how would Postfix (or anything else) look up which virtual domains it is supposed to serve, i.e. virtual_mailbox_domains?
That's the one think about which I am not yet quite clear. In our relational database there's a simple table "virtual_domains" with a name and an ID for the hosted domains. Postfix only wants the key (the domain name) to exist when doing a lookup so a query is just a simple "SELECT 1 FROM virtual_domains WHERE name='%s'", %s being the domain name of course. How would a "corresponding" LDAP query look like?
Don't try and one-to-one map an RDBMS table to an LDAP container, start by mapping the query to an ldap search (basedn, filter, result attributes) first.
You could use something like this:
dn: domain=domain1.example.com,dc=example,dc=com objectclass: domain domain: domain1.example.com
Then you would use an LDAP map with search_base = dc=example,dc=com query_filter = (&(objectclass=domain)(domain=%s)) result_attribute = domain
You may choose to have more separation, with a separate basedn for domains, but it isn't necessary.
But, depending on your requirements, you may need more complex schemas and/or layouts.
Regards, Buchan
On Friday 08 October 2010 13:39:25 Buchan Milne wrote:
- Groups and aliases must be possible
What specifically do you mean by groups?
By group support I just mean the usual way to group users together, as in ou=groups,dc=example,dc=com. Users on different hosted domains won't/can't be in the same group, if that matters any.
But, what would you use the groups for? Authorization to specific services only available to specific groups?
Exactly. That shouldn't be a problem, should it?
However, when using uid=joe@example.com,o=mysitename, how would Postfix (or anything else) look up which virtual domains it is supposed to serve, i.e. virtual_mailbox_domains?
That's the one think about which I am not yet quite clear. In our relational database there's a simple table "virtual_domains" with a name and an ID for the hosted domains. Postfix only wants the key (the domain name) to exist when doing a lookup so a query is just a simple "SELECT 1 FROM virtual_domains WHERE name='%s'", %s being the domain name of course. How would a "corresponding" LDAP query look like?
Don't try and one-to-one map an RDBMS table to an LDAP container, start by mapping the query to an ldap search (basedn, filter, result attributes) first.
Yeah, I am having some difficulties with that :)
You could use something like this:
dn: domain=domain1.example.com,dc=example,dc=com objectclass: domain domain: domain1.example.com
Then you would use an LDAP map with search_base = dc=example,dc=com query_filter = (&(objectclass=domain)(domain=%s)) result_attribute = domain
Thank you, that makes a lot of sense. Based on that it should not be too difficult to map other necessary queries from SQL to LDAP.
You may choose to have more separation, with a separate basedn for domains, but it isn't necessary.
I see now how it is supposed to work. Hopefully I'll have some time this week to build a test setup. Expect more concrete questions about OpenLDAP :)
Thanks again for your help and ideas!
Andreas
openldap-technical@openldap.org
-
Andreas Ntaflos -
Buchan Milne -
Dan White -
Quanah Gibson-Mount