First, can anyone recommend a good link or book on ldap search filters? I have a few books on LDAP ("Deploying OpenLDAP" from Apress and the O'reilly LDAP book) and they devote less than a page to the topic. There doesn't seem to be a ton of info on the Google about it either. What am I overlooking?
What I'm trying to do is create a search filter so when I search a certain tree for all DNs the result will return only DNs that are members of a certain "groupOfNames" group. Is that possible?
No, since LDAP does not allow relations in filters (LDAP is not RDBMS). Your question is not specific to OpenLDAP nor to its interoperation with other systems, so it rather belongs to a generic LDAP forum.
The specific problem you want to solve could be solved with OpenLDAP if you setup and maintain an instance of slapo-memberof(5). This will maintain for you reverse links. It means that for each group "cn=agroup", with members "member=cn=someone", "member=cn=someoneelse", the entries "cn=someone", "cn=someoneelse" will have an operational attribute "memberof" containing the DN of the groups they're members. So searching for "(memberOf=cn=agroup)" will return the DNs of the entries that are members of that group.
p.