Hello all. I've read the news posting at http://blog.suretecsystems.com/archives/40-OpenLDAP-Weekly-News-Issue-5.html... for multimaster N-Way sync. Very good stuff.
I've configured the cn=config backend, and I can browse it with my LDAP browser on both my Masters. (I have two servers)
I have created the replication agreements and are able to add them to the cn=config as documented in the URL above. No problem on both servers.
When I add the data sync, I get a little confused.
Below, for ${BACKEND} I assume I put something like bdb for the database backend correct? If I do this it does not fail, but the sync does not happen. I don't see a whole lot of errors either with the sync.
dn: olcDatabase={1}$BACKEND,cn=config objectClass: olcDatabaseConfig objectClass: olc${BACKEND}Config olcDatabase: {1}$BACKEND olcSuffix: $BASEDN olcDbDirectory: ./db olcRootDN: $MANAGERDN olcRootPW: $PASSWD olcSyncRepl: rid=004 provider=$URI1 binddn="$MANAGERDN" bindmethod=simple credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1 olcSyncRepl: rid=005 provider=$URI2 binddn="$MANAGERDN" bindmethod=simple credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1 olcSyncRepl: rid=006 provider=$URI3 binddn="$MANAGERDN" bindmethod=simple credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1 olcMirrorMode: TRUE
dn: olcOverlay=syncprov,olcDatabase={1}${BACKEND},cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov
I end up with olcDatabase={1}bdb in there twice.
What should the $BACKEND value be if not bdb. (if by db backend is bdb).
Thanks for any insight. For now, I am going to have to revert to Master+Slave via syncrepl and referrals.
Sellers
|----------------------------------------------------------------------| Chris G. Sellers, MLS Lead Internet Engineer National Institute for Technology & Liberal Education 535 West William Street, Ann Arbor, Michigan 48103 chris.sellers@nitle.org 734.661.2318
<quote who="Chris G. Sellers">
Hello all. I've read the news posting at http://blog.suretecsystems.com/archives/40-OpenLDAP-Weekly-News-Issue-5.html... for multimaster N-Way sync. Very good stuff.
I've configured the cn=config backend, and I can browse it with my LDAP browser on both my Masters. (I have two servers)
I have created the replication agreements and are able to add them to the cn=config as documented in the URL above. No problem on both servers.
When I add the data sync, I get a little confused.
Below, for ${BACKEND} I assume I put something like bdb for the database backend correct? If I do this it does not fail, but the sync does not happen. I don't see a whole lot of errors either with the sync.
bdb or hdb, depending on how you've built slapd.
Logs?
dn: olcDatabase={1}$BACKEND,cn=config objectClass: olcDatabaseConfig objectClass: olc${BACKEND}Config olcDatabase: {1}$BACKEND olcSuffix: $BASEDN olcDbDirectory: ./db olcRootDN: $MANAGERDN olcRootPW: $PASSWD olcSyncRepl: rid=004 provider=$URI1 binddn="$MANAGERDN" bindmethod=simple credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1 olcSyncRepl: rid=005 provider=$URI2 binddn="$MANAGERDN" bindmethod=simple credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1 olcSyncRepl: rid=006 provider=$URI3 binddn="$MANAGERDN" bindmethod=simple credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1 olcMirrorMode: TRUE
dn: olcOverlay=syncprov,olcDatabase={1}${BACKEND},cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov
I end up with olcDatabase={1}bdb in there twice.
What should the $BACKEND value be if not bdb. (if by db backend is bdb).
Thanks for any insight. For now, I am going to have to revert to Master+Slave via syncrepl and referrals.
Sellers
|----------------------------------------------------------------------| Chris G. Sellers, MLS Lead Internet Engineer National Institute for Technology & Liberal Education 535 West William Street, Ann Arbor, Michigan 48103 chris.sellers@nitle.org 734.661.2318
Gavin,
Thanks is what I thought. The logs don't show much. I've set things up pretty much as described in the blogpost listed below.
I see it walk the cn=config database doing a search and compare I expect, but then it does not appear to do anything else.
I've clobbered the config now so I can't give you a good logdebug but if I get it back to the same point I can share.
Thanks for validating the config settings for me.
Sellers
On Jan 4, 2008, at 5:59 PM, Gavin Henry wrote:
<quote who="Chris G. Sellers"> > Hello all. I've read the news posting at > http://blog.suretecsystems.com/archives/40-OpenLDAP-Weekly-News-Issue-5.html#extended > for multimaster N-Way sync. Very good stuff. > > I've configured the cn=config backend, and I can browse it with my > LDAP browser on both my Masters. (I have two servers) > > I have created the replication agreements and are able to add them to > the cn=config as documented in the URL above. No problem on both > servers. > > When I add the data sync, I get a little confused. > > Below, for ${BACKEND} I assume I put something like bdb for the > database backend correct? If I do this it does not fail, but the > sync > does not happen. I don't see a whole lot of errors either with the > sync.
bdb or hdb, depending on how you've built slapd.
Logs?
dn: olcDatabase={1}$BACKEND,cn=config objectClass: olcDatabaseConfig objectClass: olc${BACKEND}Config olcDatabase: {1}$BACKEND olcSuffix: $BASEDN olcDbDirectory: ./db olcRootDN: $MANAGERDN olcRootPW: $PASSWD olcSyncRepl: rid=004 provider=$URI1 binddn="$MANAGERDN" bindmethod=simple credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1 olcSyncRepl: rid=005 provider=$URI2 binddn="$MANAGERDN" bindmethod=simple credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1 olcSyncRepl: rid=006 provider=$URI3 binddn="$MANAGERDN" bindmethod=simple credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1 olcMirrorMode: TRUE
dn: olcOverlay=syncprov,olcDatabase={1}${BACKEND},cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov
I end up with olcDatabase={1}bdb in there twice.
What should the $BACKEND value be if not bdb. (if by db backend is bdb).
Thanks for any insight. For now, I am going to have to revert to Master+Slave via syncrepl and referrals.
Sellers
| ----------------------------------------------------------------------| Chris G. Sellers, MLS Lead Internet Engineer National Institute for Technology & Liberal Education 535 West William Street, Ann Arbor, Michigan 48103 chris.sellers@nitle.org 734.661.2318
______________________________________________ Chris G. Sellers | NITLE Technology 734.661.2318 | chris.sellers@nitle.org AIM: imthewherd | GTalk: cgseller@gmail.com
Chris G. Sellers wrote:
Gavin,
Thanks is what I thought. The logs don't show much. I've set things up pretty much as described in the blogpost listed below.
I see it walk the cn=config database doing a search and compare I expect, but then it does not appear to do anything else.
I've clobbered the config now so I can't give you a good logdebug but if I get it back to the same point I can share.
Thanks for validating the config settings for me.
Sellers
OK, look forward to it.
Ok, I did figure out that I had 1 password wrong in one agreement, which is why I saw the password error every so often, so that side- issue is taken care of.
I moved the agreements out of the cn=config and put it directly into the slapd.conf just to take away some of the variables.
I still get the issue where it has Sync State control issues:
daemon: waked daemon: select: listen=7 active_threads=0 tvp=zero daemon: activity on 1 descriptor daemon: activity on: 12r daemon: read activity on 12 daemon: select: listen=7 active_threads=0 tvp=zero connection_read(12): input error=-2 id=4, closing. daemon: activity on 1 descriptor daemon: removing 12 daemon: waked daemon: select: listen=7 active_threads=0 tvp=zero do_syncrep2: rid=010 got search entry without Sync State control do_syncrepl: rid=010 retrying (4 retries left) daemon: activity on 1 descriptor daemon: waked daemon: select: listen=7 active_threads=0 tvp=zero
On Jan 7, 2008, at 2:01 PM, Gavin Henry wrote:
Chris G. Sellers wrote:
Gavin,
Thanks is what I thought. The logs don't show much. I've set things up pretty much as described in the blogpost listed below.
I see it walk the cn=config database doing a search and compare I expect, but then it does not appear to do anything else.
I've clobbered the config now so I can't give you a good logdebug but if I get it back to the same point I can share.
Thanks for validating the config settings for me.
Sellers
OK, look forward to it.
-- Kind Regards,
Gavin Henry. Managing Director.
T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry@suretecsystems.com
Open Source. Open Solutions(tm).
______________________________________________ Chris G. Sellers | NITLE Technology 734.661.2318 | chris.sellers@nitle.org AIM: imthewherd | GTalk: cgseller@gmail.com
Chris G. Sellers wrote:
Ok, I did figure out that I had 1 password wrong in one agreement, which is why I saw the password error every so often, so that side-issue is taken care of.
I moved the agreements out of the cn=config and put it directly into the slapd.conf just to take away some of the variables.
I still get the issue where it has Sync State control issues:
Can you provide all your configuration please and data?
Thanks,
Gavin.
Ok, I just made a change to my config and it appears to be working.
I referenced my master+slave setup, and I saw
overlay syncprov
so I added that to both masters, restarted, and viola, it's replicating both directions well.
Do you think that could have been my problem? Here is my config (names changed to protect the innocent)
syncrepl rid=010 provider=ldap://ldap1.nitle.org:1000 binddn="cn=manager,dc=abc,dc=xyz" bindmethod=simple credentials=SECRET searchbase="dc=abc,dc=xyz" type=refreshAndPersist scope=sub interval=00:00:00:10 retry="5 5 100 5" timeout=1 schemachecking=off
attrs = "*,structuralObjectClass ,entryUUID ,entryCSN,creatorsName,createTimestamp,modifiersName,modifyTimestamp"
syncrepl rid=011 provider=ldap://ldap2.nitle.org:1000 binddn="cn=manager,dc=abc,dc=xyz" bindmethod=simple credentials=SECRET searchbase="dc=abc,dc=xyz" type=refreshAndPersist schemachecking=off scope=sub interval=00:00:00:10 retry="5 5 100 5" timeout=1
attrs = "*,structuralObjectClass ,entryUUID ,entryCSN,creatorsName,createTimestamp,modifiersName,modifyTimestamp"
overlay syncprov syncprov-checkpoint 100 10 syncprov-sessionlog 100
Both masters have the same config options.
On Jan 7, 2008, at 4:17 PM, Gavin Henry wrote:
Chris G. Sellers wrote:
Ok, I did figure out that I had 1 password wrong in one agreement, which is why I saw the password error every so often, so that side-issue is taken care of.
I moved the agreements out of the cn=config and put it directly into the slapd.conf just to take away some of the variables.
I still get the issue where it has Sync State control issues:
Can you provide all your configuration please and data?
Thanks,
Gavin.
-- Kind Regards,
Gavin Henry. Managing Director.
T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry@suretecsystems.com
Open Source. Open Solutions(tm).
______________________________________________ Chris G. Sellers | NITLE Technology 734.661.2318 | chris.sellers@nitle.org AIM: imthewherd | GTalk: cgseller@gmail.com
--On January 7, 2008 4:24:20 PM -0500 "Chris G. Sellers" Chris.Sellers@nitle.org wrote:
Ok, I just made a change to my config and it appears to be working.
I referenced my master+slave setup, and I saw
overlay syncprov
so I added that to both masters, restarted, and viola, it's replicating both directions well.
Well, it is required to have syncprov on a master. Because that's the sync provider module. Glad it's working.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Thanks for the validation. I'm going to document this, I'll be in touch.
Sellers
On Jan 7, 2008, at 4:53 PM, Quanah Gibson-Mount wrote:
--On January 7, 2008 4:24:20 PM -0500 "Chris G. Sellers" <Chris.Sellers@nitle.org
wrote:
Ok, I just made a change to my config and it appears to be working.
I referenced my master+slave setup, and I saw
overlay syncprov
so I added that to both masters, restarted, and viola, it's replicating both directions well.
Well, it is required to have syncprov on a master. Because that's the sync provider module. Glad it's working.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
<quote who="Chris G. Sellers">
Thanks for the validation. I'm going to document this, I'll be in touch.
It already it...almost ;-)
I committed the blog entry to the main docs, I've just got to clean it up.
Feel free though.
Thanks.
Sellers
On Jan 7, 2008, at 4:53 PM, Quanah Gibson-Mount wrote:
--On January 7, 2008 4:24:20 PM -0500 "Chris G. Sellers" <Chris.Sellers@nitle.org
wrote:
Ok, I just made a change to my config and it appears to be working.
I referenced my master+slave setup, and I saw
overlay syncprov
so I added that to both masters, restarted, and viola, it's replicating both directions well.
Well, it is required to have syncprov on a master. Because that's the sync provider module. Glad it's working.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
<quote who="Chris G. Sellers">
Ok, I just made a change to my config and it appears to be working.
I referenced my master+slave setup, and I saw
overlay syncprov
This was cleary stated in the blog post though:
This sets up syncrepl as a provider (since these are all masters):
CODE: dn: cn=module,cn=config objectClass: olcModuleList cn: module olcModulePath: /usr/local/libexec/openldap olcModuleLoad: syncprov.la
AND:
dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov
;-)
I do have it working, but it appears that my data replication information keeps getting lost
When I connect to the cn=config database, I see a bdb{1} and bdb{2} cn. the first (or second sometimes) is the slapd.conf information, and the other is the database records.
I'm thinking there is a conflict because after a restart for example, I end up with only one and my sync replication information for the regular data (not cn=config) disappears and I can't sync any longer.
should I be using a unique number for bdb{} in my ldif. Here is my ldif file I'm using
dn: olcDatabase={1}bdb,cn=config objectClass: olcDatabaseConfig objectClass: olcbdbConfig olcDatabase: {1}bdb olcSuffix: dc=nitle,dc=org olcDbDirectory: /home/ldap/openldap/var/openldap-data olcRootDN: cn=xxxx,dc=nitle,dc=org olcRootPW: xxxxx olcSyncRepl: rid=010 provider=ldap://ldap1.nitle.org:0000 binddn="cn=xxxx,ou=replication,dc=nitle,dc=org" bindmethod=simple credentials=xxxxx searchbase="dc=nitle,dc=org" type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1 schemachecking=off olcSyncRepl: rid=011 provider=ldap://ldap2.nitle.org:0000 binddn="cn=xxxx,ou=replication,dc=nitle,dc=org" bindmethod=simple credentials=xxxx searchbase="dc=nitle,dc=org" type=refreshOnly schemachecking=off interval=00:00:00:10 retry="5 5 300 5" timeout=1 olcMirrorMode: TRUE
dn: olcOverlay=syncprov,olcDatabase={1}bdb,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov
On Jan 7, 2008, at 5:43 PM, Gavin Henry wrote:
<quote who="Chris G. Sellers"> > Ok, I just made a change to my config and it appears to be working. > > I referenced my master+slave setup, and I saw > > overlay syncprov
This was cleary stated in the blog post though:
This sets up syncrepl as a provider (since these are all masters):
CODE: dn: cn=module,cn=config objectClass: olcModuleList cn: module olcModulePath: /usr/local/libexec/openldap olcModuleLoad: syncprov.la
AND:
dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov
;-)
--On Tuesday, January 08, 2008 2:43 PM -0500 "Chris G. Sellers" Chris.Sellers@nitle.org wrote:
I do have it working, but it appears that my data replication information keeps getting lost
You do know that if you use slapd.conf, that changes made to cn=config get lost on restarts, right? If you want them to be preserved, you need to convert to only using the config database and get rid of slapd.conf entirely.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Ah ha. I figured as much. Thanks for making less dense.
Sellers
On Jan 8, 2008, at 3:14 PM, Quanah Gibson-Mount wrote:
--On Tuesday, January 08, 2008 2:43 PM -0500 "Chris G. Sellers" <Chris.Sellers@nitle.org
wrote:
I do have it working, but it appears that my data replication information keeps getting lost
You do know that if you use slapd.conf, that changes made to cn=config get lost on restarts, right? If you want them to be preserved, you need to convert to only using the config database and get rid of slapd.conf entirely.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
Ok, when I configure it and trap the logs, one of my systems shows:
[ldap@dv1nitle05-ldap1]$ [ldap@dv1nitle05-ldap1]$ daemon: select: listen=7 active_threads=0 tvp=zero do_syncrep2: rid=011 got search entry without Sync State control do_syncrepl: rid=011 retrying (3 retries left) daemon: activity on 1 descriptor daemon: waked daemon: select: listen=7 active_threads=0 tvp=zero daemon: select: listen=7 active_threads=0 tvp=zero do_syncrep2: rid=011 got search entry without Sync State control do_syncrepl: rid=011 retrying (2 retries left) daemon: activity on 1 descriptor daemon: waked daemon: select: listen=7 active_threads=0 tvp=zero daemon: select: listen=7 active_threads=0 tvp=zero do_syncrep2: rid=011 got search entry without Sync State control do_syncrepl: rid=011 retrying (1 retries left) daemon: activity on 1 descriptor daemon: waked daemon: select: listen=7 active_threads=0 tvp=zero daemon: select: listen=7 active_threads=0 tvp=zero do_syncrep2: rid=011 got search entry without Sync State control do_syncrepl: rid=011 retrying daemon: activity on 1 descriptor daemon: waked daemon: select: listen=7 active_threads=0 tvp=zero daemon: select: listen=7 active_threads=0 tvp=zero do_syncrep2: rid=011 got search entry without Sync State control do_syncrepl: rid=011 retrying (4 retries left) daemon: activity on 1 descriptor daemon: waked daemon: select: listen=7 active_threads=0 tvp=zero
the other one is trying to do the sync with logs like
=> access_allowed: read access to "cn=manager,dc=nitle,dc=org" "hasSubordinates" requested => dn: [1] => dn: [2] cn=subschema => dn: [3] cn=log => dnpat: [4] ^([^,]*,)?ou=[^,]+,(dc=[^,]+(,dc=[^,]+)*)$ nsub: 3 => acl_get: [5] attr hasSubordinates => slap_access_allowed: result not in cache (hasSubordinates) => acl_mask: access to entry "cn=manager,dc=nitle,dc=org", attr "hasSubordinates" requested => acl_mask: to value by "cn=manager,cn=config", (=0) <= check a_dn_pat: cn=replslave,ou=replication,dc=nitle,dc=org <= check a_dn_pat: cn=mirrormode,ou=replication,dc=nitle,dc=org <= check a_dn_pat: uid=ldaprw,ou=staff,dc=nitle,dc=org <= check a_dn_pat: uid=ldapro,ou=staff,dc=nitle,dc=org <= check a_dn_pat: cn=manager,cn=config <= acl_mask: [5] applying write(=wrscxd) (stop) <= acl_mask: [5] mask: write(=wrscxd) => slap_access_allowed: read access granted by write(=wrscxd) => access_allowed: read access granted by write(=wrscxd) daemon: activity on 1 descriptor daemon: activity on: 11r daemon: read activity on 11 daemon: select: listen=7 active_threads=0 tvp=zero connection_read(11): input error=-2 id=5, closing. daemon: activity on 1 descriptor daemon: removing 11 daemon: waked daemon: select: listen=7 active_threads=0 tvp=zero
It appears I'm also getting bind failures as the cn=manager,cn=config user that I have, however, I don't always get the errors, I get successful binds as that user to get the info above, but then later after it's been running for a short while, it starts to get err49 (auth err)
I'm investigating more. I wonder if I'm overriding my databases...... On Jan 4, 2008, at 5:59 PM, Gavin Henry wrote:
<quote who="Chris G. Sellers"> > Hello all. I've read the news posting at > http://blog.suretecsystems.com/archives/40-OpenLDAP-Weekly-News-Issue-5.html#extended > for multimaster N-Way sync. Very good stuff. > > I've configured the cn=config backend, and I can browse it with my > LDAP browser on both my Masters. (I have two servers) > > I have created the replication agreements and are able to add them to > the cn=config as documented in the URL above. No problem on both > servers. > > When I add the data sync, I get a little confused. > > Below, for ${BACKEND} I assume I put something like bdb for the > database backend correct? If I do this it does not fail, but the > sync > does not happen. I don't see a whole lot of errors either with the > sync.
bdb or hdb, depending on how you've built slapd.
Logs?
dn: olcDatabase={1}$BACKEND,cn=config objectClass: olcDatabaseConfig objectClass: olc${BACKEND}Config olcDatabase: {1}$BACKEND olcSuffix: $BASEDN olcDbDirectory: ./db olcRootDN: $MANAGERDN olcRootPW: $PASSWD olcSyncRepl: rid=004 provider=$URI1 binddn="$MANAGERDN" bindmethod=simple credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1 olcSyncRepl: rid=005 provider=$URI2 binddn="$MANAGERDN" bindmethod=simple credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1 olcSyncRepl: rid=006 provider=$URI3 binddn="$MANAGERDN" bindmethod=simple credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly interval=00:00:00:10 retry="5 5 300 5" timeout=1 olcMirrorMode: TRUE
dn: olcOverlay=syncprov,olcDatabase={1}${BACKEND},cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov
I end up with olcDatabase={1}bdb in there twice.
What should the $BACKEND value be if not bdb. (if by db backend is bdb).
Thanks for any insight. For now, I am going to have to revert to Master+Slave via syncrepl and referrals.
Sellers
| ----------------------------------------------------------------------| Chris G. Sellers, MLS Lead Internet Engineer National Institute for Technology & Liberal Education 535 West William Street, Ann Arbor, Michigan 48103 chris.sellers@nitle.org 734.661.2318
______________________________________________ Chris G. Sellers | NITLE Technology 734.661.2318 | chris.sellers@nitle.org AIM: imthewherd | GTalk: cgseller@gmail.com
openldap-software@openldap.org
-
Chris G. Sellers -
Chris G. Sellers
-
Gavin Henry -
Quanah Gibson-Mount