dynlist vs memberof performance issues
by Mark Cairney
Hi,
I've been out the LDAP loop for a bit but the recent discussion of the
memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box,
removed the memberof elements from the database and replaced the
memberof overlay with dynlist the queries appear to work as expected but
are both a) slow and b) heavily CPU-intensive on the LDAP server.
2021-09-01T12:47:17.603513+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 fd=12 ACCEPT from IP=192.168.152.33:58738
(IP=129.215.17.9:636)
2021-09-01T12:47:17.687488+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 fd=12 TLS established tls_ssf=256 ssf=256
tls_proto=TLSv1.3 tls_cipher=TLS_AES_256_GCM_SHA384
2021-09-01T12:47:17.688032+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=0 SRCH base="" scope=0 deref=0
filter="(objectClass=*)"
2021-09-01T12:47:17.688470+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=0 SRCH attr=supportedSASLMechanisms
2021-09-01T12:47:17.688878+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=0 SEARCH RESULT tag=101 err=0 qtime=0.000014
etime=0.000214 nentries=1 text=
2021-09-01T12:47:17.811279+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=1 BIND dn="" method=163
2021-09-01T12:47:17.819249+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=1 RESULT tag=97 err=14 qtime=0.000030
etime=0.009084 text=SASL(0): successful result:
2021-09-01T12:47:17.908889+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=2 BIND dn="" method=163
2021-09-01T12:47:17.909836+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=2 RESULT tag=97 err=14 qtime=0.000031
etime=0.000181 text=SASL(0): successful result:
2021-09-01T12:47:17.938839+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 BIND dn="" method=163
2021-09-01T12:47:17.939621+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 BIND authcid="mcairney(a)EASE.ED.AC.UK"
authzid="mcairney(a)EASE.ED.AC.UK"
2021-09-01T12:47:17.940213+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 BIND
dn="uid=mcairney,ou=people,ou=central,dc=authorise-dev,dc=ed,dc=ac,dc=uk"
mech=GSSAPI bind_ssf=256 ssf=256
2021-09-01T12:47:17.940616+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 RESULT tag=97 err=0 qtime=0.000024
etime=0.000409 text=
2021-09-01T12:47:18.227342+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=4 SRCH
base="dc=authorise-dev,dc=ed,dc=ac,dc=uk" scope=2 deref=0
filter="(uid=mcairney)"
2021-09-01T12:47:18.227703+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=4 SRCH attr=* +
2021-09-01T12:47:31.392255+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=5 UNBIND
2021-09-01T12:47:31.460705+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=4 SEARCH RESULT tag=101 err=0 qtime=0.000031
etime=13.233679 nentries=1 text=
2021-09-01T12:47:31.461098+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 fd=12 closed
I'm guessing that as the values are computed that this will be heavier
on the CPU but it seems a bit excessive? Has anyone else noticed any
similar performance issues?
This is a relatively low-specced DEV server (2 vCPUs, 4GB RAM) so I
guess this could be a factor but there's no io waiting on the server and
no swapping?
The database is on a par in size with our Production service ( about
400K user objects with 1 group object per user and then about 80K actual
groups of users)
The config for the primary DB (ACLs and rootPW redacted) is:
dn: olcDatabase={1}mdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcMdbConfig
olcDatabase: {1}mdb
olcDbDirectory: /opt/openldap/var/openldap-data/authorise
olcSuffix: dc=authorise-dev,dc=ed,dc=ac,dc=uk
olcAddContentAcl: FALSE
olcLastMod: TRUE
olcMaxDerefDepth: 2
olcReadOnly: FALSE
olcSecurity: ssf=1
olcSecurity: update_ssf=112
olcSecurity: simple_bind=64
olcSizeLimit: unlimited
olcSyncUseSubentry: FALSE
olcTimeLimit: unlimited
olcMonitoring: TRUE
olcDbEnvFlags: writemap
olcDbEnvFlags: nometasync
olcDbNoSync: FALSE
olcDbIndex: objectClass eq
olcDbIndex: entryUUID eq
olcDbIndex: entryCSN eq
olcDbIndex: cn pres,eq,sub
olcDbIndex: uid pres,eq,sub
olcDbIndex: uidNumber pres,eq
olcDbIndex: gidNumber pres,eq
olcDbIndex: eduniType eq
olcDbIndex: gecos pres,eq,sub
olcDbIndex: eduniCategory eq
olcDbIndex: mail pres,eq,sub
olcDbIndex: eduniSchoolCode eq
olcDbIndex: eduniIDStatus eq
olcDbIndex: eduniCollegeCode eq
olcDbIndex: eduniOrgCode eq
olcDbIndex: memberOf pres,eq
olcDbIndex: eduniLibraryBarcode pres,eq
olcDbIndex: eduniOrganisation pres,eq,sub
olcDbIndex: eduniServiceCode pres,eq
olcDbIndex: krbName pres,eq
olcDbIndex: eduPersonAffiliation pres,eq
olcDbIndex: eduPersonEntitlement pres,eq
olcDbIndex: sn pres,eq,sub
olcDbIndex: eduniIdmsId pres,eq
olcDbIndex: member pres,eq
olcDbIndex: memberUid pres,eq
olcDbIndex: eduniRefNo pres,eq
olcDbIndex: eduniTitle pres,eq
olcDbIndex: title pres,eq,sub
olcDbIndex: eduniCardNumber pres,eq
olcDbIndex: eduniYearOfStudy eq
olcDbIndex: description pres,eq,sub
olcDbIndex: givenName pres,eq,sub
olcDbIndex: aliasedObjectName eq
olcDbIndex: yubiKeyId pres,eq
olcDbIndex: isMemberOf pres,eq
olcDbIndex: hasMember pres,eq
olcDbIndex: proxyAddresses pres,eq,sub
olcDbMaxReaders: 96
olcDbMaxSize: 32212254720
olcDbMode: 0600
olcDbSearchStack: 16
structuralObjectClass: olcMdbConfig
dn: olcOverlay={0}syncprov,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcConfig
objectClass: top
objectClass: olcSyncProvConfig
olcOverlay: {0}syncprov
structuralObjectClass: olcSyncProvConfig
dn: olcOverlay={1}accesslog,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcAccessLogConfig
olcOverlay: {1}accesslog
olcAccessLogDB: cn=accesslog
olcAccessLogOps: writes
olcAccessLogPurge: 02+00:00 00+04:00
olcAccessLogSuccess: TRUE
structuralObjectClass: olcAccessLogConfig
dn: olcOverlay={2}dynlist,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcDynListConfig
olcOverlay: {2}dynlist
olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames
structuralObjectClass: olcDynListConfig
--
/****************************
Mark Cairney
ITI Enterprise Services
Information Services
University of Edinburgh
Tel: 0131 650 6565
Email: Mark.Cairney(a)ed.ac.uk
*******************************/
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
4 months, 2 weeks
slow memberOf queries in 2.5 with dynlist overlay
by Paul B. Henson
I'm still trying to figure out why my servers sometimes get into a state
where queries requesting the memberOf attribute take an exceedingly long
time to process, for example:
Feb 13 19:46:05 ldap-02 slapd[13564]: conn=297643 fd=104 ACCEPT from PATH=/var/symas/run/ldapi (PATH=/var/symas/run/ldapi)
Feb 13 19:46:05 ldap-02 slapd[13564]: conn=297643 op=0 BIND dn="cn=ldaproot,dc=cpp,dc=edu" method=128
Feb 13 19:46:05 ldap-02 slapd[13564]: conn=297643 op=0 BIND dn="cn=ldaproot,dc=cpp,dc=edu" mech=SIMPLE bind_ssf=0 ssf=71
Feb 13 19:46:05 ldap-02 slapd[13564]: conn=297643 op=0 RESULT tag=97 err=0 qtime=0.000006 etime=0.000050 text=
Feb 13 19:46:05 ldap-02 slapd[13564]: conn=297643 op=1 SRCH base="dc=cpp,dc=edu" scope=2 deref=0 filter="(uid=henson)"
Feb 13 19:46:05 ldap-02 slapd[13564]: conn=297643 op=1 SRCH attr=memberOf
Feb 13 19:46:42 ldap-02 slapd[13564]: conn=297643 op=1 SEARCH RESULT tag=101 err=0 qtime=0.000012 etime=36.955710 nentries=1 text=
How is the qtime calculated? It is nice and short, despite the wall clock
reading over 30 seconds :(.
Usually I have to reboot the server completely to clear this up, but this
time I just had to restart it, and then the queries were lickity split
again:
Feb 13 19:55:01 ldap-02 slapd[89655]: conn=1556 fd=40 ACCEPT from PATH=/var/symas/run/ldapi (PATH=/var/symas/run/ldapi)
Feb 13 19:55:01 ldap-02 slapd[89655]: conn=1556 op=0 BIND dn="cn=ldaproot,dc=cpp,dc=edu" method=128
Feb 13 19:55:01 ldap-02 slapd[89655]: conn=1556 op=0 BIND dn="cn=ldaproot,dc=cpp,dc=edu" mech=SIMPLE bind_ssf=0 ssf=71
Feb 13 19:55:01 ldap-02 slapd[89655]: conn=1556 op=0 RESULT tag=97 err=0 qtime=0.000009 etime=0.000088 text=
Feb 13 19:55:01 ldap-02 slapd[89655]: conn=1556 op=1 SRCH base="dc=cpp,dc=edu" scope=2 deref=0 filter="(uid=henson)"
Feb 13 19:55:01 ldap-02 slapd[89655]: conn=1556 op=1 SRCH attr=memberOf
Feb 13 19:55:01 ldap-02 slapd[89655]: conn=1556 op=1 SEARCH RESULT tag=101 err=0 qtime=0.000013 etime=0.213149 nentries=1 text=
Over 30 seconds elapsed time to .2 seconds? I'd like to see the .2 all
the time :).
When the server gets like this, there's a very high read IO load, 200-300Mb/s,
compared to generally less than 1Mb/s when things are working right.
It almost seems like it's doing a full disk scan searching for members
every time.
Any suggestions on what to dig into?
Thanks...
1 year
Official symas docker image? Or what others do?
by Dave Macias
Hello,
Wondering if there is a plan to do an official image.
Also what others in the community doing?
Finally, there is this project that seems to be updated/supported but obviously not official https://github.com/bitnami/bitnami-docker-openldap anyone here used/seen this?
Why am i asking?
We simply looking to dockerize our infra and want to see what people here have experienced.
Our ldap env:
3-way MMR (syncrepl) on v2.6.1
Thank you,
Dave
1 year, 2 months
Delta-sync replication: is it possible to force resync delta?
by skeletor
Hi.
I use delta-sync replication on version 2.4. Sometimes, some records
don't send to slave. Insofar as this is delta-sync after a new update
slave receive only last update. Therefore slave and master are not
consistent. Is it possible run force resync from accesslog to consistent
check if all records are present on slave?
May be this is a bug on version 2.4 and it already has been fixed in
newer version?
master 2.4.57
slave 2.4.55
1 year, 2 months
Problem with ppolicy overlay and userPassword attribute
by Frederic Dussurget
Hi,
We're facing the following issue :
* From one side, we have to store two values (same password with two different encodings) within the userPassword attribute (eg. https://datatracker.ietf.org/doc/html/rfc4519#section-2.41)
* On the other side, we do have to use the ppolicy overlay in order to hash to Argon2 automatically passwords that are sent in cleartext, in order to to use the pwdLastSuccess attribute (storing last authentication date) and pwdAccountLockedTime (account deactivation), but this overlay does not seem to support multiple values in the userPassword attribute as it looks to be described in the following abstract : https://datatracker.ietf.org/doc/html/draft-behera-ldap-password-policy-1... and https://www.openldap.org/software/man.cgi?query=slapo-ppolicy&sektion=5&a...
We're not able to add a user account with two values in the userPassword attribute :
[root@openldap25 ~]# cat testuser_add.ldif
dn: uid=testuser,ou=people,dc=my-domain,dc=fr
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: top
uid: testuser
cn: testuserCN
sn: testuserSN
userPassword: password
userPassword: drowssap
[root@openldap25 ~]# ldapadd -H ldaps://localhost:636 -x -D "cn=Manager,dc=my-domain,dc=fr" -W -f testuser_add.ldif
Enter LDAP Password:
adding new entry "uid=testuser,ou=people,dc=my-domain,dc=fr"
ldap_add: Constraint violation (19)
additional info: Password policy only allows one password value
However, we're able to add a user with a single value in the userPassword attribute, then, adding a second value thru the next LDAP request :
[root@openldap25 ~]# cat testuser_add.ldif
dn: uid=testuser,ou=people,dc=my-domain,dc=fr
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: top
uid: testuser
cn: testuserCN
sn: testuserSN
userPassword : password
[root@openldap25 ~]# ldapadd -H ldaps://localhost:636 -x -D "cn=Manager,dc=my-domain,dc=fr" -W -f testuser_add.ldif
Enter LDAP Password:
adding new entry "uid=testuser,ou=people,dc=my-domain,dc=fr"
[root@openldap25 ~]# cat testuser_mod.ldif
dn: uid=testuser,ou=people,dc=my-domain,dc=fr
changetype: modify
add: userPassword
userPassword: drowssap
[root@openldap25 ~]# ldapmodify -H ldaps://localhost:636 -x -D "cn=Manager,dc=my-domain,dc=fr" -W -f testuser_mod.ldif
Enter LDAP Password:
modifying entry "uid=testuser,ou=people,dc=my-domain,dc=fr"
[root@openldap25 ~]# ldapsearch -x -H ldaps://localhost:636 -D "cn=Manager,dc=my-domain,dc=fr" -W -LLL -b "ou=people,dc=my-domain,dc=fr" "(uid=testuser)" userPassword
Enter LDAP Password:
dn: uid=testuser,ou=people,dc=my-domain,dc=fr
userPassword:: e0FSR09OMn0kYXJnb24yaWQkdj0xOSRtPTY1NTM2LHQ9MixwPTEkZjlPc3NtRFZ
DWmlzQk1IVEhWczRMZyRsMVAvbWdEdTA1bEpBc2pxcVF6aERYaENMV1BudnQyeDlRTDdweXFnVDFV
userPassword:: e0FSR09OMn0kYXJnb24yaWQkdj0xOSRtPTY1NTM2LHQ9MixwPTEkVk5ST1FMZFp
2Q0ptajNIcHQxYWtZdyRRcjJDYUpKSjZSWlhvZjFicDJmNGNOaGlRa3E5czlCU0FTbEtVNFoxYjBj
Considering configuration, we're running an OpenLDAP 2.5.7 server (LTB project) on a RHEL8 OS.
* ppolicy overlay
dn: olcOverlay={0}ppolicy,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcPPolicyConfig
olcOverlay: {0}ppolicy
olcPPolicyHashCleartext: TRUE
olcPPolicyUseLockout: FALSE
olcPPolicyForwardUpdates: FALSE
olcPPolicyDisableWrite: FALSE
olcPPolicySendNetscapeControls: FALSE
olcPPolicyDefault: cn=default,ou=ppolicies,dc=my-domain,dc=fr
* default password policy
dn: ou=ppolicies,dc=my-domain,dc=fr
objectClass: organizationalUnit
objectClass: top
ou: ppolicies
dn: cn=default,ou=ppolicies,dc=my-domain,dc=fr
objectClass: pwdPolicy
objectClass: organizationalRole
cn: default
pwdAttribute: userPassword
pwdLockout: TRUE
* password storage
dn: olcDatabase={-1}frontend,cn=config
olcPasswordHash: {ARGON2}
Question :
* Is that behaviour normal ?
* May we keep on leaning on this behaviour without fearing that this ability of adding a second value to the userPassword attribute will be revoked in the future versions/fix/patches of the service ?
Thank you in advance for your assistance.
Best regards,
Frédéric Dussurget / Maxime Schmutz
Université Lumière Lyon 2
1 year, 3 months
LMDB storage layout and page structure
by Ivan Unknown
Hello!
I have been looking at the LMDB project code trying to learn and understand
how a database could be implemented; however, I have been struggling to
answer the questions below for quite some time, partially due to my limited
knowledge of C:
- How does LMDB store keys and values in the page? I have learned that a
page consists of the header, slots, and key-value pairs but how does the
database handle keys and values that are too large to fit within a page?
- I found that LMDB does not keep a fixed number of keys per page, so that
would depend on the key-value pairs' sizes already inserted into the page.
Is this correct? Does it mean that B+tree pages (branch or leaf) could have
a different number of keys depending on the key size? How does it affect
performance or implementation of the B+tree?
- Are there any limitations on the size of a key? Can it be of an arbitrary
length?
- What is the difference between IS_LEAF and IS_LEAF2 flags in the page
header? What is the difference between these pages?
- How do overflow pages work in LMDB? From what I could understand, if a
key or a value does not fit in the page, it will be stored in the overflow
page (the entire page is allocated for that specific key or value). Is this
correct? What happens when the key size is several times larger than the
page size, e.g. 1MB value with 4KB pages?
- What is a sub-page in LMDB (F_SUBDATA)? How does it work?
I would greatly appreciate it if someone could share links to the
documentation that covers internals of the database, online videos,
research papers, mailing lists, or any notes you could share to help me
understand the above. Thank you very much!
Cheers,
Ivan
1 year, 3 months
Password policies and hashed passwords
by Felix Natter
Dear openldap-technical users,
my password policies (openldap 2.5.11) are not enforced and Roland
Gruber (author of LAM (Pro)) kindly advised me that passwords must be
stored in plaintext (Hash=PLAIN) in order to be able to enforce password
minimal length, password quality etc (i.e. when using passwd(1) on Linux
or an LDAP client on Windows).
Currently we are storing passwords as base64 encoded (::*) Salted SHA1
hashes ("{SSHA}*") (according to slapcat -n 1).
tl;dr: For enforcing password policies, what is the role of
"password-hash:PLAIN"(?) and olcPPolicyHashCleartext: TRUE (applied to
dn: olcOverlay=ppolicy,olcDatabase={1}mdb,cn=config) and what is
the security implication of these changes (we are using starttls on
linux / ssl on windows with a self-signed certificate in intranet)?
- Is it (necessary and) enough to switch to "password-hash:PLAIN" to be
able to enforce password policies? Does olcPPolicyHashCleartext: TRUE
[alone] help as written in this post [1]?
[1] https://www.openldap.org/lists/openldap-technical/201708/msg00024.html
EDIT: I think that olcPPolicyHashCleartext==ppolicy_hash_cleartext
(one is OLC, one is trad. config)? -> can we update the documentation [2]?
[2] https://www.openldap.org/doc/admin25/guide.html#Password%20Policies
- I am worried about security when storing/transferring pwds in
plain text (we are using starttls on linux / ssl on windows with a
self-signed certificate in intranet) [3]. Will
"ppolicy_hash_cleartext" [2] help with this?
[3] The manual states "Unfortunately, as dictionary and brute force
attacks are generally quite easy for attackers to successfully mount,
this advantage is marginal at best (this is why all modern Unix systems
use shadow password files)."
Many Thanks and Best Regards,
Felix
--
Felix Natter
1 year, 3 months
ldap_start_tls_s fails
by James Gordon
Hi,
I have just installed symas openldap v2.6.
Everything seems to be running ok, except that I cannot get C interface
ldap_start_tls_s() to work.
If I do something like this the program works fine:
ldap_initialize(&ld, HOST));
rc = ldap_simple_bind_s(ld, BASEDN, BASEPWD);
ldap_unbind(ld);
However, if I do something like this the program fails with ldap error
string "local error":
ldap_initialize(&ld, HOST));
ldap_set_option(ld, LDAP_OPT_X_TLS_CACERTFILE);
ldap_start_tls_s(ld, NULL, NULL);
rc = ldap_simple_bind_s(ld, BASEDN, BASEPWD);
ldap_unbind(ld);
From the command line, TLS seems to work fine:
The following works ok:
ldapsearch -H ldap://ldap.domain.com:389 -D "cn=admin,dc=domain,dc=com" -w
secret -b “ou=users,dc=domain,dc=com” -ZZ
This also works ok from a different server
openssl s_client -verify 10 -starttls ldap -showcerts -connect
ldap.domain.com:389 -CApath /etc/ssl/certs
(verification = ok):
However, if I omit the CApath it fails, not sure if that is a clue to the
problem:
openssl s_client -verify 10 -starttls ldap -showcerts -connect
ldap.domain.com:389
(Verification error: unable to get local issuer certificate).
Any help would be appreciated.
If this is the wrong list, let me know.
ldapsearch -H ldap://ldap.red0rb.com:389 -D "cn=admin,dc=red0rb,dc=com" -w
MdKlUIGYm0o63HxQ0RWYuKWkRkgr3Ohy -b “ou=users,dc=red0rb,dc=com” -ZZ
1 year, 3 months
Re: 2.5 Quick-Start guide roadblock: ldap_bind: Confidentiality required (13)
by Ben Poliakoff
>
> Steps 1-7 involve building the software, which I would not expect you to
> be
> doing if you're using Symas' packages?
>
You are, of course, correct about my having not compiled slapd, I misspoke.
> In any case, I have no idea what changes you made to the original LDIF. I
> have no such issue doing only the changes advised in the quickstart guide:
>
I sorted out what was happening. I was using the ldapadd command syntax
from the Quick-Start guide, which doesn't use the '-H ldap:///' parameter,
so ldapadd was picking up defaults from /etc/ldap/ldap.conf and happily
talking to our existing production slapd, running on a different server
(which of course does require secure binds).
However, when I added '-H ldap:///' to the ldapadd command, ensuring that
the ldap traffic was now going to the correct server (headpalm), the
command fails with "ldap_bind: Invalid credentials (49)".
At this point the olcRootPW in my slapd.ldif is the default ("secret"), and
I can see that the base64 encoded olcRootPW in
"slapd.d/cn=config/olcDatabase={1}mdb.ldif" is "olcRootPW:: c2VjcmV0"
(which is, in fact the base64 encoding for "secret").
Does anyone have debugging approaches that might help me sort out why slapd
isn't happy with the password? Also with regard to this mailing
list's protocol, would it be better to ask this question in a separate
thread?
(It's humbling to be asking such phenomenally basic questions, having built
and managed our existing openldap servers for many many years.)
Ben
1 year, 3 months
