Hi,
as I am in the situation where I have to do large imports with ldapadd, I wanted to share a little tip that is saving me a lot of time and may be helpful to others too.
As I don't care for data consistency *while* importing (if the import should abort, I can alway restart the process), I'm stopping the regular slapd and run it manually under the control of 'eatmydata', a little tool that LD_PRELOAD's all filesystem sync operations to noops. This has increased the speed of my import tests by several orders of magnitude.
Here's my script (slightly obfuscated):
systemctl stop slapd rm -f /srv/ldap/data/* systemd-run --unit=slapd-fastimport eatmydata /usr/sbin/slapd -d 0 -s 0 -h ldapi:/// -g openldap -u openldap -F /etc/ldap/slapd.d sleep 2 ldapadd -Y EXTERNAL -H ldapi:/// -Q -c -S errorlog -f data.ldif systemctl stop slapd-fastimport sleep 2 systemctl start slapd grep Error: errorlog | sort | uniq -c | sort -n
Best regards Karsten
Am Freitag, 13. Oktober 2017 15:18 CEST, Karsten Heymann karsten.heymann@gmail.com schrieb:
<....> As I don't care for data consistency *while* importing (if the import should abort, I can alway restart the process), I'm stopping the regular slapd and run it manually under the control of 'eatmydata', a little tool that LD_PRELOAD's all filesystem sync operations to noops. This has increased the speed of my import tests by several orders of magnitude.
This might be a stupid question or I don't undersand your goal, but if you shut down you operational server, why don't you use 'slapadd' (optionally with the '-q -s' option in case you know your input is consistent)?
Cheers, Ralf Mattes
Hi Ralf,
2017-10-13 15:43 GMT+02:00 Ralf Mattes rm@mh-freiburg.de:
This might be a stupid question or I don't undersand your goal, but if you shut down you operational server, why don't you use 'slapadd' (optionally with the '-q -s' option in case you know your input is consistent)?
That's not a stupid question at all, but there are several reasons:
- I like the way ldapadd logs complete entries it failed to import, that makes analyising the cause of the import error much simpler, after e.g. fixing the schema I can ldapdd these error files directly again to see if the fix works - I don't want to disable any import checks, the main reason I'm doning this imports is to hunt down import errors in subtrees from different sources. - I shortened the exaple a bit, in reality I'm adding multiple ldif files into the database during the script
But otherwise slapadd would be of course the better solution.
BR Karsten
openldap-technical@openldap.org
-
Karsten Heymann -
Ralf Mattes