openldap-devel

openldap-devel@openldap.org

1 participants
1421 discussions

Fwd: [Patch] Allocating sparse files on Windows
by Colin LeMahieu 20 Apr '15

20 Apr '15

Apologies if this arrives twice, I sent it once two days ago before subscribing to the list and it hasn't appeared yet so I'm resending after subscribing. Memory mapping files on Windows requires the file size to match the size of the memory mapped region however in the current implementation this requires creating a file the full size of the environment which may be much larger than the actual amount of data in the database. This patch creates the database as a sparse file on Windows. Windows Explorer will report the file size as the size of the entire memory mapped region but the size on disk will be proportional to the size of committed data in the database. This can be checked by checking the "size on disk" property of the file. Let me know if this is posted incorrectly, otherwise feel free to modify the patch in whatever way is necessary for approval.

2 2

www.openldap.org not mobile-friendly
by Hallvard Breien Furuseth 15 Apr '15

15 Apr '15

It will drop in Google's ratings next week because it fails the <https://www.google.com/webmasters/tools/mobile-friendly/> tests. Maybe someone can fix the web pages up a bit? I'm too busy now. -- Hallvard

4 4

Fwd: Using sendfile with lmdb
by James Rouzier 03 Apr '15

03 Apr '15

Eric thank you I would have never considered that as an issue. I tried out your code and saw the problem. You saved me going down a rabbit hole. Many Thanks On Thu, Apr 2, 2015 at 5:33 PM, Eric Wong <e(a)80x24.org> wrote: > James Rouzier <rouzier(a)gmail.com> wrote: > > Hey Guys, > > > > I am working on a proof of concept of a simple key value store service > > using lmdb as the back end. > > I am using sendfile to avoid an extra copy. > > Be careful if you're overwriting any data in LMDB. > > zero-copy sendfile means any data queued up in the kernel will be > subject to modifications as the underlying file is updated, so your > client may se data it was not intended to see. > > I wanted to do the same thing as you last year but ended up writing a > patch to the sendfile manpage instead: > > > https://git.kernel.org/cgit/docs/man-pages/man-pages.git/patch?id=7b6a32997… >

1 0

Using sendfile with lmdb
by James Rouzier 02 Apr '15

02 Apr '15

Hey Guys, I am working on a proof of concept of a simple key value store service using lmdb as the back end. I am using sendfile to avoid an extra copy. I accomplished this by creating function to extract get the mmap ptr from the env so I can calculate the offset of data. I figured that function might be useful to the bigger community. If it does not fit in the bigger vision of lmdb it is fine not to include it. You will find the diff attached. Thank you James

2 1

slaptest-ldif -l <ldif> tool or slapadd option
by Hallvard Breien Furuseth 25 Mar '15

25 Mar '15

I'd like a slap tool which verifies an LDIF before I try to ldapadd/slapadd it. "slapadd -u -o value-check=yes" is fairly close. What does it fail to catch? I can think of: - Duplicate entries. - Missing entries (if the initial DB is expected to be empty). - Child entries before parents (OK for slapadd to at least back-<bdb,hdb,mdb>). - Issues which the tool can only catch if it opens the database, like attempts to add already-existing entries. I probably don't want to do that. - Issues which overlays like slapo-unique would reject. Can't do that, since the overlay won't have a non-empty DB to check against and slap tools do not use overlays anyway. Might special-case "unique" though, since the "duplicate entries" check will need uniqueness code anyway. -- Hallvard

4 4

questions about LMDB
by Shu, Xinxin 20 Mar '15

20 Mar '15

Hi list , Recently I read docs about lmdb , there are two sentences 1) readers do not block writers 2) writers do not block readers I can understand 'readers do not block writers' , but cannot understand the second one , can someone help explain , how lmdb achieve 'writers do not block readers', below is my understandings , please correct me if anything wrong. if the access pattern is write - read, since lmdb only support two version of data , when the write has been started but not committed , the concurrent read may read stale data since write has not been committed. Cheers, xinxin

2 4

version numbers for release candidates
by Michael Ströder 08 Mar '15

08 Mar '15

HI! Sometimes many interesting changes are solely in RE24 and releasing the next true release takes some time. During this period I'd love to see pre-release version numbers assigned to RE24. Especially in case Quanah sends out an inquiry for another RE24 test round something like currently 2.4.41a1 or 2.4.41b5 or 2.4.41rc42 could be assigned (see http://semver.org/). This would make it possible to package RPMs etc. of pre-releases with non-conflicting version numbers which get replace by the real release package later. And for test rounds everybody would test the same snapshot. What do you think? Ciao, Michael.

2 2

Enhancing back-sock to use JSON
by Dagobert Michelsen 08 Mar '15

08 Mar '15

Hi, I have made some enhancements to back-sock to use JSON for the passed data and JSON-RPC to map LDAP calls to method invocations. The function signatures of the JSON-RPC calls are modeled to be similar to the ones used in json2ldap (which does just the opposite direction to talk LDAP via JSON-RPC) [1]. The previous hand-crafted format passed on the socket was harder to parse and needed a manually built parser whereas now a standard library can be used. However, handling the JSON data structures now imposes an additional dependency to Jansson (a JSON access library in C) [2]. Jansson itself is leightweight and has no dependencies itself. Due to the limited use of back-sock and the enhanced ease of use I think it would be acceptable to add this dependency. I would be glad if the modification would be possible to be applied to OpenLDAP and I happily submit a patch. Best regards -- Dago [1] JSON2LDAP interface from JSON-RPC to LDAP http://connect2id.com/products/json2ldap/web-api#ldap-compare [2] Jansson, a C library for reading and writing JSON data structures http://www.digip.org/jansson/ -- "You don't become great by trying to be great, you become great by wanting to do something, and then doing it so hard that you become great in the process." - xkcd #896

4 8

RE24 testing call #3 (2.4.41), LMDB RE0.9 testing call #3 (0.9.15)
by Quanah Gibson-Mount 03 Mar '15

03 Mar '15

OpenLDAP 2.4.41 Engineering Fixed libldap double free of request during abandon (ITS#7967) Fixed libldap segfault in ldap_sync_initialize (ITS#8001) Fixed libldap ldif-wrap off by one error (ITS#8003) Fixed libldap handling of TLS in async mode (ITS#8022) Fixed libldap null pointer dereference (ITS#8028) Fixed libldap mutex handling with LDAP_OPT_SESSION_REFCNT (ITS#8050) Fixed slapd slapadd onetime leak with -w (ITS#8014) Fixed slapd syncrepl delta-mmr issue with overlays and slapd.conf (ITS#7976) Fixed slapd syncrepl mutex for cookie state (ITS#7968) Fixed slapd syncrepl memory leaks (ITS#8035) Fixed slapd syncrepl to free presentlist at end of refresh mode (ITS#8038) Fixed slapd syncrepl to streamline presentlist (ITS#8042) Fixed slapd segfault when using matched values control (ITS#8046) Fixed slapd-mdb minor case typo (ITS#8049) Fixed slapd-mdb one-level search (ITS#7975) Fixed slapd-mdb heap corruption (ITS#7965) Fixed slapd-mdb crash after deleting in-use schema (ITS#7995) Fixed slapd-mdb minor code cleanup (ITS#8011) Fixed slapd-mdb to return errors when using incorrect env flags (ITS#8016) Fixed slapd-mdb to correctly update search candidates (ITS#8036, ITS#7904) Fixed slapd-meta TLS initialization with ldaps URIs (ITS#8022) Fixed slapo-collect segfault (ITS#7797) Fixed slapo-constraint with 0 count constraint (ITS#7780,ITS#7781) Fixed slapo-deref with empty attribute list (ITS#8027) Fixed slapo-sock result parser for CONTINUE (ITS#8048) Fixed slapo-syncprov synprov_matchops usage of test_filter (ITS#8013) Fixed slapo-syncprov segfault on disconnect/abandon (ITS#5452,ITS#8012) Fixed slapo-syncprov memory leak (ITS#8039) Fixed slapo-syncprov segfault on disconnect/abandon (ITS#8043) Fixed slapo-unique enforcement of uniqueness with manageDSAit control (ITS#8057) Build Environment Fixed libdb detection with gcc 5.x (ITS#8056) Enhanced contrib modules build paths (ITS#7782) Fixed contrib/autogroup internal operation identity (ITS#8006) Fixed contrib/passwd/sha2 compiler warning (ITS#8000) Fixed contrib/noopsrch compiler warning (ITS#7998) Fixed contrib/dupent compiler warnings (ITS#7997) Test suite: Added vrFilter test (ITS#8046) Contrib Added pbkdf2 sha256 and sha512 schemes (ITS#7977) Fixed autogroup modification callback responses (ITS#6970) Documentation Added ldap_get_option(3) LDAP_FEATURE_INFO_VERSION information (ITS#8032) Added ldap_get_option(3) LDAP_OPT_API_INFO_VERSION information (ITS#8032) LMDB 0.9.15 Release Engineering Fix txn init (ITS#7961,#7987) Fix MDB_PREV_DUP (ITS#7955,#7671) Fix compact of empty env (ITS#7956) Added workaround for fdatasync bug in ext3fs Build Don't use -fPIC for static lib Update .gitignore (ITS#7952,#7953) Cleanup for "make test" (ITS#7841) Misc. Android/Windows cleanup Documentation Fix MDB_APPEND doc Clarify mdb_dbi_open doc Thanks! --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

3 5

Fwd: multiple sequential lmdb readers + spinning media = slow / thrashes?
by Matthew Moskewicz 27 Feb '15

27 Feb '15

warnings: new to list, first post, lmdb noob. i'm a caffe user: https://github.com/BVLC/caffe in one use case, caffe sequentially streams though >100GB lmdbs at a rate of ~30MB/s in blocks of about 40MB. however, if multiple caffe processes are reading the same lmdb (opened with MDB_RDONLY), read performance becomes limiting (i.e. the processes become IO bound), even though the disk has sufficient read bandwidth (say ~180MB/s). some of the relevant caffe lmdb code is here: https://github.com/BVLC/caffe/blob/master/src/caffe/util/db.cpp however, if i *both* 1) run blockdev --setra 65536 --setfra 65536 /dev/sdwhatever 2) modify lmdb to call posix_madvise(env->me_map, env->me_mapsize, POSIX_MADV_SEQUENTIAL); then i can get >1 reader to run without being IO limited. for (2), see https://github.com/moskewcz/scratch/tree/lmdb_seq_read_opt similarly, using a sequential read microbenchmark designed to model the caffe reads from here: https://github.com/moskewcz/boda/blob/master/src/lmdbif.cc if i run one reader, i get 180MB/s bandwidth. with two readers, but neither (1) nor (2) above, each gets ~30MB/s bandwidth. with (1) and (2) enabled, and two readers, each gets ~90MB/s bandwidth. any advice? mwm PS: backstory (skippable): caffe originally used LevelDB to get better read performance for sequentially loading sets of ~1M 227x227x3 raw images (~200GB data). typically processing time is ~2 hours for this data set size, yielding a read BW need of 30MB/s or so. it's not really clear if/why LevelDB was uses aside from the fact that the caffe author was a google intern at the time he wrote it, but anecdotally i think the claim is that reading the raw .jpgs had perf. issues, although it's unclear exactly what or why. i guess it was the usual story about not getting sequential reads without using LevelDB. they switched to lmdb a while back. <openldap-devel(a)openldap.org>

3 3

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel