Re: (ITS#8048) back-sock/slapo-sock issue
by michael＠stroeder.com
This is a multi-part message in MIME format.
--------------090704060507030702010600
Content-Type: text/plain; charset=ISO-8859-2
Content-Transfer-Encoding: quoted-printable
hyc(a)symas.com wrote:
> hyc(a)symas.com wrote:
>> michael(a)stroeder.com wrote:
>=20
>>> Can I raise the debug level? Log level trace does not really show the=
ori=3D
>>> ginal
>>> byte sequence received by the external listener.
>>
>> Currently there is no debug code for this.
>=20
> Actually, if you use debug level SHELL you'll get a plaintext dump of t=
he=20
> received text. If you direct this into a file you can examine it with a=
hex=20
> editor.
I tested this again running as overlay with -d SHELL. See attachment of t=
he
log excerpt as Python string representation read with open(file,'rb') bef=
ore.
You can find *exactly* the lines from this response string generated by t=
he
external listener:
'RESULT\nmsgid: 1\ncode: 49\ninfo: HOTP value wrong\n\n'
All lines starting with sock search reading line:
(RESULT\n)
(msgid: 1\n)
(code: 49\n)
(info: HOTP value wrong\n)
Excerpt of slapd's log repeated here for better readability in the ITS. P=
lease
note the extra space after "text=3D" which is not what the external liste=
ner
returned.
------------------------------------------------
5592d039 sock search reading line (RESULT
)
5592d039 sock search reading line (msgid: 1
)
5592d039 sock search reading line (code: 49
)
5592d039 sock search reading line (info: HOTP value wrong
)
5592d039 sock search reading line (
)
5592d039 str2result (msgid: 1
code: 49
info: HOTP value wrong
) unknown
5592d039 str2result (
) unknown
5592d039 conn=3D1000 op=3D0 RESULT tag=3D97 err=3D49 text=3D HOTP value w=
rong
------------------------------------------------
Ciao, Michael.
--------------090704060507030702010600
Content-Type: text/plain; charset=UTF-8;
name="string"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename="string"
NTU5MmQwMzkgY29ubj0xMDAwIGZkPTEyIEFDQ0VQVCBmcm9tIFBBVEg9L29wdC91aS92YXIv
b3BlbmxkYXAvcnVuL2xkYXBpLXAgKFBBVEg9L29wdC91aS92YXIvb3BlbmxkYXAvcnVuL2xk
YXBpLXApXG41NTkyZDAzOSBjb25uPTEwMDAgb3A9MCBCSU5EIGRuPSJ1aWQ9c2VvZmosY249
dGVzdCxvdT1leGFtcGxlIiBtZXRob2Q9MTI4XG41NTkyZDAzOSBjb25uPTEwMDEgZmQ9MTQg
QUNDRVBUIGZyb20gUEFUSD0vb3B0L3VpL3Zhci9vcGVubGRhcC9ydW4vbGRhcGktcCAoUEFU
SD0vb3B0L3VpL3Zhci9vcGVubGRhcC9ydW4vbGRhcGktcClcbjU1OTJkMDM5IGNvbm49MTAw
MSBvcD0wIEJJTkQgZG49IiIgbWV0aG9kPTE2M1xuNTU5MmQwMzkgY29ubj0xMDAxIG9wPTAg
QklORCBhdXRoY2lkPSJnaWROdW1iZXI9MTAwK3VpZE51bWJlcj0xMDAwLGNuPXBlZXJjcmVk
LGNuPWV4dGVybmFsLGNuPWF1dGgiIGF1dGh6aWQ9ImdpZE51bWJlcj0xMDArdWlkTnVtYmVy
PTEwMDAsY249cGVlcmNyZWQsY249ZXh0ZXJuYWwsY249YXV0aCJcbjU1OTJkMDM5IGNvbm49
MTAwMSBvcD0wIEJJTkQgZG49ImNuPXJvb3Qsb3U9ZXhhbXBsZSIgbWVjaD1FWFRFUk5BTCBz
YXNsX3NzZj0wIHNzZj0yNTZcbjU1OTJkMDM5IGNvbm49MTAwMSBvcD0wIFJFU1VMVCB0YWc9
OTcgZXJyPTAgdGV4dD1cbjU1OTJkMDM5IGNvbm49MTAwMSBvcD0xIFNSQ0ggYmFzZT0idWlk
PXNlb2ZqLGNuPXRlc3Qsb3U9ZXhhbXBsZSIgc2NvcGU9MCBkZXJlZj0wIGZpbHRlcj0iKCYo
b2JqZWN0Q2xhc3M9b2F0aEhPVFBVc2VyKShvYXRoSE9UUENvdW50ZXI9Kikob2F0aFNlY3Jl
dD0qKSh8KCEodWlOb3RCZWZvcmU9KikpKHVpTm90QmVmb3JlPD0yMDE1MDYzMDE3MjIwMVop
KSh8KCEodWlOb3RBZnRlcj0qKSkodWlOb3RBZnRlcj49MjAxNTA2MzAxNzIyMDFaKSkpIlxu
NTU5MmQwMzkgY29ubj0xMDAxIG9wPTEgU1JDSCBhdHRyPW9hdGhIT1RQQ291bnRlciBvYXRo
SE9UUFBhcmFtcyBvYXRoU2VjcmV0IG9hdGhUb2tlblNlcmlhbE51bWJlciBvYmplY3RDbGFz
cyBwd2RQb2xpY3lTdWJlbnRyeSB1c2VyUGFzc3dvcmRcbjU1OTJkMDM5IGNvbm49MTAwMSBv
cD0xIFNFQVJDSCBSRVNVTFQgdGFnPTEwMSBlcnI9MCBuZW50cmllcz0xIHRleHQ9XG41NTky
ZDAzOSBjb25uPTEwMDEgb3A9MiBTUkNIIGJhc2U9ImNuPW9hdGgtcG9saWN5LWhvdHAtdXNl
cnMsY249ZXhhbXBsZSxvdT1leGFtcGxlIiBzY29wZT0wIGRlcmVmPTAgZmlsdGVyPSIob2Jq
ZWN0Q2xhc3M9b2F0aEhPVFBQYXJhbXMpIlxuNTU5MmQwMzkgY29ubj0xMDAxIG9wPTIgU1JD
SCBhdHRyPW9hdGhIT1RQQ291bnRlck1heCBvYXRoSE9UUExvb2tBaGVhZCBvYXRoT1RQTGVu
Z3RoXG41NTkyZDAzOSBjb25uPTEwMDEgb3A9MiBTRUFSQ0ggUkVTVUxUIHRhZz0xMDEgZXJy
PTAgbmVudHJpZXM9MSB0ZXh0PVxuNTU5MmQwMzkgc29jayBzZWFyY2ggcmVhZGluZyBsaW5l
IChSRVNVTFRcbilcbjU1OTJkMDM5IHNvY2sgc2VhcmNoIHJlYWRpbmcgbGluZSAobXNnaWQ6
IDFcbilcbjU1OTJkMDM5IHNvY2sgc2VhcmNoIHJlYWRpbmcgbGluZSAoY29kZTogNDlcbilc
bjU1OTJkMDM5IHNvY2sgc2VhcmNoIHJlYWRpbmcgbGluZSAoaW5mbzogSE9UUCB2YWx1ZSB3
cm9uZ1xuKVxuNTU5MmQwMzkgc29jayBzZWFyY2ggcmVhZGluZyBsaW5lIChcbilcbjU1OTJk
MDM5IHN0cjJyZXN1bHQgKG1zZ2lkOiAxXG5jb2RlOiA0OVxuaW5mbzogSE9UUCB2YWx1ZSB3
cm9uZ1xuXG4pIHVua25vd25cbjU1OTJkMDM5IHN0cjJyZXN1bHQgKFxuKSB1bmtub3duXG41
NTkyZDAzOSBjb25uPTEwMDAgb3A9MCBSRVNVTFQgdGFnPTk3IGVycj00OSB0ZXh0PSBIT1RQ
IHZhbHVlIHdyb25nXG41NTkyZDAzOSBjb25uPTEwMDAgb3A9MSBVTkJJTkRcbjU1OTJkMDM5
IGNvbm49MTAwMCBmZD0xMiBjbG9zZWRcbg==
--------------090704060507030702010600--
8 years, 2 months
Re: (ITS#8181) LMDB page leaks etc when treating DBs as data
by h.b.furuseth＠usit.uio.no
On 30. juni 2015 14:52, hyc(a)symas.com wrote:
> We have already documented this. "Database names are kept as keys in the
> unnamed database."
"Hey, that may mean I can drop a DB without having to open it
and mess with LMDB's DBI handling! Let me try this..."
> It should be obvious that if you muck with the record of an
> existing key, things will change, therefore you should not muck with those keys.
Lots of people think differently than you or me.
We're not going to agree. Not on this, and not on what you said
in your next mail. I don't see the point of rehashing those
arguments yet again, and I didn't intend to. I was just ITSing a
commit before pushing, _after_ asking you about it. I thought.
--
Hallvard
8 years, 2 months
Re: (ITS#8181) LMDB page leaks etc when treating DBs as data
by hyc＠symas.com
Hallvard Breien Furuseth wrote:
> On 29/06/15 23:05, hyc(a)symas.com wrote:
>> Seriously, why aren't we just saying "don't do this" and moving on?
>
> OK, then document it. Maybe keeping it simple, call
> mixing mainDB data and named DBs a user error and warn this
> can break the DB.
>
>> There are
>> lots of stupid things you can do with software. It's a waste of time and
>> energy to prevent them all.
>
> We totally disagree, starting with what is stupid. But we knew
> that.
Ultimately it's redundant work. I.e., every restriction you place inside the
code still requires documentation, otherwise all you've done is changed "why
doesn't this work" into "why doesn't LMDB let me do what I tried to do?"
Simpler to only write the doc and leave the code pristine.
The library's purpose is to perform its intended uses efficiently. Misuses are
not our responsibility. Preventing misuse is the responsibility of the
documentation, it doesn't belong in the code.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
8 years, 2 months
Re: (ITS#8181) LMDB page leaks etc when treating DBs as data
by hyc＠symas.com
Hallvard Breien Furuseth wrote:
> On 29/06/15 23:05, hyc(a)symas.com wrote:
>> Seriously, why aren't we just saying "don't do this" and moving on?
>
> OK, then document it. Maybe keeping it simple, call
> mixing mainDB data and named DBs a user error and warn this
> can break the DB.
We have already documented this. "Database names are kept as keys in the
unnamed database." It should be obvious that if you muck with the record of an
existing key, things will change, therefore you should not muck with those keys.
>> There are
>> lots of stupid things you can do with software. It's a waste of time and
>> energy to prevent them all.
>
> We totally disagree, starting with what is stupid. But we knew
> that. That's I asked which parts of "mdb/bundle" to push and
> which you rejected, we must have misunderstood each other. I
> thought I was just ITSing this commit properly before pushing.
I just don't see this happening in practice. Most applications will never use
subDBs. It's not like some random person is going to come along and open an
arbitrary LMDB file created by some other application and start mucking with
it, without knowing the meaning of what's inside. Embedded database libraries
are used in specific contexts by their enclosing application. They don't just
get random ops thrown at them from 3rd party code. The designer of the
enclosing app will design a schema/whatever set of tables to use and that
scheme will be fixed for the life of the application.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
8 years, 2 months
Re: (ITS#8181) LMDB page leaks etc when treating DBs as data
by h.b.furuseth＠usit.uio.no
On 29/06/15 23:05, hyc(a)symas.com wrote:
> Seriously, why aren't we just saying "don't do this" and moving on?
OK, then document it. Maybe keeping it simple, call
mixing mainDB data and named DBs a user error and warn this
can break the DB.
> There are
> lots of stupid things you can do with software. It's a waste of time and
> energy to prevent them all.
We totally disagree, starting with what is stupid. But we knew
that. That's I asked which parts of "mdb/bundle" to push and
which you rejected, we must have misunderstood each other. I
thought I was just ITSing this commit properly before pushing.
8 years, 2 months
Re: (ITS#8048) back-sock/slapo-sock issue
by hyc＠symas.com
hyc(a)symas.com wrote:
> michael(a)stroeder.com wrote:
>> Can I raise the debug level? Log level trace does not really show the ori=
>> ginal
>> byte sequence received by the external listener.
>
> Currently there is no debug code for this.
Actually, if you use debug level SHELL you'll get a plaintext dump of the
received text. If you direct this into a file you can examine it with a hex
editor.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
8 years, 2 months
Re: (ITS#8048) back-sock/slapo-sock issue
by hyc＠symas.com
michael(a)stroeder.com wrote:
> Howard Chu wrote:
>> michael(a)stroeder.com wrote:
>>> michael(a)stroeder.com wrote:
>>>> Null-bytes really needed?
>>>
>>> Hmm, with null-bytes the wrong result code is returned to the LDAP cli=
> ent.
>>>
>>> Without null-bytes the correct result code is returned to the client b=
> ut slapd
>>> complains about unknown string:
>>>
>>> 'RESULT\nmsgid: 1\ncode: 49\nmatched: uid=3Dwuqww,cn=3Dampua,ou=3Dampu=
> a\ninfo:
>>> NOK\n\n'
>>>
>>> results in log message:
>>>
>>> 559174df str2result (msgid: 1
>>> code: 0
>>>
>>> ) unknown
>>> 559174df str2result (
>>> ) unknown
>> =20
>> I would almost suspect you were running on Windows. None of the behavio=
> r
>> you're reporting occurs for me using the searchexample.pl script.
>
> Your script does not return any real 'RESULT..' responses.
Sure it does:
print "dn: cn=test, dc=example, dc=com\n";
print "cn: test\n";
print "objectclass: cnobject\n";
print "\n";
print "RESULT\n";
print "code: 0\n";
print "info: answered by process $$\n";
And the result at the client:
violino:~/OD/hobj/tests> ../clients/tools/ldapsearch -x -H ldap://:9011 -b
dc=example,dc=com
# extended LDIF
#
# LDAPv3
# base <dc=example,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# test, example.com
dn: cn=test,dc=example,dc=com
cn: test
objectClass: cnobject
# search result
search: 2
result: 0 Success
text: answered by process 3077
# numResponses: 2
# numEntries: 1
Notice there are no embedded NUL characters. Only newlines.
> I also have less issues when just returning 'CONTINUE\n'.
>
>> Sounds like your problem is a python runtime configuration, not a slapd=
>
>> bug.
>
> Not on Windows and no run-time configuration involved.
>
> Look at the raw string representations I've posted. That string goes dire=
> ctly
> to SocketServer.BaseRequestHandler.request.sendall(response_str).
>
> Can I raise the debug level? Log level trace does not really show the ori=
> ginal
> byte sequence received by the external listener.
Currently there is no debug code for this.
At any rate, the code uses fgets(3) which stops reading at a newline
character. The fact that your string isn't being parsed correctly implies
either libc fgets is broken inside your slapd process (pretty unlikely) or
your socket sender isn't really sending newline characters.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
8 years, 2 months
Re: (ITS#8181) LMDB page leaks etc when treating DBs as data
by hyc＠symas.com
h.b.furuseth(a)usit.uio.no wrote:
> Full_Name: Hallvard B Furuseth
> Version: LMDB_0.9.15
> OS:
> URL:
> Submission from: (NULL) (81.191.45.5)
> Submitted by: hallvard
>
>
> The user can overwrite or delete a named DB record
> with normal data. This leaks the pages of the DB.
> Also the new non-DB can then be treated as a DB.
> A fix is coming (from my branch mdb/bundle).
Seriously, why aren't we just saying "don't do this" and moving on? There are
lots of stupid things you can do with software. It's a waste of time and
energy to prevent them all.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
8 years, 2 months
Re: (ITS#8048) back-sock/slapo-sock issue
by michael＠stroeder.com
Howard Chu wrote:
> michael(a)stroeder.com wrote:
>> michael(a)stroeder.com wrote:
>>> Null-bytes really needed?
>>
>> Hmm, with null-bytes the wrong result code is returned to the LDAP cli=
ent.
>>
>> Without null-bytes the correct result code is returned to the client b=
ut slapd
>> complains about unknown string:
>>
>> 'RESULT\nmsgid: 1\ncode: 49\nmatched: uid=3Dwuqww,cn=3Dampua,ou=3Dampu=
a\ninfo:
>> NOK\n\n'
>>
>> results in log message:
>>
>> 559174df str2result (msgid: 1
>> code: 0
>>
>> ) unknown
>> 559174df str2result (
>> ) unknown
>=20
> I would almost suspect you were running on Windows. None of the behavio=
r
> you're reporting occurs for me using the searchexample.pl script.
Your script does not return any real 'RESULT..' responses.
I also have less issues when just returning 'CONTINUE\n'.
> Sounds like your problem is a python runtime configuration, not a slapd=
> bug.
Not on Windows and no run-time configuration involved.
Look at the raw string representations I've posted. That string goes dire=
ctly
to SocketServer.BaseRequestHandler.request.sendall(response_str).
Can I raise the debug level? Log level trace does not really show the ori=
ginal
byte sequence received by the external listener.
Ciao, Michael.
8 years, 2 months
(ITS#8181) LMDB page leaks etc when treating DBs as data
by h.b.furuseth＠usit.uio.no
Full_Name: Hallvard B Furuseth
Version: LMDB_0.9.15
OS:
URL:
Submission from: (NULL) (81.191.45.5)
Submitted by: hallvard
The user can overwrite or delete a named DB record
with normal data. This leaks the pages of the DB.
Also the new non-DB can then be treated as a DB.
A fix is coming (from my branch mdb/bundle).
8 years, 2 months
