On 02/07/2014 02:40 PM, Christopher Wood wrote:
On Fri, Feb 07, 2014 at 10:49:29AM -0700, Brent Bice wrote:
(SNIP)
I've got a few OpenLDAP instances that I use for writing log datato, so write performance is critical, but since I'm building it from log data, absitively, posolutely, guaranteed perfect DB consistency isn't. I can always replay log data to rebuild the DB if, say, I had a power outage, the UPS failed, the RAID write-cache failed, the planets aligned, and I lost data. :-)
Out of interest, what are you using this log data for, and have you tested how many reads you are getting?
In the recent past, I've setup a java script to log postfix/sendmail/cuda logs to OpenLDAP and some simple php scripts to query it. 'Makes it easier for junior admins and managerial types to be able to track how an email got from point A to point B. Say, an Exchange user sent an email to an internal list server - so it went from exchange to a postfix relay to the list server, then back to the postfix relays then to some recips on Exchange, some recips on other lists, some on departmental mail servers using sendmail, etc. I can search by to/from and/or date/time, find the email, then click on the message-ID to search by that and show the email every hop along the way as well as all the recipients who got it. Makes it faster to sort out those "I sent an email to list ABC and user XYZ didn't get it! Why not!" problems. The answer usually is "user XYZ did get it and here's the log showing it". :-)
I also recently started logging DHCP client hostnames, IPs, MAC Addresses, and (if the dhcp request came from our VPN hardware) username. That way when I'm sifting through snort/FireEye/PaloAlto logs and I see some IP with a dhcp hostname of "MyPC" I can quickly tell which user's home machine is infected with malware-du-jour. I can see who was on which IPs when.
Yeah, I coulda used MySQL or Postgres or something else. The first one (the relay logs) started off as a weekend project to edjimicate myself on the LDAP API in Java (or one of 'em). It proved useful enough we just kept it. And since I had that in place, adding on the vpn/dhcp stuff later was easy. I use the dds overlay to automagically throw away records older than X days.
For both of those, the number of writes per second we do is low - around 4 or 5 per second last I checked.
However, we have a lot of DNS servers in a lot of different geographies and I've thought about trying to centralize their logs. But the query logs can be substantial - a terabyte per region per day - more than I really want to shove over the WAN to a central spot. So it occurred to me one morning that I could leave the log data distributed, but centralize how I query it. I could have one LDAP server that had referrals to other LDAP servers, one per region, and have all the DNS servers in a given region log their queries to their local LDAP server. Then a simple php script can do one query against the root server and find any query handled by any DNS server in any region. (useful when handling an intrusion event, for instance, and you want to know every DNS query made by some system between certain dates/times).
But SGI sells HPC equipment (big storage too, btw - grin). So it's not unheard of for someone to spin up a big cluster in one location and generate thousands of DNS queries per second. So any sort of logging I do has to scale well or it just won't work. There's likely a better way, but this gave me a good excuse to try out OpenLDAP + mdb on xfs and to see if PHP's LDAP API would chase referrals. :-) I'll probably wind up using some tool to index the textual query logs and some way to search all the indexes on all the regional log servers with regex patterns instead or somethin'...
(scrolls back) Oh yeah... Reads... I haven't been paying close attention to the number of reads per second I've been getting as writes and deletes were the bottleneck I was curious about. But the last time I checked, I was getting something like 30k+ queries per second with 8 threads on one client. But this is with zero tuning of the filesystem options and with a really simple-minded bit of java - this shouldn't be taken as any sort of serious benchmark. I've learned that proper benchmarking is HARD and I only use the java tool for rough guesstimates (and comparing how different config options may improve performance - or not - in a relative sort of way).
Brent