Howard Chu writes:
I had considered MD5 before (especially since we already had code for it) but it was slower, and we're not looking for cryptographic assurances or hash distribution anyway.
Yes, I was thinking mostly of killing unnecessary code and exposed features. After a brief test, I could see no good reason to keep CRC: Neither speed nor quality. On my machine, anyway.
Basically all of these crypto hash functions are overkill, in terms of hash size and computation. We're only looking to detect casual misuse or corruption, not malicious deception.
Yes. The main reason I see to use either CRC or MD5 is that they are likely to be installed somewhere so non-programmers can use them. We could use a faster hash like MurmurHash3, but would then need to provide a 'slaputil hash' command or something for use by shell programs.
I didn't really spend a lot of time comparing the two functions' speed. But even with the memory access bottleneck, I would guess that on a loaded system with many threads running, the algorithm with fewer instructions is the better choice. Have you measured the throughput when multiple threads are executing?
On my 32-bit host, MD5 on a threaded test program had 90-95% of CRC's throughput instead of 105% or whatever it was unthreaded. OTOH crc32() from linking -lz gave ~275%. OTOH I'd expect MD5 to be more costly on an older or cheaper machine where the which hasn't outpaced memory speed as much as modern workhorses.