@AC and @Hugh McIntyre

I understand the probabilities very well. What you both don't seem to realise is that *any* probability of collision > 0 for hard drive data means that it is unacceptable for use as a method of deduplicating data.

You both seem to be arguing "it is so unlikely and so rare, it'll never happen" - but what if that block happens to match something crucial, from something in an .mp3 file to your company's core database - or what about your bank account? A dedupe collision in any of those will cause pain from "this song won't play anymore" to "what do you mean my bank account is empty?". You won't think it is acceptable then.

Again, I repeat, all hashes by definition have collisions (you can't fit 1 billion bits into 1 million spaces - mathematically you have a minimum of 1000 possible inputs for each hash assuming a equispaced distribution, it gets worse if not euqispaced). Regardless of how likely a collision is, you can't use a hash for anything more than indicating that there MIGHT be a duplicate block and to act accordingly.