DupDetector - I ran it, but it crashed after 117 photos.

Image Comparitor - I ran it. Found more duplicates than I did. Output UI is a little obscure; I wish it offered to just saved the results to a CSV.

DupImages - A dead project, but it sounds like it may have used similar approach to Alastair’s suggestion.

Duplicate File Killer - Functionality still on the drawing board.

ImageSorter - Wrong OS. Didn’t look at.

I’d be surprised if Photoshop (and possibly PIL, PHP) didn’t preserve the EXIF data. Similarly I’d be surprised if Facebook did.

In this case, I was definitely focussed on the thrill of the hunt. However, only while it meant a pleasant horse-ride across the moors. Once the fox hid amongst the thorny brambles of wavelet theory and JPEG compression (neither of which I pretend to understand in the slightest) I got bored.

The correct thing to do now, is to head off to the open-source butcher and buy some prepared code.

No, actually, the correct thing to do is to stop torturing these poor, innocent analogies.

As for EXIF data, I don’t know if Photoshop, Facebook, PIL and PHP’s graphics library all preserve them correctly during transformations. I don’t know if VueScan populates them correctly in the first place. Maybe they all do it perfectly, but I don’t have much trust in them.

A simple histogram suffers from two problems: it is likely to be distorted by simple operations to adjust the brightness, and it doesn’t allow meaningful bit-wise comparisons - you need to check for “approximately the same” rather than “exactly the same”.

Yeah, implementing wavelets and other serious maths seem to be the way to go.

Even better would be re-using the code of others.

But that requires tedious theory. It’d be more fun to sort the photos by hand!

I am quite prepared to retro-actively remove your comments if you would like, but frankly I think they both add to the conversation, so I am loathe to do so.

OK so here’s my idea. I am not in any way a graphics expert, so try not to laugh too much.

Basically you want to re-compress each image at a known resolution (meaning pixel dimensions) and quantization. So pick an appropriate width and height, such as that of your largest image. Also pick a quantization matrix - this time probably you’ll need to use the most aggressively compressed image.

Now re-compress all your images. You *could* probably use the hash functions you mentioned above (particularly if you use the YCbCr domain instead of RGB) but I think there’s a better way, provided you are willing to get your hands dirty with JPEG internals.

Basically at the same resolution and quantization you should be able to use the DC coefficient as a rough proxy for that macroblock. So: just add the DC coefficients for all the macroblocks to produce a hash value for the channel/image as a whole.

In fact you don’t need to re-compress the entire image to get this value - just enough to determine the DC coefficient for each macroblock. This has the benefit of being a fair bit quicker and also requiring only a single quantization value and not an entire matrix. The maths up to this point seems fairly easy actually.

This gives you three hash values, one for each channel, and hence allows you to detect certain types of other transforms. As you say, you can just compare the Cb and Cr hashes to detect a change in brightness. Alternatively you could just look at the Y hashes to detect an image that has been converted to B&W.

In the “just get the job done” category of solutions: have you thought about doing a hash of (key fields in) the EXIF data? Or are you concerned about removal of such data as well?