Deduplication Libraries

This package provides functionality to make use of hashing algorithms that are particularly good at finding exact duplicates as well as convolutional neural networks which are also adept at finding n…

rmlint finds space waste and other broken things on your filesystem and offers to remove it. Finds… It runs and compiles under most Unices, including Linux, FreeBSD and Darwin. The main target is Lin…

In the first phase the elasticsearch backend is checked against the current filesystem state, cleaning up database entries of files that no longer exist. This will speed up queries made later on. Alt…

Tool for managing data-deduplication within extant compressed archive files, with a heavy focus on Manga/Comic-book archive files. It maintains a database of hashes for all files it scans, and actual…

There's gonna be a lot of duplicates in almost any arbitrary collection ofimages, and it can actually be surprising how many. pHash lib, which is the core of the tool, easily detects cropped and reto…