Duplicate File Finder Engine

[david_werecat]

I'm not sure that this is the right place to put this, but I made a fast duplicate file finder and wanted to share it. It takes options from a configuration file and writes a report of all duplicates found. The different configuration commands are as follows:

verbose={0,1,2}      :  Sets the level of console verbosity
recurse={true,false} :  Sets the option to recurse into subdirectories
report={filename}    :  Sets the output filename for the report
filter={filter}      :  Sets the filter for finding files
alldirs={true,false} :  Sets the option to search all directories, not just the ones that match the search filter
AddPath {path}       :  Add a path to search for duplicates.  Multiple paths may be specified in the same configuration file

The report format should be fairly simple. Finding duplicate files occurs in several stages. First, all files from the given paths are enumerated. They are then sorted by size and divided into groups of the same size. Groups with only one item are discarded. Then, all files over 256kb have a 16 byte sample taken from the middle and are sorted and divided similarly to how they were divided by size. Afterwards, all remaining files are hashed using SHA-1 and the hashes are sorted to find duplicates.

The program is written in VB.net, so the .NET framework is required. File enumeration occurs through native API instead of using the framework classes to increase speed. If there are any errors, please let me know.

[roytam1]

Mine has one written in Perl/PHP.
http://rtfreesoft.blogspot.com/2010/...e-initial.html

[chornobyl]

I prefer this
http://www.sentex.net/~mwandel/finddupe

[david_werecat]

Good work roytam1! It's definitely a lot more portable than mine.

Thanks for the link for finddupe. I may also work on a utility that removes duplicates or replaces them with hardlinks given the report as input. The problem that I run into is choosing which file to keep from the list of duplicates. A GUI might be better suited for selecting which file to keep, but I don't think that most people have the time to look over a list of several thousand duplicates.

I also ran a benchmark on all three programs on the 'C:\cygwin' directory on my laptop, which contains 21832 files and 5192 directories. The total size of all files is 463,815,672 bytes. I ran 'dir /s C:\cygwin' before running the benchmark to ensure that all engines start on equal grounds. The tests were run in the following order:
1. FileSystemDedup.exe
2. finddupe.exe
3. finddup.pl

All times are the global times as measured by Timer 11.00. The results are as follows:

FileSystemDedup.exe      119 seconds
finddupe.exe             168 seconds
finddup.pl               245 seconds

I also ran benchmarks on the speed of a second sequential test of each program:

FileSystemDedup.exe      10 seconds
finddupe.exe             235 seconds
finddup.pl               34 seconds

However, being the author of one of the pieces of software in the benchmark, I suggest that other benchmarks are done in order to ensure the reliability of the results.

[suguruanata]

I've been using this so far

http://duplicatefilesdeleter.com/

but I guess I'll try your too!

[Karhunen]

I use http://www.joerg-rosenthal.com/en/an.../download.html for my Win32 box and its slow, but more useful for checking content duplicate pictures. Don't have access to a Win64 box, anyone who has such an OS might like to try it.

[david_werecat]

Thanks for the links. Both AntiTwin and DuplicateFilesDeleter have more robust multimedia deduplication engines, but are slower because of this. They're also more user friendly. I guess that the main difference between most deduplication programs is how they handle the tradeoff between features/user friendliness/speed.

It's difficult to benchmark GUI programs, so I'm not including them in the benchmark as of yet. However, if I find an accurate way to automate GUI benchmarking, I will include them.

[Piotr Tarsa]

Look at AutoHotKey.

[Menno de Ruiter]

Wrote one too in C#

with Parallel CPU 8 cores usage (order the input of files on filesize due to then it takes about the same time each parellel invoking
and 4 different offsets with 4 different outcome in the CRC extreme unique crc (16 bytes)


Parallel.Invoke
(
() =>
{
etc.

as well to take ownership of the file when ntfs security blocks reading a file


using (new ProcessPrivileges.PrivilegeEnabler(Process.GetCurr entProcess(), Privilege.TakeOwnership))
{
FileSecurity fSecurity = File.GetAccessControl(file);
fSecurity.AddAccessRule(new FileSystemAccessRule(@"WIN-EMU5IJJMFO6\Administrator", FileSystemRights.FullControl, AccessControlType.Allow));
File.SetAccessControl(file, fSecurity);
File.SetAttributes(file, fileAttributes);
System.Security.Principal.SecurityIdentifier sid = new System.Security.Principal.SecurityIdentifier(Syste m.Security.Principal.WellKnownSidType.LocalSystemS id, null);
System.Security.Principal.NTAccount acct = sid.Translate(typeof(System.Security.Principal.NTA ccount)) as System.Security.Principal.NTAccount;
string strEveryoneAccount = acct.ToString();
try
{
System.Security.AccessControl.FileSecurity sec = System.IO.File.GetAccessControl(file);
sec.AddAccessRule(new System.Security.AccessControl.FileSystemAccessRule (
sid,
System.Security.AccessControl.FileSystemRights.Ful lControl,
System.Security.AccessControl.AccessControlType.Al low));
File.SetAccessControl(file, sec);
}
catch (UnauthorizedAccessException)
{
// handle permissions problem
}
sid = new System.Security.Principal.SecurityIdentifier(Syste m.Security.Principal.WellKnownSidType.BuiltinUsers Sid, null);
acct = sid.Translate(typeof(System.Security.Principal.NTA ccount)) as System.Security.Principal.NTAccount;
strEveryoneAccount = acct.ToString();
try
{
System.Security.AccessControl.FileSecurity sec = System.IO.File.GetAccessControl(file);
sec.AddAccessRule(new System.Security.AccessControl.FileSystemAccessRule (
strEveryoneAccount,
System.Security.AccessControl.FileSystemRights.Rea dAndExecute,
System.Security.AccessControl.AccessControlType.Al low));
File.SetAccessControl(file, sec);
ok = true;
Released = Released + 1;
}
catch (UnauthorizedAccessException)
{
// handle permissions problem
}
}