I am a newbie to the compression world and to this forum as well.
I am looking for ways to do fast and smart compression and decompression for one of my datasets so I can quickly transfer data between a computer and a network share. My dataset constitutes a mix of large (multiple files ranging between 50-100MB) and a number of small sized files. The number of small files are at least ~5 times more than the large files. The data changes but the delta of the changes is fairly small, although the file metadata i.e., file last modified/accessed time, for most of the files gets periodically updated.
Efficient compression is not my goal but I am looking for ways to:
(1) create an archive of a given directory while compressing as quickly as possible without recompressing already compressed data (i.e., files/data that got compressed but haven't been modified).
(2) extract an archive to a given directory by decompressing the files that got modified but deleting any additional files that may exist in source (in a mirror mode)
The following is my scenario:
(1) Compressed archive, say foo.archive gets created locally on computerA and is copied to network share
(2) foo.archive gets copied from network to compA and extracted to the target directory
When step (1) repeats, I am wondering if there is a utility that updates an existing foo.archive with the latest data in a given directory by not compressing already compressed data (in a way, deduplication). When step (2) repeats, the utility should not bother decompressing data that is on the disk and has not been modified but additionally mimics the files/folders in the archive (i.e., delete any additional files in the target directory).
Is there an existing utility that meets the above requirement in full or partially? I would greatly appreciate any suggestions.