Results 1 to 7 of 7

Thread: reproductible (compressed) archives

  1. #1
    Member
    Join Date
    Dec 2011
    Location
    France
    Posts
    1
    Thanks
    4
    Thanked 0 Times in 0 Posts

    reproductible (compressed) archives

    I'm interrested in reproductible compression :

    I have a file (zip or 7z), containing several files.

    When I remove a file fom it, add it again, it shouldn't change the file.
    When I add a file to an existing archive, or create again from all the (uncompressed) files, it should make the same file.

    It's used to share an archive, transmit patchs and ensuring everybody has the same file, no matter if they just got the complete last file or updated it from a previous version.

    Here's what I found that works, and can be extracted with any extractor for the original format:

    based on zip:
    https://sourceforge.net/projects/trrntzip/ torrentzip
    http://www.gotwalls.com/files/trrntzip-comment-fix.diff a patch to fix zip files with multiple comments
    https://github.com/arogl/trrntzipDN an alternative implementation in .NET

    based on 7zip:
    https://sourceforge.net/projects/t7z/ torrent7zip

    Is there any other/better archive encoder which can give the same guarantees ?
    How hard would it be to adapt it to other archivers with compression ?

  2. #2
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    I guess http://nishi.dreamhosters.com/u/shar_v2a.rar because it doesn't include any file attributes
    (no compression though, but you can compress the whole archive)

  3. The Following 2 Users Say Thank You to Shelwien For This Useful Post:

    Cyan (7th December 2016),sebbu (8th December 2016)

  4. #3
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    freearc should be able to create reproducible archives too. we just need to figure out how to suppress all its unnecessary activity
    -mt1 - disables unpredictable parallel processing
    --recompress - for archive update, ensure that all files are recompressed, in order to make new archive content independent on previous one
    --no-dates - drop filetimes from archive, if you need to not depend on them too

    of course, you also need to use the same program version and same setting files (such as arc.groups)

  5. The Following User Says Thank You to Bulat Ziganshin For This Useful Post:

    sebbu (8th December 2016)

  6. #4
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    @Bulat: what about solid blocks or something, to improve diffs?

  7. #5
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    what's wrong with them? you can use any solid/sorting/method/arc.groups/-ma settings as far as they are the same, and freearc version is the same.

  8. #6
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    I mean, what would be the best freearc settings for archive sync? Like, update the archive, then distribute the patches with torrent?

  9. #7
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    I'm not sure that there is a best solution for everyone. if you compose multiple files into solid block, you improve compression, but increase amount of data that should be sent on update

    one of reasons why i developed fa in the first time was my own need to keep large book library and efficiently update it and extract individual book files. so, freearc can be instructed to use 8MB block sizes with -s8m and the it will apply intelligence to updating the archive - i.e. it will keep each solid block in the 6..12 MB range, so f.e. adding one book to the library will require to recompress no more than a single solid block (i.e. 12 MB one became a bit larger and being split to 2 blocks 6 MB each)

    It's the smart mode, the program default. For reproducibel archive, you don't need that intelligence, so you should use --recompress option to disable it - this way, archive will be entirely recompressed on every update.

    But for reproducible splitting of files into solid blocks there is only one option: -se. It makes a separate solid block for every file extension. We can imagine other staregies of solid block splitting, such as by year, first letter of name and even first digit of its CRC, but freearc doesn't implement them.

    Interesting to note, though, that FA'Next becomes much more user-programmable with Lua, and it should be no much problem to add Lua-programmable solid-block splitting strategies. F.e. the compression mode for each solid block is already fully constructed by the Lua code.

    I will look into adding that right into 0.12. Thanks for idea. I still look for a ways to employ Lua programmibility for real needs, and that's yet another interesting scenario.

  10. The Following 4 Users Say Thank You to Bulat Ziganshin For This Useful Post:

    Mike (8th December 2016),RamiroCruzo (12th December 2016),sebbu (8th December 2016),Shelwien (8th December 2016)

Similar Threads

  1. ZPAQ self extracting archives
    By Matt Mahoney in forum Data Compression
    Replies: 31
    Last Post: 17th April 2014, 03:39
  2. Bittorrent and solid archives
    By lunaris in forum Data Compression
    Replies: 8
    Last Post: 29th December 2010, 11:54
  3. MS CAB archives
    By nanoflooder in forum Data Compression
    Replies: 0
    Last Post: 10th April 2010, 00:58
  4. Multi-Volume Archives
    By osmanturan in forum Data Compression
    Replies: 12
    Last Post: 13th June 2009, 01:46
  5. GUI for creation of 7z-SFX-archives
    By Vacon in forum Forum Archive
    Replies: 0
    Last Post: 8th June 2007, 15:16

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •