Results 1 to 1 of 1

Thread: Preflate - An open source recompressor for non-zlib deflate.

  1. #1
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts

    Preflate - An open source recompressor for non-zlib deflate.

    Hello people! There is a software I think everybody should know. It's called "preflate", and it is a library that can re-compress non-zlib deflate streams, like those found on optimized pngs or zip files created with 7z. It's an open source alternative to reflate from Eugene Shelwien but in my very biased opinion, it's already better. The author is Dirk Steinke (NOT ME!)


    Currently, there is a test version which uses the parser from precomp so it's shipped within it. There is some chat about it on Fairytale gitter if you want to check the conversation. I'm opening this thread so we can issue bug reports and share test results openly here in the forum, because I think it's more suitable for the task (gitter is for chat, not technical reports).


    So I'm going to start with two (possible) bugs:




    1) It appears that the preflate changes tampered with the zip header recognition somehow in precomp. I cannot reproduce the issue on every zip but I managed to narrow it down to a subset of files consisting of .docx documents exported from Google Docs and documents on the ODF format saved by LibreOffice. Maybe that info will help you to reproduce the problem. .jar java applications are also affected, but I don't know with what software they were created. Usually the streams are at last recognized in brute mode but should it be off, no recompression happens whatsoever, while normal precomp doesn't need to have -brute active to process them.


    This is the output of both for a bundle of 18 of said files from my docs folder on a tar archive; each program ran with the setting "-cn -t-nj" to avoid png and jpg recognition:


    Precomp:
    Code:
    Recompressed streams: 281/283
    ZIP streams: 281/283
    Preflate:
    Code:
    Recompressed streams: 14/14
    ZIP streams: 14/14
    This is preflate for a folder with +2000 .jar and a few .zip. Notice the huge amount of 'brute mode streams' compared to the little amount of 'zip streams':
    Code:
    Recompressed streams: 37912/37947
    PDF streams: 24/24
    ZIP streams: 4191/4191
    PNG streams: 2961/2961
    PNG streams (multi): 115/115
    GIF streams: 591/626
    JPG streams: 14/14
    zLib streams (intense mode): 1353/1353
    Brute mode streams: 28663/28663
    The second issue is probably related to precomp itself rather than the preflate library but I'll include it in here for the sake of simplicity. It's about the heuristics on intense and brute mode. Precomp still 'recognize' deflate streams where there aren't any. The vanilla version usually cannot process them correctly but preflate will make them pass as valid. The problem with this is that non-deflate files takes way too long to process. Here is an example:


    Preflate:
    Code:
    Input file: /media/gonzalo/rootMX16/home/gonzalo/Fairytale/TestBed/Genetics/Acanthaster_planci.xwrt.lzp.clz_1m
    Output file: /media/gonzalo/rootMX16/home/gonzalo/Fairytale/TestBed/Genetics/Acanthaster_planci.xwrt.lzp.clz_1m.pre
    
    
    Using packJPG for JPG recompression, packMP3 for MP3 recompression.
    --> packJPG library v2.5k (01/22/2016) by Matthias Stirner / Se <--
    --> packMP3 library v1.0g (01/22/2016) by Matthias Stirner <--
    More about packJPG and packMP3 here: http://www.matthiasstirner.com
    
    
    100.00% - New size: 655725701 instead of 655664804     
    
    
    Done.
    Time: 1 hour(s), 16 minute(s), 6 second(s)
    
    
    Recompressed streams: 164/164
    zLib streams (intense mode): 116/116
    Brute mode streams: 48/48
    Precomp:
    Code:
    100.00% - New size: 655667410 instead of 655664804     
    
    
    Done.
    Time: 54 minute(s), 57 second(s)
    
    
    Recompressed streams: 2/170
    zLib streams (intense mode): 0/116
    Brute mode streams: 2/54
    
    
    You can speed up Precomp for THIS FILE with these parameters:
    -zl61 -d0

  2. The Following User Says Thank You to Gonzalo For This Useful Post:

    Bulat Ziganshin (15th May 2018)

Similar Threads

  1. reflate - a new universal deflate recompressor
    By Shelwien in forum Data Compression
    Replies: 119
    Last Post: 28th March 2018, 22:52
  2. Why not open source?
    By nemequ in forum Data Compression
    Replies: 65
    Last Post: 25th November 2013, 23:05
  3. Wondering whether to release my deflate recompressor
    By m^2 in forum Data Compression
    Replies: 15
    Last Post: 4th November 2011, 19:31
  4. DEFLATE/zlib implementations
    By GerryB in forum Data Compression
    Replies: 10
    Last Post: 7th May 2009, 17:03
  5. Interesting Deflate source
    By encode in forum Forum Archive
    Replies: 10
    Last Post: 21st April 2008, 15:30

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •