Results 1 to 9 of 9

Thread: pZLib

  1. #1
    Member Razor12911's Avatar
    Join Date
    May 2016
    Location
    South Africa
    Posts
    31
    Thanks
    51
    Thanked 53 Times in 15 Posts

    pZLib

    Pre-ZLIB (pzlib)

    This program detects and unpack zlib streams for a certain input (like precomp)
    giving you an option to apply a stronger compression such as LZMA.

    This only detects zlib streams compressed using default and maximum level
    which have headers. (It doesn't detect headerless)

    Usage:
    pZLib encode/decode [options] inputfile -o outputfile
    inputfile/outputfile can be specified as "-" for stdin/stdout

    Options:
    -t#: Number of threads to use (maximum = 8 )
    -t#p: (default = 50) Percentage of threads to use
    -v: Verification so that restoration keeps CRC the same
    -ai: Prints out additional information

    Note:
    Input and/or Output can be "-" to signal stdin and/or stdout usage.
    Multi threading only works during restoration.
    Freearc usage example included

    Example:
    pzlib encode $$arcdatafile$$.tmp -o $$arcpackedfile$$.tmp
    Attached Files Attached Files
    Last edited by Razor12911; 25th May 2016 at 00:56.

  2. The Following 10 Users Say Thank You to Razor12911 For This Useful Post:

    Bulat Ziganshin (23rd May 2016),comp1 (23rd May 2016),Dimitri (23rd May 2016),gmy (27th May 2017),lorents17 (23rd May 2016),Minimum (22nd May 2016),RamiroCruzo (23rd May 2016),Samantha (23rd May 2016),schnaader (23rd May 2016),Sergey3695 (23rd May 2016)

  3. #2
    Member Dimitri's Avatar
    Join Date
    Nov 2015
    Location
    Greece
    Posts
    48
    Thanks
    21
    Thanked 30 Times in 14 Posts
    you can use stdin and out during compression and decompression !!!

    will take 1/2 of the time

  4. #3
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Well done! The concept to process only default zLib streams on maximum compression level first seems a bit odd, but of course it makes very fast encoding possible.

    Some general comments:

    • Adding a version number to the command line output would be useful
    • Allow shortcuts "e"/"d" for "encode"/"decode", too
    • Duration output in ms is useful for comparisons, but "human readable" output in hours/minutes/seconds is often preferred
    • Open source would be better


    pzlib didn't detect any streams in FlashMX.pdf and silesia.zip, so I tested one of my other standard test files, an Ubuntu 15.10 image:
    Code:
    ubuntu-15.10-desktop-amd64.iso
    
    Original: 1,178,386,432 bytes
    Precomp 0.4.5
      -cn -intense -d0 (encode): 19 min 30 s, 2,982,177,562 bytes
        -r (decode): 10 min 47 s
      -cn -intense -d0 -zl96,97,98,99 (encode): 14 min 43 s, 2,981,716,646 bytes
    pzlib 1100
      encode: 2 min 36 s, 2,974,882,594 bytes
        decode: 10 min 58 s, not identical, output is longer (1,178,403,868 bytes) and differs (first different byte at offset 0x0D05BCDC) 
        decode to stdout: 10 min 45 s
      encode -v: 17 min 29 s, 2,972,136,388 bytes
        decode -t1 stdin, stdout: 16 min 9 s
        decode -t2 stdin, stdout: 12 min 13 s
        decode -t4 stdin, stdout: 9 min 39 s, identical
    So encoding without "-v" (verification) is very fast, but decoding fails. Encoding with "-v" is better (decoding is successful), but slower.
    Multithreading works, but the baseline is quite slow, so decoding with the max. number of threads is only a bit faster than Precomp.
    http://schnaader.info
    Damn kids. They're all alike.

  5. The Following 7 Users Say Thank You to schnaader For This Useful Post:

    Bulat Ziganshin (23rd May 2016),gmy (3rd June 2017),lorents17 (23rd May 2016),Minimum (23rd May 2016),RamiroCruzo (23rd May 2016),Razor12911 (23rd May 2016),Samantha (23rd May 2016)

  6. #4
    Member Razor12911's Avatar
    Join Date
    May 2016
    Location
    South Africa
    Posts
    31
    Thanks
    51
    Thanked 53 Times in 15 Posts
    Well, it's just a new program, otherwise, I plan to discard verification and make difflate, which is not going according to plan, it isn't a walk in the park and Multi threading is actually faster than that, but right now limited speed for safety reasons and thank you for the results.

  7. The Following User Says Thank You to Razor12911 For This Useful Post:

    gmy (3rd June 2017)

  8. #5
    Member Razor12911's Avatar
    Join Date
    May 2016
    Location
    South Africa
    Posts
    31
    Thanks
    51
    Thanked 53 Times in 15 Posts
    Update available


    Changes


    Threads inversely proportional to speed for stability (new maximum for threads is 4 instead of 8 )
    -Speed for multi threading improved
    -ms conversion to readable time added as requested
    -encode/decode can be shorted to e/d

  9. The Following 5 Users Say Thank You to Razor12911 For This Useful Post:

    Bulat Ziganshin (24th May 2016),Dimitri (25th May 2016),gmy (3rd June 2017),RamiroCruzo (24th May 2016),schnaader (25th May 2016)

  10. #6
    Member Razor12911's Avatar
    Join Date
    May 2016
    Location
    South Africa
    Posts
    31
    Thanks
    51
    Thanked 53 Times in 15 Posts
    CLS available


    Standalone CLS library for Freearc uploaded.
    It works without the need of arc.ini or pzlib.exe, it just works by itself, it has compress and decompress code.
    You can also paste it in freearc installation directory in bin folder, and use as shown in the pictures.

    Drawback is it is quite slow during compression (I'm using ZLIb compiled for delphi and I don't think it is optimized, x64 itself is slower than x86), but decompression is multi threaded, it uses half the number of threads your CPU has.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	cls-pzlib.PNG 
Views:	1425 
Size:	92.8 KB 
ID:	4418   Click image for larger version. 

Name:	cls-pzlib2.PNG 
Views:	1253 
Size:	18.9 KB 
ID:	4419  

  11. The Following 5 Users Say Thank You to Razor12911 For This Useful Post:

    Bulat Ziganshin (25th May 2016),Dimitri (25th May 2016),gmy (3rd June 2017),Minimum (25th May 2016),RamiroCruzo (25th May 2016)

  12. #7
    Member Razor12911's Avatar
    Join Date
    May 2016
    Location
    South Africa
    Posts
    31
    Thanks
    51
    Thanked 53 Times in 15 Posts
    Hi all, can you test this latest build of pZLib.

    Program seems to hang sometimes while decoding, just want to see if it hangs from everybody's end.

    Note, this is multi threaded and it uses all threads your CPU.
    Attached Files Attached Files

  13. The Following 3 Users Say Thank You to Razor12911 For This Useful Post:

    gmy (3rd June 2017),Minimum (5th October 2016),RamiroCruzo (5th October 2016)

  14. #8
    Member Razor12911's Avatar
    Join Date
    May 2016
    Location
    South Africa
    Posts
    31
    Thanks
    51
    Thanked 53 Times in 15 Posts
    Second test, decoding hang fixed.

    Note, encoding and decoding use all threads your CPU has.
    Attached Files Attached Files

  15. The Following 5 Users Say Thank You to Razor12911 For This Useful Post:

    gmy (27th May 2017),load (5th October 2016),Minimum (5th October 2016),RamiroCruzo (5th October 2016),~MAK~ (5th October 2016)

  16. #9
    Member
    Join Date
    Mar 2017
    Location
    None
    Posts
    1
    Thanks
    0
    Thanked 3 Times in 1 Post
    Newer version including source code, copied from the other forum page: http://fileforums.com/showthread.php?t=97888
    Attached Files Attached Files

  17. The Following 3 Users Say Thank You to anonmous For This Useful Post:

    Bulat Ziganshin (15th March 2017),gmy (3rd June 2017),Stephan Busch (15th March 2017)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •