Results 1 to 1 of 1

Thread: xnLZ - parallel lzma based on plzip with entropy scan

  1. #1
    Member
    Join Date
    Mar 2018
    Location
    sun
    Posts
    33
    Thanks
    18
    Thanked 15 Times in 7 Posts

    xnLZ - parallel lzma based on plzip with entropy scan

    xnLZ is a multi-threaded lzma (de)compressor heavily based on plzip, of which it is fork of and NOT compatible with anymore.
    Its main purpose is to offer alternative to existing 4x4:lzma chain(also known as xlzma) in FreeArc(or other archivers that
    can utilize external compressors), but with 64bit support and same cool features that made 4x4 so great. Although xnLZ was
    intended to be used for <stdio> mode with FreeArc, it can also work on its own with files, or can be chained with other tools
    that support external compressors.

    Here are the main features:

    - xnLZ utilize entropy check that scan incoming data blocks during compression and can copy them directly if they are not
    compresseable enough. All this can be defined by user. This means some data that do not benefit from typical lzma compression
    can bypass compress routine completely, speeding whole process. On combined data sets this could usually be anywhere
    between 50-300% speedup and decompression speed will benefit as well.

    - Data blocks that are copied directly still have their CRC attached and are verified during test or decompression.

    - xnLZ was designed to utilize all CPU cores during decompression through <stdio> for maximum speed. This feature actually
    existed(as of time of fork at least) in original plzip in complete infrastructure, all that was needed was to increase
    buffer/packet limit, which was artificially hardcoded to low value. Now user can directly define how much buffer to allocate,
    but it comes at the cost of utilizing more memory during decompression. As example, to fully utilize 4 core CPU with data
    blocks of 256MB, ~1300MB-2GB may be needed during decompression, depending on settings. In my tests I was able to load my
    CPU fully with slots value 128 on data blocks of 256MB size, which used only around ~1GB RAM.

    - xnLZ use hardcoded lzma:lc(literal context) parameter set to 8 instead of default 3 for maximum compression.
    - xnLZ expose additional lzma:mc(matchfinder cycles count) option for the user to set, which tend to have big impact on compression speed
    - xnLZ never delete original file(s) and went through other changes regarding settings and behavior

    xnLZ utilize same block structure format as plzip/lzip, but header use different ID and contain additional byte to control
    whether data were compressed or not.


    Thanks to original plzip/lzlib authors, to Matt Mahoney for entropy code and Bulat Ziganshin for assistance.


    Quick random benchmark on combined data folder with lot of zip files, few big exes and dlls:

    Code:
    xlzma(4x4:lzma) & xnlz = d64m, b64m, fb32, mc32, E99% <- both use lzma1:lc8
    7z 16.04     = 0=lzma2:d=64m:fb=32:lc=4:mc=32         <- use lzma2
    
    
    orig data: 3.24g
    
    
    xlzma(4x4)- c 1:15m, d 5s,  2.76g
    xnlz      - c 1:40m, d 13s, 2.74g
    xnlz:E98  - c 1:23m, d 11s, 2.76g
    7z        - c 8:12m, d 26s, 2.70g
    xnlz:E100 - c 6:15m, d 49s, 2.71g
    Note although I cannot confirm it yet, its possible that slower decompression with disabled entropy against 7z may be due to lc8 parameter of lzma1. I got
    this mail from original plzip author when I asked him back then why he did not used bigger lc in his software:

    "The reason to not use lc = 8 is that it requires 768 KiB of fast RAM vs the 24 KiB required by lc = 3. This would make decompression slow on machines with, say, 64 KiB of cache."

    I may need to verify if this is the case here or just lzma/xnlz being slower to lzma2 in general. Also, xlzma was lc8 as well and did not suffer.

    EDIT: here is the result on uncompressed data:
    Code:
    orig file: 660.3mb
    7z: c 1:15m, d 12s, 270.19mb
    xnlz: c 1:17m, d 7s, 270.85mb

    The reason why xnlz was slower during decompression in first test when entropy skipping was disabled and despite using all cores was because archive contained compressed data(zip files) and entropy skipping was disabled. Lzma2 was quicker because it does better job handling compressed data than lzma1. However, first it is no match on whole block entropy skipping in terms of speed when compared to xlzma or xnlz, and second here it was finally slower with inflated data decompression.
    Attached Files Attached Files

  2. The Following 5 Users Say Thank You to elit For This Useful Post:

    78372 (26th March 2018),hunman (26th March 2018),schnaader (26th March 2018),Simorq (26th March 2018),xinix (28th March 2018)

Similar Threads

  1. Replies: 23
    Last Post: 24th March 2018, 17:57
  2. jpegultrascan, an exhaustive JPEG scan optimizer
    By MegaByte in forum Data Compression
    Replies: 35
    Last Post: 28th November 2016, 10:09
  3. Replies: 9
    Last Post: 11th June 2015, 23:28
  4. ENTROPY 0.5
    By comp1 in forum Download Area
    Replies: 3
    Last Post: 7th February 2013, 00:21
  5. Implementation of JPEG2000 LLC based on LZMA
    By Raymond_NGhM in forum Data Compression
    Replies: 0
    Last Post: 19th March 2010, 01:14

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •