Results 1 to 5 of 5

Thread: Super Microsoft LZX

  1. #1
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    174
    Thanked 51 Times in 37 Posts

    Super Microsoft LZX

    Just thought I'd share this with you guys.

    Microsoft's LZX has a limited window size. It was created back in the 1990's when the memory limitations were much different from today.

    Microsoft does have a delta compression that is used in their update packages. Using a 0 byte file, I tested compression.

    Quick test on ENWIK9:
    Code:
             SIZE       D TIME
    ===========================
     CABARC  250764075   6.161
    MSDELTA  209195904  12.636
    Someone much smarter than me perhaps can test this and benefit from it.

    Attached are Windows executables for testing. Use like this:

    msdeltap.exe <patch file> <in file> <out file>
    msdeltaa.exe <patch file> <in file> <out file>

    Have fun!
    Attached Files Attached Files

  2. The Following User Says Thank You to comp1 For This Useful Post:

    load (28th November 2016)

  3. #2
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Yes, old LZX supports window size up to 2M (2^21), and msdelta supports up to 32M (see https://msdn.microsoft.com/en-us/lib...exchg.80).aspx )

    Then, there's also a new "LZMS" format in .wim ( https://wimlib.net/compression.html#LZMS ), which also has a curious E8 filter extension for x64.

  4. The Following User Says Thank You to Shelwien For This Useful Post:

    xinix (29th November 2016)

  5. #3
    Member
    Join Date
    Nov 2014
    Location
    Earth
    Posts
    38
    Thanks
    0
    Thanked 77 Times in 19 Posts
    At some point I might update wimlib to support 2^25 byte windows for LZX. Currently it only supports up to 2^21 bytes, though that's already more than Microsoft's WIM software which only supports 2^15 bytes. The way Microsoft extended LZX in LZX-DELTA to support 2^25 byte windows is a little ugly because the main Huffman code ends up with 2576 symbols (256+290*8 ), which is a lot. They must not have wanted to assign more than 17 bits to any offset slot, since > 17 bits doesn't work well with the LZX bitstream.

    LZMS's preprocessing step is a little more than just "E8" (call relative) filtering: it also recognizes, for example, REX-prefixed x86_64 instructions that use instruction pointer-relative addressing. But the basic idea is the same.

    LZX on the other hand really just does "E8 filtering". And it works, though at one point I compared it with the E8/E9 filter used by xz/7-Zip, and the xz/7-Zip one was better.

  6. The Following 2 Users Say Thank You to Zyzzyva For This Useful Post:

    comp1 (29th November 2016),Cyan (29th November 2016)

  7. #4
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    301
    Thanks
    26
    Thanked 22 Times in 15 Posts
    BTW Igor (the Inventor of 7z) has done a LZX compressor example called BIX (though not same format) and it was the bases of LZMA format.

    Regarding Wim.. is it possible to make a LZX compressor for CHM/CAB containers.

  8. #5
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > I compared it with the E8/E9 filter used by xz/7-Zip, and the xz/7-Zip one was better.

    Afaik that one (bcj) is not especially good either. Same with LZMS one - I got interested, but ended up writing a new one with the same idea:

    Code:
    http://nishi.dreamhosters.com/u/E8flt-v2.rar
    http://nishi.dreamhosters.com/u/x64flt_v2.rar
    
    33,867,264 amdocl64.dll
    33,867,394 amdocl64.bcj     // 7z a -m0=bcj mcpcom.bcj mcpcom.exe
    33,882,265 amdocl64.bcj2    // 7z a -m0=bcj2 mcpcom.bcj2 mcpcom.exe
    33,867,264 amdocl64.MSflt   // LZMS x64/E8 filter
    33,867,264 amdocl64.E8v2    // e8filter.exe c mcpcom.exe mcpcom.E8v2
    33,867,264 amdocl64.E8v3    // x64flt.exe c mcpcom.exe mcpcom.E8v3
    33,867,410 amdocl64.x64bcj  // 7zdll_vE\x64\7z a -tpa -m0=x64flt -m1=bcj mcpcom.x64bcj mcpcom.exe  
    33,882,659 amdocl64.x64bcj2 // 7zdll_vE\x64\7z a -tpa -m0=x64flt -m1=bcj2 mcpcom.x64bcj2 mcpcom.exe
    
    [7z a -bb3 -mx=9 -mf=off %%a.7z %%a]
     8,431,899 amdocl64.dll.7z
     7,878,007 amdocl64.MSflt.7z
     7,665,896 amdocl64.bcj.7z
     7,662,790 amdocl64.E8v2.7z
     7,584,010 amdocl64.x64bcj.7z
     7,580,512 amdocl64.E8v3.7z
     7,522,934 amdocl64.bcj2.7z
     7,472,154 amdocl64.x64bcj2.7z
    Of course bcj2 results would be even better if we'd let 7z properly process its streams, instead of using it as a standalone filter.
    To be specific:
    Code:
     7,315,941 amdocl_1.pa // 1.bat (lzma compression)
     7,165,490 amdocl_2.pa // 2.bat (plzma)
    
    // 1.bat
    7zdll_vE\x64\7z.exe a -bb3 -mf=off ^
    -m0=x64flt -m1=bcj2 -m2=deltb:c8M -m3=deltb:c8M ^
    -m4=plzma4:a1:mt1:d26:lc8:pb2:lp0:fb273:mc999 ^
    -m5=deltb:c8M -m6=deltb:c8M -m7=plzma4:a0:mt1:d23:lc0:pb2:lp2:fb273 ^
    -m8=deltb:c8M -m9=deltb:c8M -m10=plzma4:a0:mt1:d23:lc0:pb2:lp2:fb273 ^
    -m11=lzma2:mt1:d20:lc0:lp0:pb0:fb273 ^
    -mb00s0:1 -mb01s0:2 -mb01s1:5 -mb01s2:8 -mb01s3:11 -mb02s0:3 -mb03s0:4 -mb05s0:6 -mb06s0:7 -mb08s0:9 -mb09s0:10 ^
    amdocl_1.pa amdocl64.dll
    
    //.bat
    7zdll_vE\x64\7z.exe a -bb3 -mf=off ^
    -m0=x64flt -m1=bcj2 -m2=deltb:c8M -m3=deltb:c8M ^
    -m4=lzma:mt1:d26:lc8:pb2:lp0:fb273:mc999 ^
    -m5=deltb:c8M -m6=deltb:c8M -m7=lzma:mt1:d23:lc0:pb2:lp2:fb273 ^
    -m8=deltb:c8M -m9=deltb:c8M -m10=lzma:mt1:d23:lc0:pb2:lp2:fb273 ^
    -m11=lzma2:mt1:d20:lc0:lp0:pb0:fb273 ^
    -mb00s0:1 -mb01s0:2 -mb01s1:5 -mb01s2:8 -mb01s3:11 -mb02s0:3 -mb03s0:4 -mb05s0:6 -mb06s0:7 -mb08s0:9 -mb09s0:10 ^
    amdocl_2.pa amdocl64.dll

  9. The Following 4 Users Say Thank You to Shelwien For This Useful Post:

    comp1 (29th November 2016),maadjordan (29th November 2016),Mike (29th November 2016),xinix (29th November 2016)

Similar Threads

  1. Microsoft Office save XML zipped files
    By BetaTester in forum Data Compression
    Replies: 2
    Last Post: 14th July 2012, 09:18
  2. Microsoft Visual Studio 11
    By encode in forum The Off-Topic Lounge
    Replies: 1
    Last Post: 28th April 2012, 14:40
  3. A Microsoft study on deduplication
    By m^2 in forum Data Compression
    Replies: 1
    Last Post: 5th May 2011, 18:15
  4. LZX recompression - proof of concept
    By schnaader in forum Data Compression
    Replies: 3
    Last Post: 29th March 2011, 14:22
  5. super-fast i/o
    By Bulat Ziganshin in forum Forum Archive
    Replies: 18
    Last Post: 10th May 2007, 22:08

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •