Results 1 to 14 of 14

Thread: Why does BWT beat LZMA/LZMA2?

  1. #1
    Member
    Join Date
    Oct 2014
    Location
    South Africa
    Posts
    38
    Thanks
    23
    Thanked 7 Times in 5 Posts

    Why does BWT beat LZMA/LZMA2?

    Hi everybody.

    I don't know why BWT beats LZMA sometimes. I know that BWT sometimes works better on text files (eg enwiki8 ), but I don't get it why it outperforms LZMA while compressing Photoshop PSD files.

    I attached a sample PSD file. For this one (and most of the files with a BW layer and a RGB layer), Rings Archiver (of Nania Francesco) and Nanozip Optimum 1&2 compress better. Here are the commands used in FreeArc.

    Code:
    LZMA:    9rep+mm:1*8+9rep+9xb:273:lc8:pb0:lp1
    NZ 0.9:  9rep+9rep+nz9:co:m2g:t1:p1
    Rings:   9rep+9rep+ringsx86:m6

    Code:
    Compressor                     Size 
    -------------                   --------
    LZMA                            5,181,956 
    NZ 0.9 Optimum 1                4,684,064  
    Ringsx86                        4,501,542
    Is there anyway to improve the compression in LZMA?
    I don't prefer to use an external compressor in FreeArc as LZMA mostly compresses better than BWT.

    Thanks
    Attached Files Attached Files

  2. #2
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Well, with
    Code:
    mm.exe -c1 -w8 Ladybug.psd 3
    lzma.exe e 3 Ladybug.lzma1 -a1 -d25 -fb273 -mc999 -lc8 -lp0 -pb0 -mfbt4 -mt2
    I'm getting 5,090,190 like this, so better results _are_ possible.
    Also maybe its not an lzma problem, but simply these compressors have better filters than Bulat's mm?

  3. The Following User Says Thank You to Shelwien For This Useful Post:

    msat59 (11th December 2016)

  4. #3
    Member
    Join Date
    Oct 2014
    Location
    South Africa
    Posts
    38
    Thanks
    23
    Thanked 7 Times in 5 Posts
    Quote Originally Posted by Shelwien View Post
    Well, with
    Code:
    mm.exe -c1 -w8 Ladybug.psd 3
    lzma.exe e 3 Ladybug.lzma1 -a1 -d25 -fb273 -mc999 -lc8 -lp0 -pb0 -mfbt4 -mt2
    I'm getting 5,090,190 like this, so better results _are_ possible.
    Also maybe its not an lzma problem, but simply these compressors have better filters than Bulat's mm?
    Thanks.

    So you mean nothing cannot be done to improve the compression of FreeArc internal LZMA?

    What is that mm.exe? May you attach it here?

  5. #4
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > So you mean nothing cannot be done to improve the compression of FreeArc internal LZMA?

    You can use lp0 instead of lp1 at least.

    > What is that mm.exe? May you attach it here?

    http://freearc.org/download/research/mm11.zip

    I just used standalone programs instead of freearc, it should be the same.

  6. #5
    Member
    Join Date
    May 2007
    Location
    Poland
    Posts
    85
    Thanks
    8
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by msat59 View Post
    Thanks.

    So you mean nothing cannot be done to improve the compression of FreeArc internal LZMA?

    What is that mm.exe? May you attach it here?
    for Bitmaps and LZMA you should use delta filter and parameters like:
    0=delta:3 1=lzma:lc=7:lp=0: pb=0:fc=273:x=9 (remove the space after lp=0)

    EDIT: I guess it is one channel (don't have Photoshop) then change delta:3 to delta:1. For this file lc=8 works best:

    5*089*712 bytes
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	bitmap_lzma.png 
Views:	128 
Size:	6.0 KB 
ID:	4776  

  7. #6
    Member
    Join Date
    Oct 2014
    Location
    South Africa
    Posts
    38
    Thanks
    23
    Thanked 7 Times in 5 Posts
    Thank you all.

    The new size is 5,091,005 using:

    Code:
    /$psd=mm:1*8+9rep+9xb:273:lc8:pb1:lp0
    but still far from BWT.

  8. #7
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by msat59 View Post
    I don't know why BWT beats LZMA sometimes.
    BWT works as a very long byte-based context. LZMA has a shorter but more complex context (+ LZ77ish modeling).

    When the data benefits from a long byte-based context, BWT can be a better fit than advanced LZ77.

  9. The Following User Says Thank You to Jyrki Alakuijala For This Useful Post:

    msat59 (12th December 2016)

  10. #8
    Member
    Join Date
    Oct 2014
    Location
    South Africa
    Posts
    38
    Thanks
    23
    Thanked 7 Times in 5 Posts
    So Bulat needs to include the BWT code in FreeArc Next !


    Isn't there any BWT preprocessor, not compressor?

  11. #9
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    You're not comparing lzma vs bwt here, but whole chains of filters + codecs.
    And its not even certain that its impossible to tweak some MM options to get better results.
    Also, neither BWT nor lzma are good codecs for images, at least use FLIF or something.

  12. #10
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    149
    Thanks
    30
    Thanked 59 Times in 35 Posts
    So Bulat needs to include the BWT code in FreeArc Next!
    Already included BWT code(grzip)

  13. #11
    Member
    Join Date
    Oct 2014
    Location
    South Africa
    Posts
    38
    Thanks
    23
    Thanked 7 Times in 5 Posts
    Quote Originally Posted by xezz View Post
    Already included BWT code(grzip)
    I meant as a preprocessor; however why doesn't the size of compressed file by grzip (5,122,656) go below LZMA, like the other two BWT compressors?

    I think Nania should describe it.

    EDIT:
    I got 4,864,285 with mm:1*8+9rep+grzip. Now it's smaller than LZMA, but still bigger than other BWTs.



    Quote Originally Posted by Shelwien View Post
    Also, neither BWT nor lzma are good codecs for images, at least use FLIF or something.
    I don't think that it is possible to compress a PSD file with FLIF; however I wanted to find a solution using only FreeArc, not an external compressor. Nania's MCM gives better ratio and the (de)compression time is reasonable.
    Last edited by msat59; 13th December 2016 at 11:59.

  14. #12
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    149
    Thanks
    30
    Thanked 59 Times in 35 Posts
    I got 4,864,285 with mm:1*8+9rep+grzip. Now it's smaller than LZMA, but still bigger than other BWTs.
    grzip's maxmum bocksize is 8MB, so if file size is much larger than 8MB then maybe it can not beat NZ and Rings.
    I hope grzip to implement large blocksize.

  15. The Following User Says Thank You to xezz For This Useful Post:

    msat59 (13th December 2016)

  16. #13
    Member
    Join Date
    Oct 2014
    Location
    South Africa
    Posts
    38
    Thanks
    23
    Thanked 7 Times in 5 Posts
    Quote Originally Posted by xezz View Post
    grzip's maxmum bocksize is 8MB, so if file size is much larger than 8MB then maybe it can not beat NZ and Rings.
    I hope grzip to implement large blocksize.
    Now I got it. Thanks

  17. #14
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > I don't think that it is possible to compress a PSD file with FLIF

    Actually it is kinda possible - convert it with imagemagick to .pnm or a set of bmps, then compress these with flif,
    then decode flif, convert to psd with imagemagick and make a patch with xdelta from recoded image to original one.

    I just tested it and flif compresses this file to 4,469,149 even as 4 separate images (psd layers) - the result
    would be probably much better if we'd turn these layers into frames in the same file, or something.

    But the problem is that .psd apparently has some RLE compression, and imagemagick uses it differently,
    so only the psd produced by imagemagick has a chance of being losslessly compressed with this method atm.

Similar Threads

  1. lzma2 stream detector
    By Shelwien in forum Data Compression
    Replies: 0
    Last Post: 4th June 2016, 19:28
  2. BWT + LZMA
    By Zelex in forum Data Compression
    Replies: 10
    Last Post: 21st July 2013, 11:44
  3. 7z.exe stdin stdout (for lzma2)
    By lordvj in forum Data Compression
    Replies: 6
    Last Post: 16th May 2013, 09:22
  4. Lossless compression formats using LZMA or LZMA2 ?
    By Karhunen in forum Data Compression
    Replies: 6
    Last Post: 3rd February 2012, 05:15
  5. LZMA source
    By Shelwien in forum Data Compression
    Replies: 2
    Last Post: 29th March 2010, 18:45

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •