Page 1 of 2 12 LastLast
Results 1 to 30 of 50

Thread: lzbench - in-memory benchmark of open-source LZ77/LZSS/LZMA compressors

  1. #1
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts

    Thumbs up lzbench - in-memory benchmark of open-source LZ77/LZSS/LZMA compressors

    lzbench v1.3 has been released https://github.com/inikep/lzbench/releases

    Changes:
    - added slz 1.0.0 : slz_zlib, slz_gzip, slz_deflate (thanks to Willy Tarreau)
    - added glza 0.7.1 (thanks to Kennon Conrad)
    - zstd updated to v0.8.0 (thanks to Chip Turner)
    - brotli updated to v0.5.2
    - gipfeli updated to 2016-07-13
    - lzfse and lzvn updated to 2016-08-16
    - lz5 updated to v1.5
    - lzbench compiles on MacOS (with exception of lzham)
    - lzbench compiles with clang (tested with 3.4, 3.5, 3.6 and 3.8 )
    - added support for aggregated parameters: "-t0 -u0 -i3 -j5" is equal to "-t0u0i3j5" (only "-e" can not be aggregated)
    - added new options:
    -r = disable real-time process priority
    -v = disable progress information
    -z = show (de)compression times instead of speed

  2. The Following 5 Users Say Thank You to inikep For This Useful Post:

    Bulat Ziganshin (17th August 2016),Kennon Conrad (17th August 2016),Mike (17th August 2016),Turtle (17th August 2016),willy (6th September 2016)

  3. #2
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    lzbench v1.4 has been released at https://github.com/inikep/lzbench/releases

    Changes:
    - zstd updated to v1.0.0
    - added libdeflate 2016-08-29 (thanks to Jørgen Ibsen)
    - added the "-m" (memory_limit) option; large files are processed in (memory_limit/4) blocks
    - cleaner CSV output and added the "original size" field

  4. The Following User Says Thank You to inikep For This Useful Post:

    Cyan (1st September 2016)

  5. #3
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    lzbench v1.5 has been released at https://github.com/inikep/lzbench/releases

    Changes:
    - added the "-r" option : operate recursively on directories
    - added the "-j" option : join files in memory but compress them independently (for many small files)
    - lz5 updated to v2.0 RC2
    - lz4 updated to v1.7.3
    - zstd updated to v1.1.1
    - libdeflate updated to v0.6
    - csc updated to 2016-10-13
    - improved PowerPC support
    - fixed "-m" option (thanks to Julian Kunkel)

  6. The Following 2 Users Say Thank You to inikep For This Useful Post:

    Cyan (21st November 2016),Jyrki Alakuijala (21st November 2016)

  7. #4
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    621
    Thanks
    182
    Thanked 225 Times in 138 Posts
    Quote Originally Posted by inikep View Post
    lzbench v1.5 has been released
    Would you consider adding the large window brotli?

    I'm thinking of basing the brotli framing format on a brotli extension like it, and it would be nice to learn more about its performance with a benchmarking tool like lzbench.

    For silensia.tar, large window brotli compresses down to 50164028 bytes (23.67 %) with 30 bit window. A 24 bit window (normal brotli) gives 50397564 bytes (23.78 %). For other more homogeneous benchmarking files, such as enwik9, the savings from large window can be 10 %.

    Btw, you are comparing a 22 bit windowed brotli against 23 bit windowed zstd here: https://github.com/inikep/lzbench/ -- This can be a compression density difference of 4-5 %, but probably only 1 % for silesia.tar. It would be a more interesting comparison if all compression algorithms are run with the same window size.
    Last edited by Jyrki Alakuijala; 21st November 2016 at 22:29.

  8. #5
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Would you consider adding the large window brotli?
    I don't know details of large window brotli but I'm afraid that there will be a collision of symbols/function names. Is there any simple solution (e.g. #define) for that?

    Quote Originally Posted by Jyrki Alakuijala View Post
    you are comparing a 22 bit windowed brotli against 23 bit windowed zstd
    No, I'm comparing default compression levels (e.g. lzham has 26-bit window).

    Quote Originally Posted by Jyrki Alakuijala View Post
    It would be a more interesting comparison if all compression algorithms are run with the same window size.
    It's already done. There are aliases with 22 and 24-bit window: brotli22, brotli24, lzham22, lzham24, zstd22, zstd24 so you can try e.g.
    lzbench -ebrotli22/zstd22 silesia.tar

  9. #6
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    The following results are obtained with "lzbench -t16,16 -eall" using 1 core of Intel Core i5-4300U, Windows 10 64-bit (MinGW-w64 compilation under gcc 6.2.0) with silesia.tar.
    Full results are available here: https://github.com/inikep/lzbench/bl...ch15_sorted.md

    The pareto frontier for decompression speed and ratio:
    Code:
    | Compressor name         | Compression| Decompress.| Compr. size | Ratio |
    | ---------------         | -----------| -----------| ----------- | ----- |
    | lzlib 1.7 -9            |  1.21 MB/s |    45 MB/s |    48296889 | 22.79 |
    | xz 5.2.2 -9             |  1.80 MB/s |    58 MB/s |    48745306 | 23.00 |
    | xz 5.2.2 -6             |  1.97 MB/s |    58 MB/s |    49195929 | 23.21 |
    | lzma 9.38 -5            |  2.08 MB/s |    65 MB/s |    49720569 | 23.46 |
    | brotli 0.5.2 -11        |  0.39 MB/s |   266 MB/s |    51138054 | 24.13 |
    | zstd 1.1.1 -22          |  1.44 MB/s |   505 MB/s |    52731930 | 24.88 | 
    | zstd 1.1.1 -18          |  2.87 MB/s |   583 MB/s |    55294241 | 26.09 |
    | zstd 1.1.1 -15          |  4.97 MB/s |   639 MB/s |    58007773 | 27.37 |
    | lz5 2.0 RC2 -49         |  1.27 MB/s |  1064 MB/s |    60679215 | 28.63 |
    | lz5 2.0 RC2 -45         |  6.25 MB/s |  1073 MB/s |    65413061 | 30.86 |
    | lz5 2.0 RC2 -29         |  1.37 MB/s |  1634 MB/s |    68694227 | 32.41 |
    | lz5 2.0 RC2 -25         |  6.63 MB/s |  1734 MB/s |    74503695 | 35.15 |
    | lzsse8 2016-05-14 -16   |  6.05 MB/s |  2858 MB/s |    75464339 | 35.61 |
    | lzsse8 2016-05-14 -12   |  6.00 MB/s |  2859 MB/s |    75464339 | 35.61 |
    | blosclz 2015-11-10 -3   |   487 MB/s |  5047 MB/s |   204507781 | 96.49 |
    | blosclz 2015-11-10 -1   |   892 MB/s |  5679 MB/s |   211768481 | 99.92 |
    | memcpy                  |  7332 MB/s |  8719 MB/s |   211947520 |100.00 |
    The pareto frontier for compression speed and ratio:

    Code:
    | Compressor name         | Compression| Decompress.| Compr. size | Ratio |
    | ---------------         | -----------| -----------| ----------- | ----- |
    | lzlib 1.7 -9            |  1.21 MB/s |    45 MB/s |    48296889 | 22.79 |
    | xz 5.2.2 -9             |  1.80 MB/s |    58 MB/s |    48745306 | 23.00 |
    | xz 5.2.2 -6             |  1.97 MB/s |    58 MB/s |    49195929 | 23.21 |
    | lzma 9.38 -5            |  2.08 MB/s |    65 MB/s |    49720569 | 23.46 |
    | csc 2016-10-13 -5       |  2.49 MB/s |    50 MB/s |    49801577 | 23.50 |
    | csc 2016-10-13 -3       |  5.98 MB/s |    46 MB/s |    53477914 | 25.23 |
    | csc 2016-10-13 -1       |    14 MB/s |    48 MB/s |    56201092 | 26.52 |
    | lzma 9.38 -2            |    15 MB/s |    55 MB/s |    58867911 | 27.77 |
    | zstd 1.1.1 -11          |    16 MB/s |   613 MB/s |    59523167 | 28.08 |
    | brotli 0.5.2 -5         |    24 MB/s |   312 MB/s |    60801716 | 28.69 |
    | zling 2016-01-10 -4     |    26 MB/s |   135 MB/s |    60997465 | 28.78 |
    | zstd 1.1.1 -8           |    31 MB/s |   619 MB/s |    61026497 | 28.79 |
    | zling 2016-01-10 -2     |    32 MB/s |   135 MB/s |    61917662 | 29.21 |
    | zling 2016-01-10 -1     |    36 MB/s |   135 MB/s |    62438620 | 29.46 |
    | zling 2016-01-10 -0     |    40 MB/s |   132 MB/s |    63407921 | 29.92 |
    | zstd 1.1.1 -5           |    88 MB/s |   565 MB/s |    65002208 | 30.67 |
    | brotli 0.5.2 -2         |    96 MB/s |   283 MB/s |    68066621 | 32.11 |
    | zstd 1.1.1 -2           |   181 MB/s |   600 MB/s |    70168955 | 33.11 |
    | zstd 1.1.1 -1           |   235 MB/s |   645 MB/s |    73659468 | 34.75 |
    | lz5 2.0 RC2 -30         |   246 MB/s |   909 MB/s |    85727429 | 40.45 |
    | density 0.12.5 beta -3  |   254 MB/s |   234 MB/s |    87622980 | 41.34 |
    | pithy 2011-12-24 -9     |   257 MB/s |  1269 MB/s |    90360813 | 42.63 |
    | pithy 2011-12-24 -6     |   297 MB/s |  1285 MB/s |    92090898 | 43.45 |
    | quicklz 1.5.0 -1        |   349 MB/s |   439 MB/s |    94720562 | 44.69 |
    | pithy 2011-12-24 -3     |   354 MB/s |  1235 MB/s |    97255186 | 45.89 |
    | lzo1x 2.09 -1           |   401 MB/s |   548 MB/s |   100572537 | 47.45 |
    | lz4 1.7.3               |   440 MB/s |  2318 MB/s |   100880800 | 47.60 |
    | density 0.12.5 beta -2  |   479 MB/s |   660 MB/s |   101706226 | 47.99 |
    | lz4fast 1.7.3 -3        |   516 MB/s |  2317 MB/s |   107066190 | 50.52 |
    | lz4fast 1.7.3 -17       |   781 MB/s |  2696 MB/s |   131732802 | 62.15 |
    | density 0.12.5 beta -1  |   806 MB/s |  1028 MB/s |   133085162 | 62.79 |

  10. The Following 2 Users Say Thank You to inikep For This Useful Post:

    Jyrki Alakuijala (22nd November 2016),m^3 (22nd November 2016)

  11. #7
    Member
    Join Date
    Nov 2015
    Location
    ?l?nsk, PL
    Posts
    81
    Thanks
    9
    Thanked 13 Times in 11 Posts
    I didn't realize that the field of moderately fast to fast compressors (judging by compression speed) was this competitive.

  12. #8
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    621
    Thanks
    182
    Thanked 225 Times in 138 Posts
    Quote Originally Posted by inikep View Post
    The pareto frontier for ...
    A pareto-analysis would be more interesting if all compressors are measured with the same window size -- or alternatively would run the compressors with a large variety of allowed window sizes. Now both brotli and zstd are taking a huge hit against the others when they are run at 22 and 23 bits against possibly 28 bit? window of lzma.

    (A smaller window size means better n-way multi-processing and less resource use during streaming, but neither of these characteristics are measured in this benchmark.)

  13. The Following User Says Thank You to Jyrki Alakuijala For This Useful Post:

    encode (22nd November 2016)

  14. #9
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    A pareto-analysis would be more interesting if all compressors are measured with the same window size.
    I understand that the same window size (or even better: the same memory usage) might be useful for multitasking or servers with the specified limit of memory but there are also many other applications of data compression. I also have a few comments:
    1. Does it mean that you prefer a 16-bit window size to match LZ4/lzsse/snappy?
    2. Zstd has a different window size almost for each compression level. It even changes parameters depending on file size. Zstd -22 currently uses a 27-bit window. Forcing LZMA/XZ/Zstd to 22-bit or 24-bit could influence other parameters and destroy ratio/speed tradeoff designed for this algorithm.
    3. I'm open to accept this kind of patch for lzbench. I have no time and belief to do it by myself.

  15. #10
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    621
    Thanks
    182
    Thanked 225 Times in 138 Posts
    Quote Originally Posted by inikep View Post
    I understand that the same window size (or even better: the same memory usage) might be useful for multitasking or servers with the specified limit of memory but there are also many other applications of data compression. I also have a few comments:
    1. Does it mean that you prefer a 16-bit window size to match LZ4/lzsse/snappy?
    I think for every algorithm the only thing that matters is that what brings it to the pareto-frontier. If lower or higher window sizes bring an algorithm to the pareto-frontier, then tuning the window size can be important.

    Other thoughts on this:

    I don't think < 18 bits matters much in resource use. There is usually a 256+ kB dedicated cache per core nowadays, and most software that wants to decompress can afford a 256 kB buffer for it.

    Of course, it is technically impossible to use higher windows with some of these formats, like snappy and zlib are only up to 15 bits.

    Quote Originally Posted by inikep View Post
    2. Zstd has a different window size almost for each compression level. It even changes parameters depending on file size. Zstd -22 currently uses a 27-bit window. Forcing LZMA/XZ/Zstd to 22-bit or 24-bit could influence other parameters and destroy ratio/speed tradeoff designed for this algorithm.
    I don't think there is anything magical in LZMA etc that tunes to the window size. Here, a simple comparison on the window size impact on silesia.tar with lzma:9 and brotli:11

    lzma:9 48933295 49675940 51116450 (window size 24, 22 and 20)
    4.27 % between 24 bit and 20 bit window
    2.82 % between 22 bit and 20 bit window

    brotli:11 50397564 51139327 52632475 (window size 24, 22 and 20)
    4.25 % between 24 bit and 20 bit window
    2.84 % between 22 bit and 20 bit window

    I believe it is safe to assume that LZ77-based approaches benefit from a larger window size in a similar manner, and that it is harmful to compare them using different window sizes in different algorithms.

  16. The Following 2 Users Say Thank You to Jyrki Alakuijala For This Useful Post:

    Bulat Ziganshin (23rd November 2016),Turtle (25th November 2016)

  17. #11
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    I think for every algorithm the only thing that matters is that what brings it to the pareto-frontier.
    The pareto frontier shows only selected properties of the algorithm. In my results one can see quite different algorithms in the pareto frontier for compression speed/ratio and decompression speed/ratio. Moreover it depends on used computer, input files, compiler, and so on.
    The bigger issue with my pareto results is that "lzbench -eall" selects every 3rd ot 4th compression level for most of compressors because there are so many compressors and levels (look at lzo or lz5 which now has 40 levels). So it's also the pareto frontier limited to compressors available in lzbench with limited amount of levels. But everybody is invited to make their own tests.

    Quote Originally Posted by Jyrki Alakuijala View Post
    that it is harmful to compare them using different window sizes in different algorithms.
    I think it's valid when you're taking only ratio into account. A different window size highly influences on compression speed and may destroy ratio/speed tradeoff designed by author for this algorithm. Look for example at https://github.com/facebook/zstd/blo....c#L3130-L3152
    Last edited by inikep; 23rd November 2016 at 10:46.

  18. #12
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    lzbench v1.6 has been released at https://github.com/inikep/lzbench/releases

    Changes:
    - zstd updated to v1.1.3
    - lz4 updated to v1.7.5
    - lz5 updated to v2.0
    - improved: touch pages in buffer allocations prior to running first algorithm (thanks to travisdowns)
    - fixed: check if input name is a directory
    - fixed: median time (the "-p3" option, thanks to Alexey Tourbin)

  19. The Following 5 Users Say Thank You to inikep For This Useful Post:

    anormal (1st June 2017),Bulat Ziganshin (9th February 2017),Cyan (7th February 2017),jibz (7th February 2017),Samantha (10th March 2017)

  20. #13
    Member Samantha's Avatar
    Join Date
    Apr 2016
    Location
    italy
    Posts
    38
    Thanks
    31
    Thanked 7 Times in 4 Posts
    Hi @inikep, wanted to know if there were updates for lzbench 1.6, is whether there was the possibility of adding other compressors not lz, example (FB9, FP8, nanozip, paq8px, zcm and zpaq).
    I fixed through the GUI, the reading limit for large files by calculating the average of the divided parts of the file, for each test bench.
    At the end I used lzbench 1.6 for a wide range of compressors and stability.
    If you want to try my utility let me know, I'll send in PM.

    Click image for larger version. 

Name:	Arc+LZB_2.5G.jpg 
Views:	114 
Size:	813.8 KB 
ID:	4845

    Thanks again for your work.
    Last edited by Samantha; 10th March 2017 at 16:41. Reason: Add Pic Test


  21. #14
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    Quote Originally Posted by Samantha View Post
    possibility of adding other compressors not lz, example (FB9, FP8, nanozip, paq8px, zcm and zpaq)
    It's not possible to add nanozip and zcm because these are closed-source compressors.
    I think that I will not add non-LZ compressors by myself but I will accept them as pull requests.
    More information at https://github.com/inikep/lzbench/issues/21

  22. #15
    Member Samantha's Avatar
    Join Date
    Apr 2016
    Location
    italy
    Posts
    38
    Thanks
    31
    Thanked 7 Times in 4 Posts
    OK, I hope in the future there are new updates, like the ability to add an option with only compression speed for each scanned file, instead of both options, in this mode, the scan will be even faster for files of large size.


  23. #16
    Member RamiroCruzo's Avatar
    Join Date
    Jul 2015
    Location
    India
    Posts
    15
    Thanks
    137
    Thanked 10 Times in 7 Posts
    Greetings,
    I was reading lzbench's readme on GitHub where you've said that you can't add more compressors because of unavailability of source code. You can just create an option for piped compressor where a user can define compression and decompression commands, of course, it'll only work for Softwares with stdio, still, it'll add a lot of more compressors in the benchmark.
    Not all the stars shine in the light. Some hide within the shadows.

  24. #17
    Member Samantha's Avatar
    Join Date
    Apr 2016
    Location
    italy
    Posts
    38
    Thanks
    31
    Thanked 7 Times in 4 Posts
    Hi @Ramiro, Well yes, it certainly is an idea, but I'd like to have support from @Inikep, so it can improve and expand lzb1.6 version with more options, or in any way get the possibility to test compressors not lz. I could get a perfect GUI able to test all the compressors, in an easy, fast and intuitive, at your disposal for all users, here or on FF.


  25. #18
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    Quote Originally Posted by Samantha View Post
    OK, I hope in the future there are new updates, like the ability to add an option with only compression speed for each scanned file, instead of both options, in this mode, the scan will be even faster for files of large size.
    I've added "--compress-only" option in https://github.com/inikep/lzbench/co...32405df9d6b05d.
    If you want to have a single compression call for each compressor you can use "--compress-only -t0".


    Quote Originally Posted by RamiroCruzo View Post
    You can just create an option for piped compressor where a user can define compression and decompression commands, of course, it'll only work for Softwares with stdio, still, it'll add a lot of more compressors in the benchmark.
    Testing executables has many disadvantages:
    - instead of testing memory-to-memory compression speed we are also taking input/output into consideration
    - executable usually will be compiled with a different compiler and can be compiled with different optimizations
    - do we want to keep executables within GitHub repo? (are we allowed?)

  26. The Following User Says Thank You to inikep For This Useful Post:

    Samantha (13th March 2017)

  27. #19
    Member Samantha's Avatar
    Join Date
    Apr 2016
    Location
    italy
    Posts
    38
    Thanks
    31
    Thanked 7 Times in 4 Posts
    Hi inikep you have read the P.M. ?


  28. #20
    Member Samantha's Avatar
    Join Date
    Apr 2016
    Location
    italy
    Posts
    38
    Thanks
    31
    Thanked 7 Times in 4 Posts
    Hi Inikep thank you bro, a small clarification if it was possible, with the "--compress -t0-only" option I get the message "Error" under name decompression, you could write "Disabled" instead of "Error", but does the same I will not disturb you any more time...


  29. #21
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    lzbench v1.7 has been released at https://github.com/inikep/lzbench/releases

    Changes:
    - added lizard v1.0 (formerly lz5)
    - libdeflate updated to v0.7
    - lzfse and lzvn updated to 2017-03-08
    - lzlib updated to v1.8
    - snappy updated to v1.1.4
    - xz updated to v5.2.3
    - zlib updated to v1.2.11
    - zstd updated to v1.1.4

  30. The Following 3 Users Say Thank You to inikep For This Useful Post:

    Cyan (18th March 2017),Jyrki Alakuijala (18th March 2017),Samantha (19th March 2017)

  31. #22
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    621
    Thanks
    182
    Thanked 225 Times in 138 Posts
    Quote Originally Posted by inikep View Post
    lzbench v1.7 has been released at https://github.com/inikep/lzbench/releases

    Changes:
    - added lizard v1.0 (formerly lz5)
    - libdeflate updated to v0.7
    - lzfse and lzvn updated to 2017-03-08
    - lzlib updated to v1.8
    - snappy updated to v1.1.4
    - xz updated to v5.2.3
    - zlib updated to v1.2.11
    - zstd updated to v1.1.4
    There is a nice improvement in brotli compression at mid to high qualities and large data (1 MB+). Consider updating it, too?

  32. #23
    Member Samantha's Avatar
    Join Date
    Apr 2016
    Location
    italy
    Posts
    38
    Thanks
    31
    Thanked 7 Times in 4 Posts
    excellent Inikep, I expecting an update, sorry curiosity, but why change the name from LZ5 a lizard... for what reason?


  33. #24
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    Quote Originally Posted by Samantha View Post
    why change the name from LZ5 a lizard... for what reason?
    https://encode.ru/threads/2721-Do-yo...uld-be-changed


    Quote Originally Posted by Jyrki Alakuijala View Post
    There is a nice improvement in brotli compression at mid to high qualities and large data (1 MB+). Consider updating it, too?
    I can do it but I was waiting for a new release at https://github.com/google/brotli/releases

  34. #25
    Member Samantha's Avatar
    Join Date
    Apr 2016
    Location
    italy
    Posts
    38
    Thanks
    31
    Thanked 7 Times in 4 Posts
    [QUOTE=inikep;52124]https://encode.ru/threads/2721-Do-yo...uld-be-changed

    Blheeee... long names do not like for compressors.....now I have to change part of the code for this nonsense...


  35. #26
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    lzbench v1.7.1 has been released at https://github.com/inikep/lzbench/releases
    I also updated results with the pareto frontier for compression speed/ratio and decompression speed/ratio at https://github.com/inikep/lzbench/bl...h171_sorted.md

    v1.7.1
    - brotli updated to 2017-03-10
    - glza updated to v0.8
    - lzma updated to v16.04

    v1.7
    - added lizard v1.0 (formerly lz5)
    - libdeflate updated to v0.7
    - lzfse and lzvn updated to 2017-03-08
    - lzlib updated to v1.8
    - snappy updated to v1.1.4
    - xz updated to v5.2.3
    - zlib updated to v1.2.11
    - zstd updated to v1.1.4

  36. The Following 2 Users Say Thank You to inikep For This Useful Post:

    Jyrki Alakuijala (20th March 2017),Samantha (19th March 2017)

  37. #27
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    2,965
    Thanks
    153
    Thanked 802 Times in 399 Posts
    Why lzlib can use fb273 and lzma can't?

  38. #28
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    Quote Originally Posted by Shelwien View Post
    Why lzlib can use fb273 and lzma can't?
    In lzbench I'm testing only predefined levels and lzma -9 is the highest level. I added results of lzma 16.04 -9.

  39. #29
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    2,965
    Thanks
    153
    Thanked 802 Times in 399 Posts
    I think its not enough, and its very misleading.
    lzip is lzma port, so it can't compress that much better than lzma.
    lzma defaults are clearly optimized for speed, and it never uses fb>64.
    Compare these:
    https://github.com/inikep/lzbench/bl...ssors.cpp#L568
    https://github.com/inikep/lzbench/bl.../LzmaEnc.c#L82

    I'd suggest explicitly setting fb273 at least for higher levels, in lzbench handler.
    Also, LzmaEncProps_Normalize most likely has to be completely replaced - atm it only uses two threads at level 5+.
    Higher lc could make sense in many cases (while it can't be changed in lzlib), etc.

    Btw, I've also been wondering about defining parameter profiles for these "levels" -
    is there some theory behind it? Like, exponential growth of some composite metric?

  40. #30
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    Why it's misleading? The executable of lzip -9 gives better compression than lzma -9.
    This is the same issue as supporting a large window for Brotli. It can be solved with:
    1. Modifying existing levels or adding new levels to compressors but it might be very confusing for users (IMHO not a good idea).
    2. Adding a support for advanced compression options ("lzbench -elzma,9:fb=273") but it requires a lot of work.

Page 1 of 2 12 LastLast

Similar Threads

  1. Replies: 23
    Last Post: 24th March 2018, 17:57
  2. Replies: 109
    Last Post: 29th August 2016, 20:40
  3. Silesia Open Source Compression Benchmark
    By Alexander Rhatushnyak in forum Data Compression
    Replies: 15
    Last Post: 22nd May 2016, 15:34
  4. In-memory open source benchmark of entropy coders
    By inikep in forum Data Compression
    Replies: 30
    Last Post: 14th October 2015, 09:36
  5. Open source JPEG compressors
    By inikep in forum Data Compression
    Replies: 8
    Last Post: 22nd October 2011, 00:16

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •