Results 1 to 5 of 5

Thread: LZ5 v2.0 beta

  1. #1
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 166 Times in 63 Posts

    Lightbulb LZ5 v2.0 beta

    LZ5 v2.0 beta can be downloaded from https://github.com/inikep/lz5/releases

    Changes:
    - this version is optimized for decompression speed (instead of ratio for previous versions)
    - levels from 1 to 9 use LZ4-type codewords and give about 10% better decompression speed than LZ4
    - levels from 10 to 18 use new codewords with decompression speed about 2 times faster than LZ5 v1.5

    The results below are performed using the same laptop with the same compiler (gcc 6.2.0 MinGW-w64):
    Code:
    LZ5 v2.0 beta compiled from latest "lz5_v2.0" branch:
     1#silesia_tar       : 211947520 -> 103402971 (2.050), 346.5 MB/s ,2577.0 MB/s
     2#silesia_tar       : 211947520 ->  93861621 (2.258), 251.0 MB/s ,2412.5 MB/s
     3#silesia_tar       : 211947520 ->  89443564 (2.370), 118.0 MB/s ,2430.6 MB/s
     4#silesia_tar       : 211947520 ->  86646242 (2.446),  72.0 MB/s ,2480.6 MB/s
     5#silesia_tar       : 211947520 ->  83975809 (2.524),  58.6 MB/s ,2525.3 MB/s
     6#silesia_tar       : 211947520 ->  81223047 (2.609),  56.2 MB/s ,2521.2 MB/s
     7#silesia_tar       : 211947520 ->  79381197 (2.670),  40.7 MB/s ,2402.4 MB/s
     8#silesia_tar       : 211947520 ->  78021788 (2.717),  19.5 MB/s ,2448.3 MB/s
     9#silesia_tar       : 211947520 ->  77588075 (2.732),   2.8 MB/s ,2460.8 MB/s
    10#silesia_tar       : 211947520 ->  89799372 (2.360), 171.1 MB/s ,1749.6 MB/s
    11#silesia_tar       : 211947520 ->  85225312 (2.487), 107.2 MB/s ,1674.6 MB/s
    12#silesia_tar       : 211947520 ->  82302947 (2.575),  33.5 MB/s ,1777.8 MB/s
    13#silesia_tar       : 211947520 ->  79328026 (2.672),  21.8 MB/s ,1748.3 MB/s
    14#silesia_tar       : 211947520 ->  75818444 (2.795),  13.1 MB/s ,1699.0 MB/s
    15#silesia_tar       : 211947520 ->  74584415 (2.842),   8.3 MB/s ,1684.3 MB/s
    16#silesia_tar       : 211947520 ->  72979690 (2.904),   4.5 MB/s ,1714.6 MB/s
    17#silesia_tar       : 211947520 ->  70500121 (3.006),   3.0 MB/s ,1620.6 MB/s
    18#silesia_tar       : 211947520 ->  68539649 (3.092),   1.5 MB/s ,1502.5 MB/s
    
    LZ4 v1.7.2 compiled from latest "dev" branch:
     1#esty\silesia_tar : 211947520 -> 100923164 (47.62%),  437.9 MB/s , 2265.7 MB/s
     2#esty\silesia_tar : 211947520 -> 100923164 (47.62%),  435.2 MB/s , 2254.8 MB/s
     3#esty\silesia_tar : 211947520 ->  82428029 (38.89%),   66.1 MB/s , 2150.8 MB/s
     4#esty\silesia_tar : 211947520 ->  80589674 (38.02%),   54.7 MB/s , 2201.5 MB/s
     5#esty\silesia_tar : 211947520 ->  79427759 (37.48%),   45.1 MB/s , 2209.9 MB/s
     6#esty\silesia_tar : 211947520 ->  78747219 (37.15%),   37.3 MB/s , 2235.3 MB/s
     7#esty\silesia_tar : 211947520 ->  78354189 (36.97%),   31.0 MB/s , 2248.2 MB/s
     8#esty\silesia_tar : 211947520 ->  78140269 (36.87%),   26.3 MB/s , 2239.6 MB/s
     9#esty\silesia_tar : 211947520 ->  78036683 (36.82%),   23.8 MB/s , 2243.9 MB/s
    
    LZ5 v1.5.1 compiled from latest "lz5_v1.5" branch:
     1#silesia_tar       : 211947520 -> 113538427 (1.867), 428.7 MB/s ,1304.2 MB/s
     2#silesia_tar       : 211947520 -> 104082573 (2.036), 346.6 MB/s ,1251.9 MB/s
     3#silesia_tar       : 211947520 ->  95575943 (2.218), 261.6 MB/s ,1129.6 MB/s
     4#silesia_tar       : 211947520 ->  86503541 (2.450), 149.2 MB/s , 957.5 MB/s
     5#silesia_tar       : 211947520 ->  84460413 (2.509), 115.0 MB/s , 884.6 MB/s
     6#silesia_tar       : 211947520 ->  80575745 (2.630),  42.3 MB/s , 972.8 MB/s
     7#silesia_tar       : 211947520 ->  78854960 (2.688),  37.2 MB/s , 940.9 MB/s
     8#silesia_tar       : 211947520 ->  77996518 (2.717),  27.3 MB/s , 915.4 MB/s
     9#silesia_tar       : 211947520 ->  74228639 (2.855),  20.3 MB/s , 917.8 MB/s
    10#silesia_tar       : 211947520 ->  72267610 (2.933),  13.4 MB/s , 916.6 MB/s
    11#silesia_tar       : 211947520 ->  71056249 (2.983),   9.7 MB/s , 824.1 MB/s
    12#silesia_tar       : 211947520 ->  69485691 (3.050),   7.6 MB/s , 837.8 MB/s
    13#silesia_tar       : 211947520 ->  67937025 (3.120),   4.5 MB/s , 864.9 MB/s
    14#silesia_tar       : 211947520 ->  65805781 (3.221),   2.9 MB/s , 851.7 MB/s
    15#silesia_tar       : 211947520 ->  65555479 (3.233),   1.6 MB/s , 868.3 MB/s

  2. The Following 10 Users Say Thank You to inikep For This Useful Post:

    Bulat Ziganshin (1st October 2016),Christoph Diegelmann (2nd October 2016),comp1 (2nd October 2016),encode (19th November 2016),JamesB (2nd October 2016),Mike (2nd October 2016),milky (3rd October 2016),tobijdc (2nd October 2016),Turtle (8th October 2016),xcrh (19th October 2016)

  3. #2
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 166 Times in 63 Posts
    Do you have any comments on v2.0 beta?

  4. #3
    Member
    Join Date
    Nov 2015
    Location
    boot ROM
    Posts
    83
    Thanks
    25
    Thanked 15 Times in 13 Posts
    I've stumbled on it recently but only had time to do several quick runs only on x86-64 laptop (i.e. no fun with ARM things and so on) and only got overall impression. On x86_64 its speed looks impressive, but it seems achievable ratio has suffered a bit. Also, it comes with quite unusual bitstream format. Am I correct that decompression basically looks like this: get size of stream - go to current + size, get size of next stream, etc, etc for all 5 streams? Does such design leads to better speeds? (and why? better cpu cache hits?)

    I've also spotted some few issues/inconsistences. Do you prefer 'em to get listed here or filed on github as "issues"?

    Also I have question: do you have some plans to add lz5 v2 to lzbench? I think I've mostly understood what it takes to add new codec to lzbench (and since LZ5v2 differs from 1.5 considerably IMHO it better to be new codec?), so I guess I could try something like that, but I would dislike if it interferes with your own plans and I'm not exactly sure I could do it properly on first try, I could miss something for sure :P.

  5. #4
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 166 Times in 63 Posts
    Quote Originally Posted by xcrh View Post
    Also, it comes with quite unusual bitstream format. Am I correct that decompression basically looks like this: get size of stream - go to current + size, get size of next stream, etc, etc for all 5 streams? Does such design leads to better speeds? (and why? better cpu cache hits?)
    The speed is similar to 1 stream but it helps for Huffman encoding. You can try it using "make lz5huf" and then "lz5huf -l3".


    Quote Originally Posted by xcrh View Post
    Do you prefer 'em to get listed here or filed on github as "issues"?
    Github should be better.


    Quote Originally Posted by xcrh View Post
    Also I have question: do you have some plans to add lz5 v2 to lzbench?
    Sure, I will add lz5v2 to lzbench. I was thinking about replacing v1.5 with v2.0. Do you think that I should keep also v1.5?

  6. #5
    Member
    Join Date
    Nov 2015
    Location
    boot ROM
    Posts
    83
    Thanks
    25
    Thanked 15 Times in 13 Posts
    Quote Originally Posted by inikep View Post
    The speed is similar to 1 stream but it helps for Huffman encoding. You can try it using "make lz5huf" and then "lz5huf -l3".
    I've seen some huff-related things, but straightforward attempts to use it lead me to few things which are probably bugs

    Github should be better.
    Ok, then I'll go here.

    Sure, I will add lz5v2 to lzbench. I was thinking about replacing v1.5 with v2.0. Do you think that I should keep also v1.5?
    After few quick runs I've got idea LZ5v2 offers somewhat different tradeoffs, being a bit less dense but considerably faster (at least on my x86_64 laptop and few desktops; I haven't ran tests on ARM things yet, etc). So I've got idea it could be interesting to run 'em side by side.

Similar Threads

  1. Replies: 69
    Last Post: 16th August 2016, 18:46
  2. My examples like lz4 & lz5
    By lz77 in forum Data Compression
    Replies: 4
    Last Post: 17th February 2016, 15:23
  3. PIM v2.50 is here!
    By encode in forum Data Compression
    Replies: 6
    Last Post: 24th July 2008, 14:43
  4. PIM v2.41 BETA is here!
    By encode in forum Data Compression
    Replies: 33
    Last Post: 19th July 2008, 22:11
  5. PrePAQ v2 (aka paq8o8pre v2)
    By schnaader in forum Forum Archive
    Replies: 10
    Last Post: 18th January 2008, 17:38

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •