Page 1 of 2 12 LastLast
Results 1 to 30 of 58

Thread: LzTurbo Byte coding vs. Zhuff FSE

  1. #1
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    456
    Thanks
    46
    Thanked 164 Times in 118 Posts

    LzTurbo Byte coding vs. Zhuff FSE

    LzTurbo with byte coding beats Zhuff with Huffman/FSE
    Decompression is
    4 to more than 12 times faster.

    Single core in memory benchmarks with i7-2600K CPU at 3.4GHz.
    Corpus from:
    http://compressionratings.com/
    Code:
                        size     ratio    C MB/s     D MB/s     
    app1.tar          141049191  28.0     316.40    2210.77    lzturbo 21 
    app3.tar           52026869  52.0     231.39    1817.04    lzturbo 21 
    app4.tar          129543129  31.7     239.01    1730.35    lzturbo 21 
    audio1.tar        105295850  99.4     414.27    4952.80    lzturbo 21
    cr_txt1            46960579  47.0     152.80    1204.00    lzturbo 21 
    enwik8             42537930  42.5     157.94    1166.46    lzturbo 21 
    game1.tar         280591204  40.0     208.73    1668.39    lzturbo 21
    game2.tar         265484767  43.8     248.74    1970.52    lzturbo 21
    img1.tar           64282288  70.6     170.99    1887.94    lzturbo 21
    src1.tar           26717627  26.4     253.91    1712.09    lzturbo 21
    ubuntu500m.tar    163795024  32.8     227.52    1665.55    lzturbo 21    
    Total:           1318284458
    
    
    app1.tar          167730707  33.3     213.4      460.3     zhuff v0.9c
    app3.tar           53083567  53.0     163.2      345.0     zhuff v0.9c
    app4.tar          154295202  37.8     171.9      235.4     zhuff v0.9c
    audio1.tar         98528678  93.1     167.3      271.1     zhuff v0.9c
    cr_txt1            43099756  43.1     138.1      270.5     zhuff v0.9c
    enwik8             40643947  40.6     145.5      293.1     zhuff v0.9c
    game1.tar         276499255  39.5     171.8      353.1     zhuff v0.9c
    game2.tar         251632184  41.5     178.9      388.2     zhuff v0.9c
    img1.tar           54985329  60.4     122.3      302.2     zhuff v0.9c
    src1.tar           28318301  28.0     208.7      401.3     zhuff v0.9c
    ubuntu500m.tar    178265751  35.7     180.8      150.2     zhuff v0.9c
    Total:           1347082677

    Last edited by dnd; 19th January 2014 at 23:02.

  2. #2
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    Does lzturbo use any preprocessors/ filters/ special models?

  3. #3
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    lzturbo uses STOLEN tornado codebase plus its own, buggy and slow optimal parser. afair, it was up to 100 times slower than original code in independent tests

    as usual, hamidi compares apples to oranges - zhuff employs only one thread. as one can see in the CR benchmarks, 4x4 or nanozip easily outperform that piece of stolen code
    Last edited by Bulat Ziganshin; 19th January 2014 at 17:35.

  4. #4
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Bulat, he did not steal it. At most, he did not comply with your license.

    dnd, what lzturbo is this? Can you provide a link to download, so your claim can be independently verified?

  5. #5
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    m^2, what is the difference?

    hamidi removes previous versions of his program and prohibits anonymous downloads in order to complicate analysis of stolen things. but as we previously found, he uses matchfinders and entropy coders from my program, up to exact contents of len/dist encoding tables

  6. #6
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    456
    Thanks
    46
    Thanked 164 Times in 118 Posts
    - Does lzturbo use any preprocessors/ filters/ special models?
    Only e8e9 filter for ".exe", ".dll" and ".tar" files.

    - link to download LzTurbo: https://sites.google.com/site/powturbo/

    I don't want to start again the eternal discussion. Everybody can see from the benchmarks that Lzturbo has nothing to do with Tornado.
    Bulat, please go watch TV or go search for a girl friend.

  7. #7
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Bulat, what's the similarity?

    Come on, these 2 things are extremely different. To steal is to take away. There's loss. That's much more than to act against one's will even if in most countries, civil law gives this will some extraordinary power.

  8. #8
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    it's not the discussion - that's the OBVIOUS thing. the first version of lzturbo has EVERYTHING copied from the tornado 0.1 plus the optimal parser. It was buggy and slow, are you remember? It shows your programming level. Noone understanding LZ can believe that you have developed exact copy of tornado but was unable to provide optimal parser of reasonable quality. later you continued to copy code from new versions of tornado, releasing new lzturbo versions in a few days after tornado releases. it is how lzturbo got huffman encoder, for example

    - link to download LzTurbo: https://sites.google.com/site/powturbo/

    i tried several times and never succeed. that shows the difference - i publish my sources and you prohibit me even to download your compiled code since you fear the disassembly analysis

  9. #9
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    m^2, without idea of intellectual property you will never see the films, books or software except those written and filmed by enthusiasts. stealing IP made me lost money that i can got by licensing the tornado sources

  10. #10
    Member
    Join Date
    Jun 2008
    Location
    G
    Posts
    372
    Thanks
    26
    Thanked 22 Times in 15 Posts
    hm but i can download lzturbo, neednt to register

  11. #11
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    Attaching the only download that is available from the site for anyone who can't download it from the Google pages.
    Attached Files Attached Files
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  12. The Following User Says Thank You to Black_Fox For This Useful Post:

    Bulat Ziganshin (19th January 2014)

  13. #12
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    m^2, without idea of intellectual property you will never see the films, books or software except those written and filmed by enthusiasts.
    Which is not a problem for me. I avoid others anyway. Which doesn't change the fact that many of those enthusiast actually make money from their creations.
    Quote Originally Posted by Bulat Ziganshin View Post
    stealing IP made me lost money that i can got by licensing the tornado sources
    The following assumes that dnd indeed took your code. I want to make it clear that I'm not sure it's true, though I believe it is.
    You seem to assume that if dnd hadn't taken the code that you have written for free, he would have paid for it. Which ignores several other, legal possibilities:
    * he would have published lzturbo sources
    * we would have taken some other code
    * we would have written lzturbo from scratch
    * he wouldn't have written lzturbo at all

    Reminds me of bankers who just after giving you a credit already lend the profits they are to take.
    In case of banks the scale is large enough for statistics to work, so they can model their losses and still it proved to be highly unstable.
    Last edited by m^2; 19th January 2014 at 18:37.

  14. #13
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by thometal View Post
    hm but i can download lzturbo, neednt to register
    try to sign out of the google account and then download again. it doesn't need to regitser, hamidi just prohibited to download either anonymously or from my account. of course, i can find another one - it just shows how he fear that i will analyze his code

  15. #14
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    m^2, in this case you should leave this forum since it was developed by a commercial company and was unable without selling the licenses. and i can't beleive that you really never seen any of Hollywood films

    i don't expect that he will pay for anything, i mean that he SELLS my code. and that he make clumsy assistance for all real iranian programmers trying to sell results of their work

  16. #15
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by m^2 View Post
    The following assumes that dnd indeed took your code. I want to make it clear that I'm not sure it's true
    just recall the first lzturbo version. what was the only difference against tornado 0.1, published a few monthes ago? optimal parser. i have wrote better one in a 3 days, it is much simpler than master a multitude of match finders and encoders. and even that part of lzturbo was slow and buggy, he was fixing it more than a year. i think this tells enough about his programming expertise. nevertheless, the rest of lzturbo code was quick and stable from the first version

    at the March 2008 i released tornado 0.4 that was added Huffman coder. In just a week or so it was implemented by lzturbo, and it got two new versions in quick succession although there were no releases in a half-year. once again - he don't get any troubles inplementing Huffman coder while was still uable to debug his 6 monthes-old optimal parser. It shows a clear distinction between his own and stealed code
    Last edited by Bulat Ziganshin; 19th January 2014 at 19:13.

  17. #16
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    456
    Thanks
    46
    Thanked 164 Times in 118 Posts
    Bulat, instead of spreading lies please show to the world where is your optimal parser. UNTIL TODAY, THERE IS NO OPTIMAL PARSER IN TORNADO!!!!

    I have already published the source code of the huffman coder and the match finders used in the previous versions of lzturbo, but you are still ignoring the reality.

    Your Tornado is GPL, so you can't prevent somebody to sell the code, if he want to do that.

    If you are a better programmer, then try to outperform the current version of lzturbo.

  18. #17
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    m^2, in this case you should leave this forum since it was developed by a commercial company and was unable without selling the licenses. and i can't beleive that you really never seen any of Hollywood films
    I said I avoid products that were not created by enthusiasts. There are 2 things about that. First, enthusiasm is orthogonal with commerce. Second, I avoid, but when it conflicts with other interests, I make a pick. In this case, I chose to participate because I do like this place. If this forum moved to a google group, I would leave though.
    When it comes to modern Hollywood movies, I watch c.a. 1/year; only when something has super-positive reviews. Often I'm disappointed. Though I don't watch many movies in general, maybe 1/month, maybe even less.

    Quote Originally Posted by Bulat Ziganshin View Post
    i don't expect that he will pay for anything, i mean that he SELLS my code. and that he make clumsy assistance for all real iranian programmers trying to sell results of their work
    dnd is Iranian? Because there's no copyright for foreign works in Iran. Selling closed, GPL-derived code from abroad would be an interesting case study. IANAL, but I believe that it would be legal, though users would have to comply with GPL, as it fits in their local law.

    As to whether dnd indeed used Tornado:
    Like I said, I believe you. I haven't been here at the time yet, but I've read the old discussions. But I don't think I've seen a nice investigation with comparison of assembly; Some shared constants come to my mind, but they turned out to be common ones.
    Frankly, the form in which you remind about the issue is not a very good one. There's an accusation and no immediately available proof. There's a lot of personal remarks. I suggest that you collect your evidence, put together in one place and link to it whenever dnd shows up.
    Last edited by m^2; 19th January 2014 at 19:20.

  19. #18
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    hamidi, how i can outperform the program i can't download?

    you confirmed that you stole my code, first by using my set of entropy coders, later by saying that everyone can use Huffman coder. now you says that you can sell my GPL code - so you mean that you really do it. you have not published sources of huffman coder used in lzturbo - just pointed out to one of millions of huffman coders around. the same is with match finder - you have published variant of my own code. it's easy to check that this match finder was used only by two programs around. it will be easy to prove that i lie by publishing full code of lzturbo 0.0.1 but you will never do it. so please stop lyiong - you stealed my code and trying to sell it

  20. #19
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    456
    Thanks
    46
    Thanked 164 Times in 118 Posts
    AGAIN!!! Please show to the world where is your optimal parser. UNTIL TODAY, THERE IS NO OPTIMAL PARSER IN TORNADO!!!!

    Look at the reality:
    http://encode.ru/threads/1722-lzturb...ll=1#post33136

    and
    http://encode.ru/threads/1722-lzturb...ll=1#post33137
    Last edited by dnd; 19th January 2014 at 19:27.

  21. #20
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    hamidi, please show us the sources of lzturbo 001. otherwise i can't stop telling that you have stealed my entire program and sell it

    optimal parser
    oh, sorry, i just need to add some comments in order to help you to steal even more of my code

  22. #21
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    456
    Thanks
    46
    Thanked 164 Times in 118 Posts
    AND AGAIN!!! Please show to the world where is your optimal parser. UNTIL TODAY, THERE IS NO OPTIMAL PARSER IN TORNADO!!!!

  23. #22
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    the former is exactly tornado matchfinder. there is no other programs using the same idea. so either you have invited the same thing as me in the same time, and even give it the same name, or ...

    the later is just a huffman coder. you need to prove that it was used by the first lzturbo version with huffman, release in April 2008, otherwise it can't prove anything. are you agree with me?

  24. #23
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by dnd View Post
    AND AGAIN!!! Please show to the world where is your optimal parser. UNTIL TODAY, THERE IS NO OPTIMAL PARSER IN TORNADO!!!!
    it seems that you believe that optimal coder is hard to develop. how much time you spend developing OP for lzturbo 001? it was month or two?

  25. #24
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    456
    Thanks
    46
    Thanked 164 Times in 118 Posts
    M^2 thanks for your fairness.
    Bulat seems not understanding, please ask him where to find the "optimal parser" in Tornado. UNTIL TODAY, THERE IS NO OPTIMAL PARSER IN TORNADO!!!!
    You can't steal something that's not available.

  26. #25
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    hamidi, it is why your OP is so slow? you need to steal a faster one because you can't write it yourself? may be i should write some manual to help you stealing my code? sorry, i don't know Iranian

  27. The Following User Says Thank You to Bulat Ziganshin For This Useful Post:

    Black_Fox (22nd January 2014)

  28. #26
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    dnd:
    Can you disable the filters and redo the tests?

    Also what about CPU times in addition to real times?

    From what I understood, zhuff is just an afterthought (ie a LZ4 with Huffman/ FSE) and more like experimentation playground than a serious competitor. Thus, to ensure apples to apples comparison you should consider my requests.

  29. #27
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    lz4/zhuff clearly outperforms tornado (with bytecoder and hufcoder, respectively) in the terms of speed. you can see it from hamidi tests - he compares single-threaded zhuff with 8-threaded lzturbo. by dividing lzturbo results by 5, you can find that it compresses much slower than zhuff, and decompression is faster because he compared bytecoder in lzturbo with huffman coder in zhuff

    tornado easily outperfroms lz4/zhuff in the terms of compression ratio since it supports dictionaries up to 1GB. of course this gives us slower compression and slightly faster decompression
    Last edited by Bulat Ziganshin; 19th January 2014 at 22:47.

  30. #28
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    456
    Thanks
    46
    Thanked 164 Times in 118 Posts
    Well there is no big difference. These are in memory benchmarks (No disk I/O is involved) without multithreading.
    The option -b0 -t1 is used for zhuff, the tests are repeated 5 times and only the best timings are reported.

    Hier the tests for lzturbo without any filter and without multithreading.
    Code:
                           size      ratio    C MB/s     D MB/s      
    app1.tar           147863826     29.4     308.46    2180.71    lzturbo 21
    app3.tar            52812595     52.8     192.08    1793.70    lzturbo 21 
    app4.tar           133045756     32.6     222.41    1729.11    lzturbo 21 
    audio1.tar         105295578     99.4     284.64    4924.25    lzturbo 21 
    cr_txt1             46960579     47.0     150.25    1203.20    lzturbo 21 
    enwik8              42537930     42.5     153.95    1166.17    lzturbo 21 
    game1.tar          279854797     39.9     209.22    1668.98    lzturbo 21 
    game2.tar          264914277     43.7     225.23    1975.44    lzturbo 21 
    img1.tar            64275996     70.6     149.44    1802.41    lzturbo 21 
    src1.tar            26714171     26.4     243.45    1482.85    lzturbo 21 
    ubuntu500m         168946014     33.8     217.57    1656.76    lzturbo 21 
    Total:            1333221519
    Note that there are also the methods 30,31,32 and 39 in LzTurbo with much better compression.

    Bulat, please go watch TV! All tests are performed WITHOUT multithreading.
    Last edited by dnd; 19th January 2014 at 22:54.

  31. #29
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    hamidi, you not mentioned in the first post until now that tests are single core. so how i can check it? so it seems that after 6 years of development you have mastered second piece of your own code. my congrats! how many years we should wait before you will stop to violate german copyright laws and will publish sources of all lzturbo versions using my code and ideas stolen from my code? this includes all the forms of HT matchfinders, stolen arithmetic, huffman and bitwise encoders. so you can sell lzturbo that has only bytecoder+optimal matchfinder and it will not break my IP. everything else in lzturbo is mine

  32. #30
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    I've done quick tests on enwik8 and it seems that dnd's results are legit and he tests on single core. However, there may be some catch - Zhuff has small sliding window size while lzturbo may use bigger, thus achieve good compression without breaking a sweat on optimal parsing and good modelling. I've tested mode 30 in lzturbo and it lost it speed advantage while providing better compression. Curious to see the sliding windows compared.

Page 1 of 2 12 LastLast

Similar Threads

  1. Zhuff - fast compression
    By Cyan in forum Data Compression
    Replies: 38
    Last Post: 5th February 2014, 11:27
  2. Fast 32-byte aligned word histogram calculation
    By Intensity in forum Data Compression
    Replies: 2
    Last Post: 28th December 2013, 22:14
  3. LZTURBO
    By dnd in forum Data Compression
    Replies: 28
    Last Post: 24th May 2013, 22:13
  4. Traffic compression (1500 byte chunks)
    By blackhaz in forum Data Compression
    Replies: 4
    Last Post: 21st January 2012, 22:44
  5. Byte oriented LZ77 compressor
    By Matt Mahoney in forum Data Compression
    Replies: 21
    Last Post: 30th December 2011, 18:27

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •