Results 1 to 27 of 27

Thread: LZTURBO 0.91 Parallel Compressor (Win32/Linux)

  1. #1
    Member
    Join Date
    Aug 2007
    Posts
    30
    Thanks
    0
    Thanked 0 Times in 0 Posts
    New version & Benchmarks: lzturbo 0.91 parallel compressor
    at: http://consultant-berater.de/lzturbo

    - Compress better & faster (Levels 1..
    - (Level 9 minor changes in speed)

    Please compare lzturbo to thor, tornado, slug, quicklz. Specially on newer hardware (like core 2 duo).
    ------- SINGLE CORE WAS YESTERDAY -------

    Please read lzturbo.txt.
    Please read the benchmarks at:
    http://consultant-berater.de/lzturbo for the proper switches to choose.

    Enjoy!

  2. #2
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Hamid!

  3. #3
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quick (-59) test...

    A10.jpg > 839,071
    AcroRd32.exe > 1,417,442
    english.dic > 687,644
    FlashMX.pdf > 3,736,621
    FP.LOG > 789,021
    MSO97.DLL > 1,810,448
    ohs.doc > 833,243
    rafale.bmp > 1,006,476
    vcfiu.hlp > 677,425
    world95.txt > 576,470

    Total = 12,373,861 bytes

  4. #4
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    MOC Test
    option -59 -> 159.922.717 b comp. 465,715 s. - dec. 40,782 s.

  5. #5
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Another quick test...

    Test machine: Intel PIII (Coppermine) @750 MHz, 512 MB RAM, Windows 2000 Pro SP4

    Test File: ENWIK9 (1,000,000,000 bytes)

    Timed with AcuTimer v1.2


    Compression settings: -11

    Compressed size: 430,453,227 bytes
    Elapsed Time: 00:02:05.594 (125.594 Seconds)


    Compression settings: -12

    Compressed size: 375,431,333 bytes
    Elapsed Time: 00:02:40.155 (160.155 Seconds)

  6. #6
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    301
    Thanks
    26
    Thanked 22 Times in 15 Posts
    great work..10x thanks

  7. #7
    Member
    Join Date
    Aug 2007
    Posts
    30
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thanks LovePimple, Nania & Maadjordan.

    Updated new version & benchmarks: lzturbo 0.92
    at: http://consultant-berater.de/lzturbo
    (Level 9 unchanged).

    Nania Francesco Antonio
    Thanks for testing. The decompressor from 0.9 code is unchanged
    in this version, but you are reporting other timings in your quick-test (40s for 0.91 versus 7s for 0.9).

  8. #8
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Hamid!

  9. #9
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Unfortunately this program had given me problems with the test and evidently has to be us an error in the precedents effected test! Have had to rewrite the code of the program of test for every type of compressor! I have referred the MOC test and the result it is the same.

  10. #10
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by donotdisturb
    ------- SINGLE CORE WAS YESTERDAY -------
    Suddenly I feel sooo old...
    Do I have to bury my AMD 1700+ now?


    Best regards though!

  11. #11
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    Quote Originally Posted by donotdisturb
    ------- SINGLE CORE WAS YESTERDAY -------
    Not sure bout that. I think, mainly, all your noise about multi-core systems is no more than just LZTURBO advertisment. The power of dual/quad/... core systems applying to the data compression is overestimated, generally speaking. Maybe the coolest part is that you can compress something and at the same time do something another with minor performance loss - i.e. browsing the web, whatching movies... Using two or more cores not means that youll getting 2X-4X... speed boost - as you know, and especially for LZ-based coders, the main bottleneck is the memory access due to string searching. So LZ77 benefits more from faster memory/cache - not faster CPU. With LZTURBO, I suggest, instead of a TORNADO clone a la "lots of stuff" inside, make some LZMA clone with greater performance - i.e. higher compression/faster decompression... It whould be interesting - since the main purpose of any experimental compressor is to introduce something new, new idea, new approach, new algorithm...
    Recently, got a new GloFiish SmartPhone/Communicator (Or how to call the unit which has ALL stuff, including GPS,Wi-Fi,Camera,even FM-transmitter... Crazy device). And saw how efficient Deflate (ZIP) is, under Windows Mobile 6. Hm, maybe Ill write something BALZ-based for WM... PocketBALZ, BALZ in the Pocket, or whatsoever... PocketLZ... The first experimental file compressor for Pocket PC. Sounds great! Does anyone have SmartPhone/PocketPC/... just to see, are anyone able to test the stuff... You know what Im sayin...

  12. #12
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    @encode
    I?m irritated to read such a post from a developer of many compression programs and also how you write it. There aren?t many programs availible yet, taking advantage of multicore systems thats right, but if you see 7zip for example it is much faster using threads there. With more and more cores the advantage isn?t so big and also the hard drive could be a second slow down (but raid systems going to be common more and more which can speed up the access).
    At the moment the multicore systems almost only have the advantage you mentioned (several processes) but on XP this doesn?t work as good as it should (own experience).

    I thank donotdisturb to take this step because most developers only seem to be to lazy to implement a thread algorithm. Don?t you think it could speed up the slow compression time of balz?

  13. #13
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    I just have my own opinion, which can be straight - just as I think. I hope youre correctly understand my position.

    Quote Originally Posted by Simon Berger
    Don?t you think it could speed up the slow compression time of balz?
    As I noted, mostly, the time taken by memory access, that explains the great difference in performance on my and Matts (LTCB) PC. I may greatly improve the compression speed, switching match finder to binary trees, instead of a hash chains, someone may say that using hash chains with matching from each byte is lame - at some point, yes it is. Hash chains are very efficient with fast, unoptimized parsing - with greedy/lazy matching. However, currently I use this approach, even I know that it isnt optimal solution.
    Later will play with Multi-Threaded things, but as I posted, MTs benefits are overestimated by the users, I just understand there is no magic. Multi PC compression is more advanced idea - combining the power of a few PCs together, syncronizing it via USB/LAN/Wi-Fi, since each unit has independed memory/cache and CPU, we may achive some interesting performance... And here, two cores but physically single PC - one motherboard, one or two memory channel(s), etc. Im very sceptic.

  14. #14
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by encode
    all your noise about multi-core systems is no more than just LZTURBO advertisment
    HW manufacturers produce quad-core CPUs as a mainstream (and already there are plans for hexa-core at Intel and octal-core at AMD) and even GPUs are using higher number of chips, but I bet you are aware of that and just mean another thing: Programs performance cannot magically scale just because more threads are available, there have to be some changes on the code part. Games will be paralelized in some years dramatically, as CPUs/GPUs will be able to render raytracing in real-time, some programs also as well. However, there is little to speed up in compression world - only thing, that can be done, is splitting input into more parts, which degrades ratio Is there any theoretical chance, that (almost perfectly) scalable algorithm will be ever found?

  15. #15
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    772
    Thanks
    63
    Thanked 270 Times in 190 Posts

  16. #16
    Guest
    Code:
     
    4Gb Game (45653 files) 
    Core2 1.67 , DDR2-666 2Gb 
     
    Ratio   //   Comp.   //  Decomp.          // Archiver 
    35.043% //  1420kb/s // 18095kb/s(no out) // 7-zip 4.56 Ultra -d110m 
    43.240% //  6407kb/s // 17929kb/s         // Tornado 0.3 -6 (stored 7z) 
    43.629% //  8059kb/s // 18179kb/s         // Tornado 0.3 -5 (stored 7z) 
    44.223% //  8387kb/s // 16561kb/s         // Lzturbo 0.92 -44 (stored 7z) 
    45.026% //  4587kb/s //  8590kb/s         // Lzturbo 0.92 -55 (stored 7z) 
    46.404% // 12942kb/s // 18653kb/s         // Tornado 0.3 -4 (stored 7z) 
    46.631% // 10205kb/s // 16053kb/s         // WinRar v3.71 Fastest (stored 7z) 
    50.938% // 17449kb/s // 19740kb/s         // Tornado 0.3 -3 (stored 7z)

    Hdd limit is somewhere ~17mb/s so keep in mind.

  17. #17
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    Quote Originally Posted by Black_Fox
    HW manufacturers produce quad-core CPUs as a mainstream (and already there are plans for hexa-core at Intel and octal-core at AMD)
    Manufacturers just desperate to grab the last cent from our pocket...

  18. #18
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Zonder
    for tornado yoo can use -o option (w/o parameter) to (de)compress into null

  19. #19
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by Simon Berger
    There aren?t many programs availible yet, taking advantage of multicore systems thats right
    i wonder whether you really use any single-file compressors? my opinion is that programs published here are mostly research in compression technologies rather than really useful products. they usually dont include any multithreading technologies exactly because its trivial to implement. imho, for real usefulness the program should be either an archiver or compressor that supports linux/windows, crc checking and stdin-to-stdout mode

    what are opinions of other readers? does such programs have real applications? if you use/want to use standalone compressors, what you need from such programs?

  20. #20
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Quote Originally Posted by Bulat Ziganshin
    i wonder whether you really use any single-file compressors? my opinion is that programs published here are mostly research in compression technologies rather than really useful products.
    You are right on this point but I will ask you a counter question. Do you try to create a compression engine which isn?t competitive and slow at all?
    Multi threading techniques are the future (what we can foresee from now) because there going to come many more cores. I also don?t like programming with threads but I learn and do it because it?s not trivial to implement them and you will need them more and more because the market will ask more and more
    I don?t see your point why a "fun project" and research shouldn?t include multi threading. It isn?t trivial to change a algorithm to multi threading and sometimes impossible (to get a good from it).

  21. #21
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by Simon Berger
    Do you try to create a compression engine which isn?t competitive and slow at all?
    try to compare lzturbo and tornado pure compression power. force lzturbo to single-thread mode and look at "user time" printed by Timer

    Quote Originally Posted by Simon Berger
    I also don?t like programming with threads but I learn and do it because it?s not trivial to implement them
    its easy, you just need to use proper tools/techniques. freearc runs multiple threads just by using unix-style notation:

    reader |> filter |> filter |> compressor |> writer

    as i already said, we dont add MT techniques to our fun projects exactly because its trivial to compress multiple data blocks in parallel. OTOH, lzturbo is aimed to practical usage and its natural to add all those bells and whistles here. it will be even better to provide MT environment just around proven tornado core, but i cant force him to do it

    tor0.1 split data into independent chunks and compressed them separately. it probably becomes source of inspiration for Hamid who added MT features to my core. lzturbo 0.91 was probably
    upgraded to tor03 core but still uses independent chunks

    OTOH, implementing multicore compression *engine* isnt trivial task. its why they are so rare and of course we all agree that such researches will be highly useful

  22. #22
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    i need to add that only invention of fast and memory-efficient match finder in tornado 0.1 made it possible lzturbo to appear. if it used lzma compression engine, its memory reqs will be 6-8x larger!!!

  23. #23
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    I think there's some suboptimality in LZTurbo's optimal parsing. I have a tar archive, which is very compressible (7z compresses it from 63 MB to 426 kB)... it took 3 hours to compress with PAQ8o9 (using one core) and 2 hours for latest LZTurbo using both cores...

  24. #24
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    i'm pretty sure that reason is the naive binary tree implementation which makes lzturbo extremely slow on any binary data. try cabarc on this file - it was also naive otoh, Hamid optimizes lzt for enwik9, so i don't expect that he will make any changes for other datatypes

  25. #25
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Good idea, but that's not it - cabarc finished in about a minute. Maybe the difference comes from dictionary size.

    I made just a simple test - some results are here and the 7zipped file is here
    saves.tar is original file, saves.rep is saves.tar processed with rep

  26. #26
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    well, another idea: lzt uses hc or ht with a very deep search (say, 1000 elements) what makes it very slow when there are lot of 4-byte repetitions

  27. #27
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    Quote Originally Posted by Bulat Ziganshin
    hc or ht with a very deep search (say, 1000 elements)
    1000 is not so deep. All versions of BALZ uses HC with 4096 "elements"... as Deflate does - with highest compression it restricts the chain length to 4096.

Similar Threads

  1. Replies: 23
    Last Post: 24th March 2018, 17:57
  2. lrzip a file compressor for linux
    By joerg in forum Data Compression
    Replies: 2
    Last Post: 9th December 2009, 15:20
  3. LZTURBO 0.9 parallel compressor
    By donotdisturb in forum Forum Archive
    Replies: 18
    Last Post: 6th March 2008, 01:23
  4. I cannot find jpegoptim win32 build
    By SvenBent in forum Forum Archive
    Replies: 2
    Last Post: 30th November 2007, 23:41
  5. LZTURBO 0.1 parallel compressor
    By donotdisturb in forum Forum Archive
    Replies: 5
    Last Post: 7th October 2007, 22:44

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •