Page 1 of 2 12 LastLast
Results 1 to 30 of 46

Thread: LZP2 - compression program by a newbye

  1. #1
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    856
    Thanks
    447
    Thanked 254 Times in 103 Posts

    LZP2 - compression program by a newbye

    Hello

    I'm hesitatingly presenting here a first compressor for PC/Win32.
    This was the first time i programmed again in C since more than 15 years, so obviously, i needed something very simple to get back in the game.

    I selected an LZP algorithm, for its very trivial implementation. The end result is a very fast compressor, much faster than any HDD, USB flash, Ethernet link, well whatever fast peripheral a computer may communicate with. And frankly, this is the only reason for which this algorithm could be usefull for, because the downside, of course, is poor compression rating, thus poor archiver performance.

    Anyway, here it is :
    http://LZP2.Webhop.org

    The program provides an internal in-memory benchmark, using the -bench argument, for quick evaluation.

    Regards
    Last edited by Cyan; 18th April 2009 at 02:19.

  2. #2
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts

  3. #3
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Cyan!

  4. #4
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    215
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hey, it is seriously, I mean really really fast. Maybe with some optimizations to improve the compression ratio it could be a nice choice for something like a file system

  5. #5
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    139
    Thanked 198 Times in 108 Posts

    Post

    Paq8 was running at same time.
    Code:
    lzp2 -bench  enwik8
    Reading source file....
    Setting HIGH_PRIORITY_CLASS...
    Benchmarking LZP2 , please wait...
    
    Compressed 100000000 bytes into 74323999 bytes (74.3%) at 157.0 Mbyte/s.
    
    Decompressed at 238.9 Mbyte/s.
    
    (1 MB = 1000000 byte)
    No other cpu intensive apps running.
    Code:
    lzp2 -bench  enwik8
    Reading source file....
    Setting HIGH_PRIORITY_CLASS...
    Benchmarking LZP2 , please wait...
    
    Compressed 100000000 bytes into 74323999 bytes (74.3%) at 167.3 Mbyte/s.
    
    Decompressed at 246.2 Mbyte/s.

    Made 2 runs. Identical.
    KZo


  6. #6
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    tested on enwik8:
    Compressed 100000000 bytes into 74323999 bytes (74.3%) at 234.0 Mbyte/s.

    Decompressed at 348.9 Mbyte/s.
    q6600@3.25GHz

  7. #7
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    856
    Thanks
    447
    Thanked 254 Times in 103 Posts
    Thanks very much for feedback. They are very helpful.
    I've stored your results on lzp2 homepage, for reference.

    Do you feel compression speed is an interesting feature then ? This would help to select an objective, between speed and compression ratio, for the next software release.

    Maybe with some optimizations to improve the compression ratio it could be a nice choice for something like a file system
    Do you mean it would sound sensible to provide a kind of DLL or library for inclusion into a 3rd party software ?

    Regards

  8. #8
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Pentium D 2.66
    Bookstar:
    Code:
    Compressed 35594240 bytes into 30438536 bytes (85.5%) at 98.3 Mbyte/s.
    
    Decompressed at 141.2 Mbyte/s.
    Some bitmap:
    Code:
    Compressed 100663350 bytes into 35938139 bytes (35.7%) at 171.9 Mbyte/s.
    
    Decompressed at 324.4 Mbyte/s.

  9. #9
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    yes lzp2 is very fast

    (has someone it compared with fastlz or quicklz ?),

    but can it compress files > 4 GBytes ?

    best regards

  10. #10
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by joerg View Post
    yes lzp2 is very fast

    (has someone it compared with fastlz or quicklz ?),

    but can it compress files > 4 GBytes ?

    best regards
    It seems about FastLZ, QuickLZ is faster.

    I just made 1 direct comparison, decompression of ~100 MB executable:
    Code:
    Program                        size      CPU time
    LZOP -9 -F                     46023879  0.656
    Quick -3                       49342629  0.687
    Quick -2                       52595713  0.718
    Quick -1                       56463297  0.734
    LZSS ex                        47882127  0.828
    FastLZ -2                      57871075  0.875
    FreeArc -mrep                  89407238  0.875
    LZP2                           61953767  0.890
    FastLZ -1                      59193978  0.921
    FastLZ opt -1                  59383836  0.921
    FastLZ opt -2                  58255894  0.953
    UPX NRV2B                      37060096  0.968
    UPX NRV2D                      36717568  0.984
    UPX NRV2E                      36585472  1.000

  11. #11
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    rep is preprocessor, and fa involves crc calculation overhead, i rather suggest to check tor -1 (or -11 -c1 )

  12. #12
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    m^2 - thank you very much !

    the results are near the results from fastlz ..

    have you seen the filesize - limit too?

    best regards

  13. #13
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Is it wrong to test a preprocessor?
    CRC - I don't remember if I used standalone rep, but probably not. My bad.

    The results posted above (except for LZP2) are a part of this.
    Tor is there too, but only 0.4.


    joerg,
    I didn't test large file support.
    Last edited by m^2; 18th April 2009 at 19:57.

  14. #14
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by m^2 View Post
    Is it wrong to test a preprocessor?
    just not very useful

  15. #15
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    just not very useful
    I don't agree with you here. It's good to know how much time does each (de)compression step takes to optimize it well. Sure modifications here affect later stages, but that's often negligible.

    And besides, I'm coming here to learn. Knowing that (in FA) REP decompresses much faster than LZP is nice and can be useful.

  16. #16
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    856
    Thanks
    447
    Thanked 254 Times in 103 Posts
    LZP strength, which is its context table, becomes a burden when decompression speed comes into mind.
    You have to maintain the table, and emulate its evolution as would do the encoder. In fact, decoder complexity is almost the same as the encoder, you merely avoid testings to find match length.

    LZSS has the edge at this game. LZSS requires no table, pure relative index, so it's very fast.

  17. #17
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    It doesn't work on my Pentium III 750Mhz, 512MB RAM, Windows 2000 Pro machine.

    Code:
    C:\>lzp2 -bench enwik8
    LZP2 (Apr 18 2009) by Yann Collet, with kind support from Bruno Wyttenbach
    Reading source file....
    Setting HIGH_PRIORITY_CLASS...
    Benchmarking LZP2 , please wait...
    
    C:\>
    I have tried it with several different files. Same result.

  18. #18
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    probably he compiled with sse2, sse3, sse8.5

  19. #19
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    856
    Thanks
    447
    Thanked 254 Times in 103 Posts
    Thanks for information, LovePimple.

    I actually had another similar report this morning from an Athlon XP system.
    As a consequence, i've just compiled a new LZP2 version specifically disabling SIMD instructions. Speed doesn't change anyway.

    Could you please have a look at it ?
    It can be downloaded at :
    http://img40.xooimage.com/files/2/d/6/lzp2-da70a7.zip

  20. #20
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Same problem with this version. It just drops out without warning, leaving behind a zero byte archive file.

  21. #21
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    856
    Thanks
    447
    Thanked 254 Times in 103 Posts
    Hummm...

    Then this is a bit more complex...
    I've tried to disable a few more optimisation options in the compilator parameters.
    This gives this version : http://img26.xooimage.com/files/3/f/9/lzp2-da821f.zip

    There are still a few parameters left. So a bit of tuning is still possible.

    I wish i could find an emulator that could reproduce your problem.
    Unfortunately, even VMWare & consorts does not allow to mimic another processor... Well, if anyone has an idea...

  22. #22
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Still the same. I'm ready to test the next version.

  23. #23
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Cyan View Post
    Hummm...

    Then this is a bit more complex...
    I've tried to disable a few more optimisation options in the compilator parameters.
    This gives this version : http://img26.xooimage.com/files/3/f/9/lzp2-da821f.zip

    There are still a few parameters left. So a bit of tuning is still possible.

    I wish i could find an emulator that could reproduce your problem.
    Unfortunately, even VMWare & consorts does not allow to mimic another processor... Well, if anyone has an idea...
    VMWare does not use emulation but virtualization.
    Try QEMU or BOCHS.

  24. #24
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    856
    Thanks
    447
    Thanked 254 Times in 103 Posts
    OK

    This version has nearly every optimisation removed,
    and as a consequence, is seriously slower. about 3 times slower.

    http://img22.xooimage.com/files/5/8/c/lzp2-da89d0.zip

    It's only purpose is to check if there is a link between your problem and compiler optimisations. If it does, i will carefully re-enabled compiler switches to find the optimal working set.

  25. #25
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Its working perfectly. Now, if would like to start to add optimisation, I am happy to keep trying different versions until you find out exactly what was causing the problem.

  26. #26
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    856
    Thanks
    447
    Thanked 254 Times in 103 Posts
    ok, good news
    we have something to look for.

    You'll find at following address multiple version of LZP2 with various switches combinations :
    http://img42.xooimage.com/files/4/a/...lp2-da9268.zip

    I recommend following the numbers.
    Hopefully, the first version with basic optimisation should work. The,.. well, we'll see where it fails.
    I've intentionnally not provided SIMD versions. They don't provide any benefit anyway...

    Regards

  27. #27
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    may be "fast fp" optimization uses sse2?

  28. #28
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    856
    Thanks
    447
    Thanked 254 Times in 103 Posts
    may be "fast fp" optimization uses sse2?
    Humm, maybe,
    but that would be unwise from the compiler.
    There is a separate option which specifically enable/disable SSE2.
    So having a "hidden enabler" on an intentionnally disabled option wouldn't help.
    But well, sometimes, strange things happen...

    note : there is a very long article on MSDN related to this optimisation :
    http://msdn.microsoft.com/en-us/library/aa289157.aspx

  29. #29
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by Cyan View Post
    ok, good news
    we have something to look for.

    You'll find at following address multiple version of LZP2 with various switches combinations :
    http://img42.xooimage.com/files/4/a/...lp2-da9268.zip

    I recommend following the numbers.
    Hopefully, the first version with basic optimisation should work. The,.. well, we'll see where it fails.
    I've intentionnally not provided SIMD versions. They don't provide any benefit anyway...

    Regards
    OK, testing now...

  30. #30
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    1_LZP2-basic OK

    2_LZP2-inline OK

    3_LZP2-intrinsek OK

    4_LZP2-frameless OK

    5_LZP2-fastfp OK

    All of these versions work perfectly. I'm ready to test the next version.

Page 1 of 2 12 LastLast

Similar Threads

  1. LZBW1 - compression tool by another newbye :)
    By stfox in forum Data Compression
    Replies: 4
    Last Post: 28th April 2009, 16:33
  2. RTIME - New freeware program
    By LovePimple in forum The Off-Topic Lounge
    Replies: 3
    Last Post: 4th July 2008, 10:47
  3. XTIME - New freeware program
    By LovePimple in forum The Off-Topic Lounge
    Replies: 4
    Last Post: 4th July 2008, 03:40

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •