Results 1 to 22 of 22

Thread: comprox/comprolz updates

  1. #1
    Member RichSelian's Avatar
    Join Date
    Aug 2011
    Location
    Shenzhen, China
    Posts
    156
    Thanks
    18
    Thanked 50 Times in 26 Posts

    comprox/comprolz updates

    comprox verion 0.9.0 is here: http://comprox.googlecode.com/files/...x-0.9.0.tar.gz
    comprolz version 0.2.0 is here: http://comprox.googlecode.com/files/...z-0.2.0.tar.gz

    a simple test for enwik8 (with option "-b100 -f"):
    Code:
    comprox  100000000 => 24281529
    comprolz 100000000 => 24280609
    Last edited by RichSelian; 18th October 2012 at 13:25.

  2. #2
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts

  3. #3
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    I added comprox 0.9.0 and comprolz 0.2.0 to the Silesia corpus.
    Comprox crashes when compressing samba with -f.
    http://mattmahoney.net/dc/silesia.html

  4. #4
    Member RichSelian's Avatar
    Join Date
    Aug 2011
    Location
    Shenzhen, China
    Posts
    156
    Thanks
    18
    Thanked 50 Times in 26 Posts
    Quote Originally Posted by Matt Mahoney View Post
    I added comprox 0.9.0 and comprolz 0.2.0 to the Silesia corpus.
    Comprox crashes when compressing samba with -f.
    http://mattmahoney.net/dc/silesia.html
    Thanks very much for benchmark and bug report. a bugfix version is here: http://comprox.googlecode.com/files/...bugfix1.tar.gz

  5. #5
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    Thanks. I updated the Silesia benchmark.

  6. #6
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    872
    Thanks
    457
    Thanked 175 Times in 85 Posts
    Comprox 0.90 bugfix still crashes on some of my .tar files.
    The commandline used were 'comprox -b160 -f e app.tar a.comprox' and 'comprox -m100 -b160 -f e app.tar a.comprox'
    Using a smaller blocksize didn't help.

  7. #7
    Member RichSelian's Avatar
    Join Date
    Aug 2011
    Location
    Shenzhen, China
    Posts
    156
    Thanks
    18
    Thanked 50 Times in 26 Posts
    Quote Originally Posted by Stephan Busch View Post
    Comprox 0.90 bugfix still crashes on some of my .tar files.
    The commandline used were 'comprox -b160 -f e app.tar a.comprox' and 'comprox -m100 -b160 -f e app.tar a.comprox'
    Using a smaller blocksize didn't help.
    Thanks for reporting. Could you upload app.tar or paste a download link for me? It will help a lot with the problem.

  8. #8
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    872
    Thanks
    457
    Thanked 175 Times in 85 Posts

  9. #9
    Member RichSelian's Avatar
    Join Date
    Aug 2011
    Location
    Shenzhen, China
    Posts
    156
    Thanks
    18
    Thanked 50 Times in 26 Posts
    Quote Originally Posted by Stephan Busch View Post
    Thank you very much, the bug is fixed.
    New version is here:
    http://comprox.googlecode.com/files/...x-0.9.1.tar.gz
    http://comprox.googlecode.com/files/...z-0.2.1.tar.gz

  10. #10
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    872
    Thanks
    457
    Thanked 175 Times in 85 Posts
    This version of COMPROX still can't compress my app.tar.
    The source was compiled using MinGW 4.6.2 and this commandline: gcc -O3 -march=native -s *.c

    The executable always crashes as soon as the LZ77 encoding wants to start.

  11. #11
    Member RichSelian's Avatar
    Join Date
    Aug 2011
    Location
    Shenzhen, China
    Posts
    156
    Thanks
    18
    Thanked 50 Times in 26 Posts
    Quote Originally Posted by Stephan Busch View Post
    This version of COMPROX still can't compress my app.tar.
    The source was compiled using MinGW 4.6.2 and this commandline: gcc -O3 -march=native -s *.c

    The executable always crashes as soon as the LZ77 encoding wants to start.
    That's very strange, I tested this version with valgrind under linux, compressing app.tar for hours but without an error. Maybe it's a problem of windows threading? (I defined pthread_create() to CreateThread() on windows.)
    I also tested with mingw32-4.7.0 and WINE on linux. it runs as good as native linux executables. But I don't have a windows computer, so I'm not able to locate the bug and fix it.

  12. #12
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    Same here. comprox -b80 e app.tar app.tar.cpx crashes. Also -b96, -b128, -b160, but -b64 is OK. Compiled with gcc 4.7.0 -O3 -s -march=native (also just -O3) in 32 bit Windows Vista.

  13. #13
    Member RichSelian's Avatar
    Join Date
    Aug 2011
    Location
    Shenzhen, China
    Posts
    156
    Thanks
    18
    Thanked 50 Times in 26 Posts
    Quote Originally Posted by Matt Mahoney View Post
    Same here. comprox -b80 e app.tar app.tar.cpx crashes. Also -b96, -b128, -b160, but -b64 is OK. Compiled with gcc 4.7.0 -O3 -s -march=native (also just -O3) in 32 bit Windows Vista.
    I fix a bug that may cause memory access violation, but I'm not sure that's the reason causing windows version crash. give it a try, please.
    Attached Files Attached Files

  14. #14
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    872
    Thanks
    457
    Thanked 175 Times in 85 Posts
    This bugfix version seems to work. I compiled using 'gcc -O3 -march=native -s *.c'; the executable is 91.648 Bytes in size.
    Thank you Rich.

  15. #15
    Member RichSelian's Avatar
    Join Date
    Aug 2011
    Location
    Shenzhen, China
    Posts
    156
    Thanks
    18
    Thanked 50 Times in 26 Posts
    0.10.0 is here: http://comprox.googlecode.com/files/...-0.10.0.tar.gz

    I made comprox and comprolz together in a package since they share many common codes. This version should have chance compressing enwik9 to less than 200MB!

  16. #16
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Comprox v. 0.10.0 tested in WCC!
    Usually I always congratulate them all when they start and this time I will respond differently because I think you can do much better seen achievements in bmp, text,dat,xml and exe.

    Great work but you have to improve speed of decompression and maybe try and better than with a traditional arithmetic coding with the contexts for the bits and bytes to not!

    Overall great improvement!

  17. #17
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    ComproLZ v. 0.10.0 tested in WCC. It is slower than comprox!

  18. #18
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    Updated LTCB. Compression is much improved. http://mattmahoney.net/dc/text.html#1987

  19. #19
    Member RichSelian's Avatar
    Join Date
    Aug 2011
    Location
    Shenzhen, China
    Posts
    156
    Thanks
    18
    Thanked 50 Times in 26 Posts
    version 0.11.0 is here: http://comprox.googlecode.com/files/...-0.11.0.tar.gz

    Now the dictionary is generated from input file and no more static dictionary file is needed.

  20. #20
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    Compression is worse on LTCB but better on Silesia corpus.
    http://mattmahoney.net/dc/text.html#1987
    http://mattmahoney.net/dc/silesia.html (but 0.7.0 is still tops).

    I guess that is because previous static dictionary was derived from LTCB. Also comprox -m100 it took 30 times longer to compress enwik9 than enwik8. I guess that is due to growing hash chains. With default -m40, it is 12 times longer.

    Also comprox -b250 enwik9 crashes (but enwik8 is OK). I had to use -b200. comprolz -b250 was OK.

    Nice to have .exe included so I don't have to compile.

  21. #21
    Member RichSelian's Avatar
    Join Date
    Aug 2011
    Location
    Shenzhen, China
    Posts
    156
    Thanks
    18
    Thanked 50 Times in 26 Posts
    Quote Originally Posted by Matt Mahoney View Post
    Compression is worse on LTCB but better on Silesia corpus.
    http://mattmahoney.net/dc/text.html#1987
    http://mattmahoney.net/dc/silesia.html (but 0.7.0 is still tops).

    I guess that is because previous static dictionary was derived from LTCB. Also comprox -m100 it took 30 times longer to compress enwik9 than enwik8. I guess that is due to growing hash chains. With default -m40, it is 12 times longer.

    Also comprox -b250 enwik9 crashes (but enwik8 is OK). I had to use -b200. comprolz -b250 was OK.

    Nice to have .exe included so I don't have to compile.
    a bugfix version: http://comprox.googlecode.com/files/...bugfix1.tar.gz
    v0.11.0 should compress enwik8 to about 23070KB with default options. but due to an "undefined behavior" in the code, the win32 version gave a worse result. now the bug is fixed

  22. #22
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    The bug fix gives a big improvement on LTCB (moves up in rankings) and a small improvement on the Silesia corpus.

Similar Threads

  1. zpaq updates
    By Matt Mahoney in forum Data Compression
    Replies: 2527
    Last Post: 4th May 2019, 12:33
  2. comprox-0.1
    By RichSelian in forum Data Compression
    Replies: 40
    Last Post: 29th September 2012, 10:52
  3. comprox updates
    By RichSelian in forum Data Compression
    Replies: 1
    Last Post: 5th November 2011, 23:19
  4. recommended formats for game data and partial updates
    By willvarfar in forum Data Compression
    Replies: 14
    Last Post: 23rd November 2010, 19:26
  5. Metacompressor.com benchmark updates
    By Sportman in forum Data Compression
    Replies: 79
    Last Post: 22nd April 2009, 03:24

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •