Results 1 to 26 of 26

Thread: BCM v0.03 is here! [!]

  1. #1
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts

    Exclamation BCM v0.03 is here! [!]

    OK, I have something special for you this time! Please welcome the brand new BCM v0.03! This is a complete rewrite of a program featuring an extremely new and unique Fast CM that designed specially for BWT-output. No more words. Check it out and enjoy!
    Attached Files Attached Files

  2. #2
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    calgary.tar (3,152,896 bytes)
    BCM -> 787,078 bytes
    BLIZ -> 790,491 bytes
    BBB -> 798,705 bytes
    MCOMP -> 800,402 bytes

    book1 (768,771 bytes)
    BCM -> 211,554 bytes
    BLIZ -> 212,130 bytes
    BBB -> 213,162 bytes
    MCOMP -> 217,403 bytes


  3. #3
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    I'm amazed by your progress.
    Code:
    BOOKSTAR:
    BCM 0.03		9262010	~28.5s.
    BCM 0.02		9547951	64.610
    BBB f 160m		9226430	~128.5
    Blizzard c		9590717	21.203
    Blizzard c 140000000	9107041	25.641
    Blizzard f		9753676	17.719
    Blizzard f 140000000	9257193	22.204
    YBS  2m			9998527	~21.5
    YBS 16m			9444092	~27

  4. #4
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Ilia!

  5. #5
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts

    Hi Ilia!

    Please Ilia help me , MOC Test failed for BCM in file :
    http://veimages.gsfc.nasa.gov/2433/l..._topo_8192.tif conveted to BMP 24B

  6. #6
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    1.0337 in generic test
    http://cs.fit.edu/~mmahoney/compression/uiq/

    #37 on LTCB
    http://cs.fit.edu/~mmahoney/compression/uiq/

    LTCB would probably benefit from a larger block size. Compression and speed is pretty good considering the low memory usage.

  7. #7
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    Quote Originally Posted by Nania Francesco View Post
    Please Ilia help me , MOC Test failed for BCM in file :
    http://veimages.gsfc.nasa.gov/2433/l..._topo_8192.tif conveted to BMP 24B
    I tried the original TIF (not converted to BMP) and verified decompression was OK.
    27,797,968 -> 25,723,903.

  8. #8
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts

    Hi Matt!

    The problem is in the BMP converted file!

  9. #9
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    Quote Originally Posted by Nania Francesco View Post
    The problem is in the BMP converted file!
    I'll fix the bug. Probably it's the fake alert in the decoder...

  10. #10
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    i make a first test on a new machine with Core 2 Duo / 2 GB RAM

    The Testfile db.dmp (Oracle-dump-file) has 648331264 bytes.

    bcm003 ratio: 0.392 bpb compress to 31783496 bytes 260 s

    RINGS v.1.3 (FCM Fast Context Mixing) file compressor. Only for testing
    Copyright 2007 by Nania Francesco Antonio (Italy). All rights reserved.

    rings13 c compress to 26356464 bytes time= 20.77 s.

    RINGS v.1.5c (FCM Fast Context Mixing) file compressor. Only for testing
    Copyright 2007-2008 by Nania Francesco Antonio (Italy). All rights reserved.

    rings15c c 1 compress to 32986358 bytes time= 23.89 s.
    rings15c c 2 compress to 30505178 bytes time= 24.09 s.
    rings15c c 3 compress to 28717515 bytes time= 24.28 s.
    rings15c c 4 compress to 27461544 bytes time= 24.11 s.
    rings15c c 5 compress to 26580793 bytes time= 24.08 s.
    rings15c c 6 compress to 25955867 bytes time= 24.03 s.
    rings15c c 7 compress to 25649655 bytes time= 24.33 s.
    rings15c c 8 compress to 25385543 bytes time= 24.44 s.
    rings15c c 9 compress to 25222852 bytes time= 23.83 s.

    my result:

    bcm performs significantly faster on better hardware
    but for now
    rings 1.3 and rings 1.5c wins clearly on textfile or oracle-dmp-file
    (size and time is lower)

    bcm compress better then
    rings15c c 1
    in the one case - if rings use only low memory

    i do not know - may be
    bcm would compress better if it use more memory ?

    best regards

  11. #11
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    Quote Originally Posted by joerg View Post
    i make a first test on a new machine with Core 2 Duo / 2 GB RAM

    The Testfile db.dmp (Oracle-dump-file) has 648331264 bytes.

    bcm003 ratio: 0.392 bpb compress to 31783496 bytes 260 s

    RINGS v.1.3 (FCM Fast Context Mixing) file compressor. Only for testing
    Copyright 2007 by Nania Francesco Antonio (Italy). All rights reserved.

    rings13 c compress to 26356464 bytes time= 20.77 s.

    RINGS v.1.5c (FCM Fast Context Mixing) file compressor. Only for testing
    Copyright 2007-2008 by Nania Francesco Antonio (Italy). All rights reserved.

    rings15c c 1 compress to 32986358 bytes time= 23.89 s.
    rings15c c 2 compress to 30505178 bytes time= 24.09 s.
    rings15c c 3 compress to 28717515 bytes time= 24.28 s.
    rings15c c 4 compress to 27461544 bytes time= 24.11 s.
    rings15c c 5 compress to 26580793 bytes time= 24.08 s.
    rings15c c 6 compress to 25955867 bytes time= 24.03 s.
    rings15c c 7 compress to 25649655 bytes time= 24.33 s.
    rings15c c 8 compress to 25385543 bytes time= 24.44 s.
    rings15c c 9 compress to 25222852 bytes time= 23.83 s.

    my result:

    bcm performs significantly faster on better hardware
    but for now
    rings 1.3 and rings 1.5c wins clearly on textfile or oracle-dmp-file
    (size and time is lower)

    bcm compress better then
    rings15c c 1
    in the one case - if rings use only low memory

    i do not know - may be
    bcm would compress better if it use more memory ?

    best regards
    The catch in BWT's block size. Of course on large text or similar files BWT benefits from a larger block size. BCM has a fixed 32 MB block. Try to compare Rings and BCM on files <=32 MB in size. At the same time a larger block may hurt compression with binary data.

  12. #12
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    ---
    The catch in BWT's block size.
    Of course on large text files BWT benefits from a larger block size.
    BCM has a fixed 32 MB block.
    Try to compare Rings and BCM on files <=32 MB in size.
    At the same time a larger block may hurt compression with binary data
    ---
    thank you very much for your quickly answer

    i will test this

    Would it be possible to make a version with a 64 MB block or 128 MB block ?

    thank you for your work!

    best regards

  13. #13
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    Quote Originally Posted by joerg View Post
    ---
    The catch in BWT's block size.
    Of course on large text files BWT benefits from a larger block size.
    BCM has a fixed 32 MB block.
    Try to compare Rings and BCM on files <=32 MB in size.
    At the same time a larger block may hurt compression with binary data
    ---
    thank you very much for your quickly answer

    i will test this

    Would it be possible to make a version with a 64 MB block or 128 MB block ?

    thank you for your work!

    best regards
    BCM v0.04 that will be released ASAP (due to a bug fix), will have 64 MB block (~324 MB mem use - still OK). I'm talking about block size importance because you may not beleive how large block may improve compression on files like your one. I'll post some results with various block sizes on ENWIK9 to see the difference a little bit later...

  14. #14
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    thank you again for your quickly answer

    bcm 0.04 sounds very interesting

    best regards

  15. #15
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    The bug was fixed. It's indeed a fake alert in the decoder. Since haste makes a waste I'll release a new version tomorrow...

    Some testing results on ENWIK9 with different block sizes:

    32m -> 192,194,478 bytes
    64m -> 185,846,162 bytes
    128m -> 180,548,624 bytes
    256m -> 175,167,185 bytes

    64m block is quite reasonable.

    Just tested RINGS v.1.5c on standard files:
    book1 -> 226,672 bytes
    calgary.tar -> 831,821 bytes
    It's no so good...

  16. #16
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Why won't you make it configurable?
    It will be beneficial for users and for your scores too.
    I suggest independent settings for BWT and CM memory.

  17. #17
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    ---
    results on ENWIK9 with different block sizes:

    32m -> 192,194,478 bytes
    64m -> 185,846,162 bytes
    128m -> 180,548,624 bytes
    256m -> 175,167,185 bytes
    ---

    i think too it would be nice to experiment with several amonts of blocksize

    at least it would be wonderful to have a 64mb-
    and a (as you has shown better compressing) 256mb- version

    if i understand well, then this is the blocksize for bwt - is this right ?
    i think - in several cases cm can profit from a bigger blocksize too
    but a "step by step" - optimizing is better

    best regards

  18. #18
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    Well, for me it's not really interesting to gain compression on weird files like huge pictures and enormous text files like ENWIK using a huge BWT block. As I said, in practice a larger block doesn't mean a better compression. But the main reason is that I do plan to add an auto-block size - i.e. kind of segmentation for higher binary compression...

  19. #19
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by encode View Post
    Well, for me it's not really interesting to gain compression on weird files like huge pictures and enormous text files like ENWIK using a huge BWT block. As I said, in practice a larger block doesn't mean a better compression. But the main reason is that I do plan to add an auto-block size - i.e. kind of segmentation for higher binary compression...
    How about solid blocks? They are common. With a lot of text files setting large block size should definitely help. And it's not that artificial example.

    Really, nobody's gonna use it on binary files unless by mistake.

  20. #20
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    BCM is a general purpose file compressor. Although it is the best for texts and pictures.

    BWT, I even further improved CM compression for next version...

  21. #21
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    thanks!

    --- improved CM compression for next version ---

    sounds wonderful !

    --- I do plan to add an auto-block size ---

    "step by step" ?
    i think for now it would be enough to have two versions
    one with 64 MB and one with 256 MB blocksize
    then we can see the 256 MB-Version is significantly better or not

    best regards

  22. #22
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    Quote Originally Posted by joerg View Post
    ---
    results on ENWIK9 with different block sizes:

    32m -> 192,194,478 bytes
    64m -> 185,846,162 bytes
    128m -> 180,548,624 bytes
    256m -> 175,167,185 bytes
    best regards
    There is a similar table for BBB. For text, there doesn't seem to be any point where using a bigger block size doesn't help.
    http://cs.fit.edu/~mmahoney/compression/text.html#1640

    The only reason BBB does well on enwik9 is that it uses a single 1 GB block size with some tricks so it runs with 1.25 GB memory.

  23. #23
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Here's a little test on my Athlon 64 3000+ (single core) machine with BCM.

    bcm003 enwik8 - 22,007,655 bytes ( Compress: 69sec, Decompress 61sec)
    bcm002a enwik8 - 23,761,415 ( Compress: 138sec, Decompress: 56 sec )

    Great progress! Can't wait for the next version.

  24. #24
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    To next version I already added an optimized SSE with linear interpolation at last.
    BCM have interesting CM consisting of two parts:
    1. Set of models with static mixing with not just order-0, but with special order-1 and order-2
    2. SSE model (which may be considered as an additional order-0 model)

    All these stuff provides one of the best results of BWT-output coding so far! Especially, new BCM v0.04. It closely beats PPMd on book1!

  25. #25
    Member
    Join Date
    Feb 2009
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts

    BCM packer Rocks !

    наша фирма заинтересована в использование этого компрессора в программирование видео игр для приставок Nintendo DS, iPhone, Wii

    1) заинтересованы ли вы в том, чтобы мы использовали ваш комппессор?
    2) возможно ли декомпрессировать етим компрессором формат BCM с процессором 66Mhz и небольшой RAM?

    easy

  26. #26
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    Quote Originally Posted by easy View Post
    наша фирма заинтересована в использование этого компрессора в программирование видео игр для приставок Nintendo DS, iPhone, Wii

    1) заинтересованы ли вы в том, чтобы мы использовали ваш комппессор?
    2) возможно ли декомпрессировать етим компрессором формат BCM с процессором 66Mhz и небольшой RAM?

    easy
    Компрессор BCM имеет опции которые можно настроить для нужной производительности и с учетом конкретного оборудования. Например при малом оъеме доступной оперативной памяти, возможно использование более малого блока для сжатия. Для более слабого процессора можно "облегчить" контекстную модель - например убрав некоторые составляющие части этой модели как то SSE/APM, интерполяцию, отдельные счетчики. В этом плане BCM очень гибок.
    Я заинтересован и готов к сотрудничеству!
    Также со мной можно связаться через Private Messages на этом форуме или через ICQ - номер моей аськи указан у меня в профиле!

Similar Threads

  1. BCM v0.10 is here!
    By encode in forum Data Compression
    Replies: 45
    Last Post: 20th June 2010, 21:39
  2. BCM's future
    By encode in forum Data Compression
    Replies: 17
    Last Post: 9th August 2009, 01:00
  3. BCM v0.06,0.07 is here! [!]
    By encode in forum Data Compression
    Replies: 34
    Last Post: 31st May 2009, 16:39
  4. BCM v0.05 is here! [!]
    By encode in forum Data Compression
    Replies: 19
    Last Post: 8th March 2009, 21:12
  5. BCM v0.04 is here! [!]
    By encode in forum Data Compression
    Replies: 64
    Last Post: 5th March 2009, 17:07

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •