Page 1 of 3 123 LastLast
Results 1 to 30 of 85

Thread: CCM 1.25 is here!

  1. #1
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 150 Times in 18 Posts
    Hi everyone!

    The changelog is very short - as always:
    1) in this release CCMx got a ~5-10% speed boost over version 1.23
    2) that's it

    I'm on an X2 here - I hope that the additional speed is perceivable on other platforms, too.

    http://www.mytempdir.com/2034717

    I have not tested this version much. If there are any problems please let me know.


    Happy compressing,

    Christian

  2. #2
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Chris!

    Mirror: Download

  3. #3
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Intel Core duo 2 6600 2GB RAM
    Tested CCMX :
    Enwik8 c 7 -> 20.819.733 speed: 1203.81 KiB/s
    SFC Test Maximum compression c 7->10.734.204
    UCLC Wave test c 7 ->33.351.652
    UCLC PGM image test->5.531.088

  4. #4
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Thanks Chris, it's over 10% improvement for CCMx in my test

  5. #5
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    is it compatible with odler versions ?
    Can i decompress my old file with this new release?

    its seem athat CCM and CCMx are not compatible with each other.

    and now back to the testing

  6. #6
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    Quick feedbakc

    WinXP Pro SP2 Dan OEM.iso
    606.128.128 bytes


    CCMx v1.23 (6)
    549.835.520 bytes
    392.766 seconds

    CCMx v1.25 (6)
    549.813.493 bytes
    383.070 seconds

    2.5% faster
    and just slightly smaller.
    buts is progress and I'm loving it

  7. #7
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Some more numbers
    __compressor___output size_____in / out speed (kB/s)
    CCMx 1.25 (-6)__11 947 674___ 676 / 679
    CCMx 1.23 (-6)__11 947 923___ 592 / 607
    CCM 1.25 (-6)___11 974 703___757 / 738
    CCM 1.23 (-6)___11 974 852___712 / 731

  8. #8
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    just to answer my own question

    NO you cannot use v1.25 to decompress data compressed with v1.23

  9. #9
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 150 Times in 18 Posts
    Thanks for the numbers BlackFox, Nania and Sven!

    @Sven:
    Sorry, I'm a little late with my answer. As you've already found out - CCM and CCMx are incompatible as well as 1.23 and 1.25.

    Only 2,5% is a little bit disappointing, but well, it's better than a bug report. By the way, which platform do you use?

  10. #10
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    WinxP pro 64 SP2 English
    C23Q Q 6600 OC 3.2Ghz (9x356Mhz)
    965 cchipset
    2GB of ram runnig 712mhz (1:! with fsb, Cas 4)

  11. #11
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Christian, great to have you back. Was this just a quick release, or are you back to developing ccm? These "internal changes" that you made are great.

    From what I can tell you're using the GCC compiler. Any chance on compiling for linux i386, and x64? I know 64 bit versions of paq8 show great performance improvements versus their 32bit equivalents. I'd love to test a 64bit linux version to see if it performs better.

    Out of curiosity, were some of these "internal changes" having to do with caching from the hard disk? I seem to get less of a performance improvement when working off a RAM disk.

    Also, are you still considering making a 3rd mode with higher ratio like we had talked about a few months ago?

    Well, here are some tests....

    Test System: AMD Athlon 64 3000+ (single core) with 2GIG DDR (dual channel) Corsair CL2 RAM

    ccmx123.exe c 7 enwik8 enwik8.ccmx123 (20,819,656 Bytes)- 2m 27sec
    ccmx123.exe d enwik8.ccmx123 enwik8 - 2m 15sec
    ccmx125.exe c 7 enwik8 enwik8.ccmx125 (20,819,733 Bytes)- 2m 8 sec ( 12.9% faster! )
    ccmx125.exe d enwik8.ccmx125 enwik8 - 2m 10sec (3.8% faster)

    Other Tests with input and output file located in ramdisk (formated as ext3) to eliminate disk effects:

    ccmx123.exe c 7 enwik8 enwik8.ccmx123 - 2m 16sec
    ccmx125.exe c 7 enwik8 enwik8.ccmx125 - 2m 7sec ( 7% faster )

    ccmx123.exe c enwik8 enwik8.ccmx123 ( 21,601,466 Bytes ) - 2m 12sec
    ccmx125.exe c enwik8 enwik8.ccmx125 ( 21,608,042 Bytes ) - 2m 3sec (6.8% faster)

    ccm123.exe c 7 enwik8 enwik8.ccm - ( 21,960,016 Bytes ) 1m 41 sec
    ccm125.exe c 7 enwik8 enwik8.ccm - ( 21,960,598 Bytes ) 1m 38 sec (3% faster)

  12. #12
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 150 Times in 18 Posts
    Hi hahobas!
    Thank you for testing!

    Quote Originally Posted by Hahobas
    Was this just a quick release, or are you back to developing ccm?
    This was just a quick release. I "stumbled" over the latest source and I just had to code a few lines.

    Quote Originally Posted by Hahobas
    Out of curiosity, were some of these "internal changes" having to do with caching from the hard disk?
    No, nothing disk relevant. I reordered the context mixing part and improved the hashing again.

    Quote Originally Posted by Hahobas
    I seem to get less of a performance improvement when working off a RAM disk.
    I have no idea what happend here. But CCM is extremly sensible to memory bandwidth. Even when I play a MP3 on my dual core system while benchmarking, CCM is actually ~3% percent slower!

    Quote Originally Posted by Hahobas
    Also, are you still considering making a 3rd mode with higher ratio like we had talked about a few months ago?
    Yep, Im still considering this. But dont expect this to be anytime soon. Measured by the time devoted this will take like a decade.

    Quote Originally Posted by Hahobas
    Any chance on compiling for linux i386, and x64? I know 64 bit versions of paq8 show great performance improvements versus their 32bit equivalents.
    Since Im running a 32bit version of Linux Ill have a look at this.

  13. #13
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Christian
    But CCM is extremly sensible to memory bandwidth.
    I think youre right. Ive been doing a lot of testing, and found something interesting. I have two hard disks on my system, both are SATA, but one is a 10,000 RPM WD raptor, and the other is a 7200 Seagate. If I do the following, I get these timings.

    ccmx.exe c enwik8_on_seagate enwik8.ccm_on_seagate - 2m 22sec
    ccmx.exe c enwik8_on_seagate enwik8.ccm_on_raptor - 2m 12sec
    ccmx.exe c enwik8_on_raptor enwik8.ccm_on_raptor - 2m 12 sec

    So it appears that my seagate bandwidth isnt fast enough to hold an upstream, and a downstream without impacting performance.

    I wonder if any performance could be gained by doing some type of cached write of the compressed data (since it flows a lot slower than the incoming data), so that people without 10,000 rpm hard drives can have faster performance.

    On a different note, I wrote a benchmarking bash script that compares any two versions with all their modes and switches of CCM so if youd like any exhaustive tests done, let me know. Testing a 32 and 64bit linux version would be especially nice

  14. #14
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    Hahobas
    i suspect that pefrormance loss is caused by disk fragmentation or simulteanous use by other programs. hdds have bandwidth of tens of megabytes per second while ccmx compress at no more than a few megabytes per second (on very fast systems).

  15. #15
    Member
    Join Date
    Jun 2008
    Location
    G
    Posts
    372
    Thanks
    26
    Thanked 22 Times in 15 Posts
    How about big file support? I mean about 4gb?

  16. #16
    Member
    Join Date
    May 2008
    Location
    CN
    Posts
    45
    Thanks
    0
    Thanked 0 Times in 0 Posts
    somebody say ccm and ppmd are both Symmetry
    while lzma or lzpm are not, can you explain it

  17. #17
    Member
    Join Date
    Jun 2008
    Location
    G
    Posts
    372
    Thanks
    26
    Thanked 22 Times in 15 Posts
    that mean:

    Symmetry: compressions/decompressions speed -> time are nearly equal
    Assymmetry: compressions/decompressions speed -> time are very different most compressions is X times slower as decompression

  18. #18
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    Some additional info:

    Symmetric (PPM, CM, etc.)
    The compression speed is about the same as the decompression, in some cases decompression speed can be even slightly slower. PPM-based compressors and similar, at decompression just build the same data model as with compression - this takes the same amount of time as with the compression.

    Asymmetric (BWT)
    The decompression is faster than the compression. This is sorting related thing - the reverse sorting is faster.

    Highly asymmetric (LZ)
    The decompression is much faster than the compression (10X times and more). The LZ coder most time spends on string searching and parsing optimizations. The decoder is simplest - it only decodes literals and copies strings (set of bytes) from previously seen data.

    Hope this helps.

  19. #19
    Member
    Join Date
    Jul 2008
    Posts
    54
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I tried this and it's really fast vs the last version, and now a question/request:
    Next version will be optimized for a SMP or dual/quad cores CPu's?

  20. #20
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    YES SMP and big filesupport would be nice,

  21. #21
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by thometal
    How about big file support? I mean about 4gb?
    I completely agree

    On another note in relation to optimizing for multiple cores, I know that it is not the easiest thing to do with context mixing. However if you download the latest gcc 4.2.2 you could experiment with OpenMP (auto-parallelization) directives which could offer some multi-core performance boosts with very little work on your part.

    Just an idea......

  22. #22
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    CCM vs FreeArc?
    Comment's!

  23. #23
    Member
    Join Date
    Oct 2007
    Posts
    51
    Thanks
    0
    Thanked 0 Times in 0 Posts
    A very basic question:

    Will CCM be released as Open Source (GPL) ?

    (I'm asking the same for FreeArc)

    Both CCM/FreeArc are good but if they want a chance to success, (and their creators too) I believe the only way is with Open Source.

  24. #24
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Nania Francesco Antonio
    CCM vs FreeArc?
    Comments!
    FreeArc is more flexible in terms of speed vs. ratio, CCM is more powerful

    Quote Originally Posted by Gish
    Will CCM be released as Open Source (GPL) ?

    (Im asking the same for FreeArc)
    Some of promising projects (these two or Precomp for instance) are not yet opensource, because their authors either think that the source code could use some more optimizing or that the source code is rather messy at the moment and needs cleaning up before releasing it.

  25. #25
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Overal, I think the person who invents a way to make the context mixing algorithm SMP friendly will be the person with the next "big thing" in compression especially since the world is going to multiple cores.

    I'm wondering if a pipe-line style approach could be used with CCM to get increased speed and/or ratio on multiple cores. It's pretty obvious that traditional context mixing is hard to split among the cores, but CCM is not context mixing alone. What if one core did the pre-processing (file filters, perhaps work like "precomp", then piped that into another core that did the context mixing etc. Is this type of thing worth it?

  26. #26
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 150 Times in 18 Posts
    Hello there!

    At all people requesting features - I'm quite busy with other things right now, thus ccm's development is more or less frozen.

    Just a few thoughts on multithreading. ccm's filters are very fast - there's no gain putting them in another thread. The same applies for the data detection which is only done during compression, not decompression (note, decompression is isn't any faster than compression). But as Hahobas proposed - some expensive preprocessing like precomp could be "outsourced" (assuming that it'd be used in ccm).
    But if you want to gain speed the main task remains - the creation of a multithreaded compression core. You can do this by just dividing the input data in 2,3,... chunks and you're set. But memory requirements grow linearly with the number of threads and some compression efficiency is lost (this heavily depends on the block size). If the block size is big enough this shouldn't be a problem. But still, this solution is ugly. On the other hand, context mixing is a concept which can be hardly parallelized. So, whatever you do here, you'll have to make some sacrifices.

    Finally, I compiled a 32-bit linux version for hahobas. Please do not post this version anywhere else. I hope it works as expected.

    link removed...

  27. #27
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by Christian
    Finally, I compiled a 32-bit linux version for hahobas. Please do not post this version anywhere else.
    its important step! i hope that you will make it public as dust settles

  28. #28
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 150 Times in 18 Posts
    Quote Originally Posted by Bulat Ziganshin
    its important step! i hope that you will make it public as dust settle
    Sure. If there arent any problems I dont see a reason for keeping it back. But for now, I just want to make sure, that it is working ok before making it public.
    So, if theres anything strange with this compile please leave me a line.
    I removed the link above because ccm (without x) was missing. This one includes both versions.

    link removed...

  29. #29
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by Christian
    Please do not post this version anywhere else.
    OK

  30. #30
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Christian, thanks for making this linux version. The performance hit really surprises me though:

    Test System: AMD Athlon 64 3000+ (Venice, single core) with 2GIG Corsair DDR CL2 RAM

    linux->ccm c 3 enwik8 (22325509 Bytes) - 1 m 57 s -
    linux->ccm d enwik8 - 1 m 59 s -
    windows->ccm.exe c 3 enwik8 (22325509 Bytes) - 1 m 34 s - compression 24.4% faster
    windows->ccm.exe d enwik8 - 1 m 37 s - decompression 22.6% faster
    -----------------
    linux->ccm c 7 enwik8 (21960598 Bytes) - 2 m 1 s -
    linux->ccm d enwik8 - 2 m 2 s -
    windows->ccm.exe c 7 enwik8 (21960598 Bytes) - 1 m 37 s - compression 24.7% faster
    windows->ccm.exe d enwik8 - 1 m 40 s - decompression 22.0% faster
    -----------------
    linux->ccmx c 3 enwik8 (21608042 Bytes) - 2 m 26 s -
    linux->ccmx d enwik8 - 2 m 30 s -
    windows->ccmx.exe c 3 enwik8 (21608042 Bytes) - 2 m 2 s - compression 19.6% faster
    windows->ccmx.exe d enwik8 - 2 m 3 s - decompression 21.9% faster
    -----------------
    linux->ccmx c 7 enwik8 (20819733 Bytes) - 2 m 32 s -
    linux->ccmx d enwik8 - 2 m 34 s -
    windows->ccmx.exe c 7 enwik8 (20819733 Bytes) - 2 m 6 s - compression 20.6% faster
    windows->ccmx.exe d enwik8 - 2 m 7 s - decompression 21.2% faster
    -----------------

    In case you're wondering why my timings for the windows version are faster than in some of my previous posts, it's because I upgrade d my kernel.

    Looking forward to the source being released.

    Also, just for curiosity, I tested my ccm.exe in windows XP as opposed to Wine, and got:
    ccm.exe c 3 enwik8 enwik8.ccm 1m 33sec

    which is about the same.

Page 1 of 3 123 LastLast

Similar Threads

  1. CCM decompiling
    By Shelwien in forum Data Compression
    Replies: 13
    Last Post: 17th August 2009, 20:47
  2. CCM 1.3x
    By Christian in forum Forum Archive
    Replies: 67
    Last Post: 25th April 2008, 20:22
  3. CCM(x) multithreaded ?
    By SvenBent in forum Forum Archive
    Replies: 2
    Last Post: 15th September 2007, 15:29
  4. CCM 1.2x branch
    By Christian in forum Forum Archive
    Replies: 107
    Last Post: 8th June 2007, 17:56
  5. CCM - 1.1.x branch
    By Christian in forum Forum Archive
    Replies: 105
    Last Post: 19th March 2007, 23:50

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •