Page 1 of 3 123 LastLast
Results 1 to 30 of 68

Thread: CCM 1.3x

  1. #1
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 150 Times in 18 Posts
    A happy new year to everyone!
    I hope everyone has had a beautiful holiday. Last weekend I found some time and motivation to look at CCM again. The new version is a bit faster again.
    Most speed is gained by reducing/removing some data structures or LUTs. The compression loss is more or less compensated by a slightly improved match model.
    I want to thank Stephan Busch very much for testing some interim versions over the weekend.

    download CCM 1.30

  2. #2
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Chris!

    Mirror: Download

  3. #3
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Thanks Chris! CCMX Monster of compression!

  4. #4
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    thank you Chris

    Best regards!

  5. #5
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Just when I thought such a good thing couldn't get better, it did. Here's some benchmarks of 1.26b vs 1.30 on my system:

    Test System: AMD Athlon 64 3000+ (Venice, single core) with 2GIG Corsair DDR CL2 RAM

    ccm126b->ccm.exe 3 enwik8 (22325507 Bytes) - 1 m 36 s -
    ccm126b->ccm.exe d enwik8 - 1 m 37 s -
    ccm130->ccm.exe c 3 enwik8 (22351098 Bytes) - 1 m 25 s - compression 12.9% faster size .1% larger
    ccm130->ccm.exe d enwik8 - 1 m 26 s - decompression 12.7% faster

    ccm126b->ccmx.exe 3 enwik8 (21608039 Bytes) - 2 m 3 s -
    ccm126b->ccmx.exe d enwik8 - 2 m 5 s -
    ccm130->ccmx.exe c 3 enwik8 (21646059 Bytes) - 1 m 53 s - compression 8.8% faster size .1% larger
    ccm130->ccmx.exe d enwik8 - 1 m 57 s - decompression 6.8% faster

    ccm126b->ccm.exe 7 enwik8 (21960600 Bytes) - 1 m 39 s -
    ccm126b->ccm.exe d enwik8 - 1 m 40 s -
    ccm130->ccm.exe 7 enwik8 (21980533 Bytes) - 1 m 29 s - compression 11.2% faster size .09% smaller
    ccm130->ccm.exe d enwik8 - 1 m 28 s - decompression 13.6% faster

    ccm126b->ccmx.exe 7 enwik8 (20819731 Bytes) - 2 m 7 s -
    ccm126b->ccmx.exe d enwik8 - 2 m 10 s -
    ccm130->ccmx.exe 7 enwik8 (20857925 Bytes) - 1 m 57 s - compression 8.5% faster size .1% larger
    ccm130->ccmx.exe d enwik8 - 2 m 3 s - decompression 5.6% faster

    I don't know about everyone else out there, but ccm seems to be a terrific balance of speed and ratio. However, I wouldn't mind seeing CCMX's ratio increased at the cost of some speed (since it is only is about 30% slower). Maybe down to half the speed of regular CCM if it means getting 5% or so more ratio. It would also make CCM harder to catch by FreeArc ratiowise.

    Thanks for all your hard work Christian!

  6. #6
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Here's a few more this time with enwik9, and lots of memory.

    ccm126b->ccmx.exe 7 enwik9 (174146696 Bytes) - 20 m 4 s -
    ccm126b->ccmx.exe d enwik9 - 20 m 24 s -
    ccm130->ccmx.exe 7 enwik9 (174142092 Bytes) - 18 m 41 s - compression 7.4% faster size .0026% smaller
    ccm130->ccmx.exe d enwik9 - 18 m 58 s - decompression 7.5% faster

  7. #7
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 150 Times in 18 Posts
    Thank you all!

    @Francis
    Ive seen your benchmark, good job! Though, there are some broken characters when I view the page. Anyway, cant wait for new numbers.


    @Hahobas
    Thanks for giving the new version a try. Its a little bit sad, that the increase in performance is so different on different platforms. Ive moved to C2D recently and there I have much better results:

    CCM 3 on ENWIK8:
    1.26 -> 21802.25 KiB (ratio 22.33%, speed 2355 KiB/s), ~41s
    1.30 -> 21827.24 KiB (ratio 22.35%, speed 2891 KiB/s), ~34s

    The first interim version Stephen tested was much faster on his Operton, too. Ill look into this further. Perhaps its just a compiler switch again.

    Quote Originally Posted by Hahobas
    I dont know about everyone else out there, but ccm seems to be a terrific balance of speed and ratio. However, I wouldnt mind seeing CCMXs ratio increased at the cost of some speed
    In theory this is a very good suggestion. But practically, CCMX is just CCM plus ~10 lines more code. I cant improve the one version much without heavily effecting the other one. And I do not want to develop two seperate branches because of serveral reasons - mostly time and motivation.
    Nonetheless, Ill probably rewrite CCM or something similar from scratch one day. And this one will most probable have stronger compression.

  8. #8
    Member
    Join Date
    May 2008
    Location
    CN
    Posts
    45
    Thanks
    0
    Thanked 0 Times in 0 Posts
    ratio is down?
    CCM 3 on ENWIK8:
    1.26 -> 21802.25 KiB (ratio 22.33%, speed 2355 KiB/s), ~41s
    1.30 -> 21827.24 KiB (ratio 22.35%, speed 2891 KiB/s), ~34s

  9. #9
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by Hahobas
    I dont know about everyone else out there, but ccm seems to be a terrific balance of speed and ratio.
    being modest, i think that ccm is 2nd best compression program of the year

    Christian, isnt it possible to add more models which will be used only in ccmx?

    another suggestions:
    - better table analysis with ability to substract successive elements
    - splitting imput file into several pieces and compressing them in parallel (for multicore cpus)
    - segmentation (it seriously improves compression for durilca - look at MFC results with and without -t1 switch). http://www.compression.ru/ds/seg_file.rar is publicshed by DS, durilcas author - i dont know whether it is technique actually used there

  10. #10
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > - better table analysis with ability to substract successive elements

    I'd never seen a single case where a simple delta was the best solution,
    actually. Sometimes just a proper 2D context is enough, sometimes its
    good to subtract an extrapolation by some previous lines, but never
    simple delta was the best.

    Btw, I'm currently thinking about a design with dynamic symbol ranking
    for unary coding. Well, typical solution is to sort symbols by context
    order, then by MtF rank (aka number of different symbols since last
    occurence). And it works much better than seemingly "more valid"
    solutions like descending probability order. But anyway these are
    empiric methods without any foundations.

    Well, it might seem unrelated, but its actually the same thing as with
    deltas in 2D table encoding. I think its obvious if you consider unary
    encoding - delta gives rank0 to the same value as previous etc.

    So, my current opinion is that rank0 should be assigned to the symbol
    with max codelength in some context (descending codelength order).
    Higher (=lesser) rank usually means more complex model and more precise
    estimation in unary schemes, so that's why.

    Any suggestions?

    > - segmentation (it seriously improves compression for durilca - look at MFC
    > results with and without -t1 switch). http://www.compression.ru/ds/seg_file.rar
    > is publicshed by DS, durilca's author - i don't know whether it is technique
    > actually used there

    Code:
     
    -t1 - segmentation of file (default: (5,(0-3))); 
        -t1[(k[,(b1-t1)[,...]])] 
            k - precision of segmentation; 
            (b1-t1) - apply trick number t1 to block number b1; 
            (0-t1)  - apply trick number t1 to all blocks;

    Actually, in first durilca releases was even a hidden "-l" switch
    for dumping these segments into separate files... With it I made
    a bruteforce optimizer of durilca parameters for Werner

  11. #11
    Member
    Join Date
    Jan 2008
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Do you have a linux version available yet ?
    I love to compress my large syslogfiles with ccmx since I can use the stdin and stdout now and since it's text I get rather high ratios.

    I'd love the speed gain to be available on linux too.

  12. #12
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 150 Times in 18 Posts
    Quote Originally Posted by l1t
    ratio is down?
    It depends on the data - for some files ratio is "better". Anyway, in my opinion the speed increase more than justifies the very tiny compression loss.

    Quote Originally Posted by Bulat Ziganshin
    being modest, i think that ccm is 2nd best compression program of the year
    Being modest myself, I think I could live with that.


    Quote Originally Posted by Bulat Ziganshin
    Christian, isnt it possible to add more models which will be used only in ccmx?
    It is possible, but this would be too much work. CCM isnt as construction-kit-like as PAQ where you just plug in a new model and youre set.

    Quote Originally Posted by Bulat Ziganshin
    another suggestions:
    - better table analysis with ability to substract successive elements
    - splitting imput file into several pieces and compressing them in parallel (for multicore cpus)
    - segmentation...
    Currently, I dont do any table analysis. Is it really worth it? But all your suggestions are already on my todo list. Honestly, I work very rarely on compression stuff these days. CCM means sparetime and fun - when I look at that todo list its not fun anymore. So I just put it aside.


    Quote Originally Posted by Jeroen
    Do you have a linux version available yet?
    ...
    Id love the speed gain to be available on linux too.
    Sorry, I first want to sort out the compiler-related speed anomalies on windows. Maybe the next version will get a linux compile again.

  13. #13
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Christian, please make unix versions too. this means that all your users will be happy and ccm will be useful to transfer data between systems

  14. #14
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 150 Times in 18 Posts
    Here is 1.30a which is fully compatible with 1.30. I just used different compiler switches. On my system both versions are equally fast, but on Stephan's Operton 1.30a is ~10% faster. So I recommend the new version.
    If anyone has some time to spare feel free to compare both versions for speed and post your results (speed, system, # of runs).

    download CCM 1.30a

    About the linux version - I'll put something together in the next days.

  15. #15
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Chris!

    Mirror: Download

  16. #16
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Have made a will the version 1.30a and on my Intel Core duo 2 E6600 has the same speed around of the precedent! however always stays "Monster of compression!"

  17. #17
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Surprisingly CCM 1.30a didn't perform much differently. Out of curiosity, what compiler switches are you using?

    Test System: AMD Athlon 64 3000+ (Venice, single core) with 2GIG Corsair DDR CL2 RAM

    ccm130->ccm.exe 3 enwik8 (22351098 Bytes) - 1 m 24 s -
    ccm130->ccm.exe d enwik8 - 1 m 25 s -
    ccm130a->ccm.exe 3 enwik8 (22351098 Bytes) - 1 m 24 s - No Change
    ccm130a->ccm.exe d enwik8 - 1 m 25 s - No Change
    -----------------
    ccm130->ccmx.exe 3 enwik8 (21646059 Bytes) - 1 m 51 s -
    ccm130->ccmx.exe d enwik8 - 1 m 56 s -
    ccm130a->ccmx.exe 3 enwik8 (21646059 Bytes) - 1 m 51 s - No Change
    ccm130a->ccmx.exe d enwik8 - 1 m 54 s - decompression 1.7% faster

  18. #18
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    The best compressor of the world! for me!
    MONSTER OF COMPRESSION
    ________________-Time comp-Time dec.- 306.189.145
    1) CCMX 1.30a___ 343,695 344,236 -141.140.962
    2) CCM 1.30a____ 324,527 323,926 -141.572.117
    3) LPAQ8________ 787,837 794,032 -141.591.047

  19. #19
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 150 Times in 18 Posts
    Thanks Francesco and Hahobas.
    Quote Originally Posted by Hahobas
    Surprisingly CCM 1.30a didnt perform much differently. Out of curiosity, what compiler switches are you using?
    Im using these switches: "-fomit-frame-pointer -fexpensive-optimizations -O3 -mmmx -s"
    Before, I used "-march=pentiumpro" in addition to that. It turned out, that without it, CCM is 500 seconds faster on Stephans system (5000 vs 5500 seconds). For you and me it obviously doesnt make a difference. Any ideas?

  20. #20
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    What version of GCC are you using?

  21. #21
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Hi Christian. Fantastic new release. Thanks that you give us your skill without fun while working one it .
    It?s absolutly more then a good ratio improvement. The bigger file size is like nothing. Keep up the way you go in developing a superb fast CM. Please don?t improve compression with loss of speed (other direction ins fine :-P).

  22. #22
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 150 Times in 18 Posts
    Quote Originally Posted by Hahobas
    What version of GCC are you using?
    Last time I checked, it was 3.4.2.

    Quote Originally Posted by Simon Berger
    It?s absolutly more then a good ratio improvement. The bigger file size is like nothing. Keep up the way you go in developing a superb fast CM. Please don?t improve compression with loss of speed
    Thanks! Substancial compression improvements are unlikely to happen until I do a complete rewrite. Regarding further speed improvements - if I simplify the algorithm even further therell only be NOOP-methods left.

  23. #23
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    For Christian "King of Compression"
    For more speed (+10-20%)
    download : gcc-4.1.2-mingw installer
    download :code::Blocks

  24. #24
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Thanks a lot for the new version, Christian!

    Quote Originally Posted by Nania Francesco Antonio
    For more speed (+10-20%)
    download : gcc-4.1.2-mingw installer
    It depends... I dont have any experience myself, but tests on one media codec packs forum have shown, that GCC 4 binaries are 1-2% slower (and smaller) than those produced by GCC 3.

  25. #25
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    all tests run with default settings
    no time information (sorry)

    redline.tar - 363 MB (381,501,952 bytes)

    redline.tar.ccm126 - 190 MB (199,526,883 bytes)
    redline.tar.ccmx126 - 190 MB (199,344,496 bytes)
    redline.tar.ccm130 - 190 MB (199,316,296 bytes)
    redline.tar.ccmx130 - 189 MB (199,131,526 bytes)
    redline.tar.ccm130a - 190 MB (199,316,296 bytes)
    redline.tar.ccmx130a - 189 MB (199,131,526 bytes)


    So i se a compression efficiency gain in the 1.30 version


    System
    Core2 Quade Q6600 OC to 3.2ghz (8x400)
    4x1 GB cas4 800mhz memory (1:1)
    Win Vista 64bit ultima

  26. #26
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    ccmx 7 compression is slightly worse on enwik8, slightly better on enwik9. Speed is better.
    http://cs.fit.edu/~mmahoney/compression/text.html#1741

  27. #27
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    There's really no one answer as to which GCC version is the best for performance. If the source code were available I'd be happy to mess around with different compiler options.

    I'm no expert, but, I thought I'd do some messing with switches for Matt's fpaqc.cpp for kicks and share the result.

    I started out with your switches you use for CCM. I was able to get about 27% better performance using gcc 4.1.2-33 instead of gcc 3.4. So I would definitely try gcc 4.1.2 or newer with CCM.

    Next, I tried fiddling with the switches to get even more performance out of fpaqc. By adding the following switches I was able to squeeze about 8% more speed out of it.

    -floop-optimize2 -mtune=prescott

    As I understand it, using mtune still maintains backwards compatibility with older processors, but just optimizes for a particular platform. I also found it strange that the -mtune=prescott (A flavor of Pentium 4's) gave me slightly better performance than -mtune=athlon64 (which is what my processor is).

    Later, for curiosity sake, I tried 64bit compiles and was able to get an additional 8% performance. So if you ever feel generous, you could provide us with 64bit binaries.

    But again, this was not with CCM, so who knows what the best all around compiler options for everyone are.

  28. #28
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by toffer
    When looking to assembly code produces by gcc, i was shocked, that it wasnt translated to mmx or sse (compiler switches turned on!), something like:

    a[0] += b[0]
    a[1] += b[1]
    a[2] += b[2]
    a[3] += b[3]
    This is quite a surprising discovery by toffer. Is there any code in CCM that could be suffering from gccs poor mmx/sse support? I suppose one way easy way to find out is to compile it with intels compiler and compare.

  29. #29
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 150 Times in 18 Posts
    Hi again!
    Sorry for the late reply. So, finally I do have some fresh linux binaries for you. I included binaries produced with GCC 3.4.6 and 4.1.2.

    CCM 1.30a (win32+linux32)

    Im quite sure, that the ones created with 4.1.2 are slower, but now you can try. There wont be any intel binaries. I do not want to spend my time on compiler related issues/trials.
    The windows binaries are untouched. GCC 4.12 is much slower for me - at least with CCM.

    Quote Originally Posted by Nania Francesco Antonio
    For more speed (+10-20%)
    download : gcc-4.1.2-mingw installer
    download :code::Blocks
    I tried code::blocks - its really nice. Thanks for the tip.

    Quote Originally Posted by Hahobas
    Is there any code in CCM that could be suffering from gccs poor mmx/sse support?
    Frankly, Im sure that GCC is messing up some code. But there is not any code like the one posted. Anyway, I rather prefer looking into changes/improvements to the algorithm than tweaking code.

  30. #30
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I wonder how you do weighting without a dot product?! (or some vector math)
    M1, CMM and other resources - http://sites.google.com/site/toffer86/ or toffer.tk

Page 1 of 3 123 LastLast

Similar Threads

  1. CCM decompiling
    By Shelwien in forum Data Compression
    Replies: 13
    Last Post: 17th August 2009, 20:47
  2. CCM 1.25 is here!
    By Christian in forum Forum Archive
    Replies: 84
    Last Post: 16th November 2007, 11:00
  3. CCM(x) multithreaded ?
    By SvenBent in forum Forum Archive
    Replies: 2
    Last Post: 15th September 2007, 15:29
  4. CCM 1.2x branch
    By Christian in forum Forum Archive
    Replies: 107
    Last Post: 8th June 2007, 17:56
  5. CCM - 1.1.x branch
    By Christian in forum Forum Archive
    Replies: 105
    Last Post: 19th March 2007, 23:50

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •