CCM decompiling

[Shelwien]

http://...

sub_405AC0() is the main encoding function,
sub_401C50() is the model init function.
(dbg.log after encoding contains the actually compressed bytes -
the filtering result can be observed).
I only decompiled the source, but didn't really look at the
algorithm (maybe toffer would).
But overall it seems far from efficient - especially its rangecoder
with 31-bit registers and extra shifts (seems like Schindler's original
or a simple derivative).

[joerg]

if i understand well
- you have done a Re-Engineering
of the ccm130c from christian martelock

further you have improved the main loop of the program

is here a downloadable working windows-compile
for this ccm130c_improved ?

best regards

[osmanturan]

if i understand well
- you have done a Re-Engineering
of the ccm130c from christian martelock

further you have improved the main loop of the program

is here a downloadable working windows-compile
for this ccm130c_improved ?

best regards

No, not improved. Still it's as is. BTW, it should be CCM not CCMX. And posted files already include executables. But, they are ~2x slower due to reserve engineering works. We (shelwien, toffer and me) are still trying to figure out it's algorithms.

[Skymmer]

http://.../

sub_405AC0() is the main encoding function,
sub_401C50() is the model init function.
(dbg.log after encoding contains the actually compressed bytes -
the filtering result can be observed).
I only decompiled the source, but didn't really look at the
algorithm (maybe toffer would).
But overall it seems far from efficient - especially its rangecoder
with 31-bit registers and extra shifts (seems like Schindler's original
or a simple derivative).

Though I don't know what does Christian thinks about it and from my point of view its a questionable ethics here, anyway, Shelwien, nicely done!
I've made a couple of tests on enwik8 at 5 level.

         CCM v1.30c = 71.709
           deCCM_v0 = 211.416
deccm_v0_sub_405AC0 = 211.621
           deCCM_v1 = 202.411

Obviously, decompiled version almost 3 times slower. More exactly speaking 2.82 times. All output files are identical.
This fact and some other things bring some questions from me. Sorry if I dumb here
1.) Why decompiled versions are too much slower than originals? I remember that some time ago one dude called Z0MbiE ripped NRV engine from UPX and it was too slow too.
2.) I don't completely understand how its implemented. Decompiled ccm.exe is almost identical to original - there are only 3 differences in it. Two of them are in the header and last one is just changed imported DLL name. Original version imports KERNEL32.dll and decompiled one imports 1.dll. So what is that 1.dll file ? Where it comes from and what does it contain ?

[Shelwien]

> Though I don't know what does Christian thinks about it
> and from my point of view its a questionable ethics here,

I don't intend to use it in commercial programs,
so my conscience is silent. And people who want to prevent
reverse-engineering should at least write about that
in the license (but preferably also add a software protection).

> All output files are identical.

Well, thanks for testing then, as I changed the code quite a bit
and didn't really test it

> 1.) Why decompiled versions are too much slower than originals?

Because decompiled C code has to simulate the behaviour of "original"
asm code (to an extent).
And access to something defined like
uint& input_ptr = *(uint*)0x153B2A0;
is obviously slower than to just
uint input_ptr
as compiler can handle the later case much better.

> 2.) I don't completely understand how its implemented.
> Decompiled ccm.exe is almost identical to original - there
> are only 3 differences in it. Two of them are in the
> header and last one is just changed imported DLL name.
> Original version imports KERNEL32.dll and decompiled one
> imports 1.dll. So what is that 1.dll file ? Where it comes
> from and what does it contain ?

<Simon|B> and what does the dll do?
<Shelwien> as actually the stuff i already decompiled contains all the main algorithm
<Shelwien> except for filtering and rc
<Shelwien> but rc is dumb so i won't decompile it probably
<Shelwien> <osman> yep. but i wonder what that hell? using a DLL?
<Shelwien> <Shelwien> its my new decompiling method
<Shelwien> <Shelwien> attaching a dll to a program
<Shelwien> <Shelwien> and incrementally decompiling and moving the functions from program to dll
<Shelwien> <Shelwien> its really helpful
<Shelwien> so ccm.exe there does commandline handling, filtering, buffering and rangecoding
<Shelwien> and model init and encoding loop are in dll
<Simon|B> is it comparable in tems of speed with original?
<Shelwien> atm not sure. probably slower because of some thunks

So, 1.dll forwards some symbols from kernel32.dll and ccm.exe
uses it instead of kernel32.dll.
And then, when executable is loaded, its dlls are also loaded,
and their entrypoints are called before passing control to
the actual program.
Thus, 1.dll is able to patch some code and substitute
the functions contained in it instead of some of original ones.

[Simon Berger]

Also note that it's without any optimization. I compiled it myself and got 49 secfor original and and 64 sec for this version. Not so much difference. But as Shelwien said it's not really important at this point.

[Christian]

Hey guys.

Though I don't know what does Christian thinks about it and from my point of view its a questionable ethics here, anyway, Shelwien, nicely done!

It's still a mystery to me why everyone is still so obsessed with CCM.

But I think it should be pretty obvious that I don't want people to reverse-engineer my old compressors and release patched (or any) versions of it. Therefore, please stop doing it. Thanks!

PS: For additional information please refer to "The Twelve Compression Commandments".

[Nania Francesco]

They agree with Christian is not correct to reconstruct the code of others if owner of the code doesn't want that am open!

[joerg]

@christian martelock

welcome back into the forum

you were abstinent for a long time

1. if you do not want a re-engineering of your compressors
i think we should not do this further

but ccm is a very impressing program

please look at

https://sourceforge.net/forum/messag...msg_id=4173069

2007-02-22:
"It would be a crime if this amazing compressor wasn't developed to its full potential"

2. may be you can give us some hints about the algorithm

or

3. may be you can decide to release the source of one
of your older program versions

or maybe of slug 1.2x

you wrote on your webpage:

"Maybe next year, there will be Slug 1.3x coming
with a new approach... "

or any other

to allow us a further development

best regards

Joerg

and thank you again for your wonderful ccm

[Rugxulo]

@christian martelock

welcome back into the forum

you were abstinent for a long time

I don't think that means what you think it means (no offense intended, just funny!).

Main Entry: ab?sti?nence
Pronunciation: \ˈab-stə-nən(t)s\
Function: noun
Etymology: Middle English, from Anglo-French, from Latin abstinentia, from abstinent-, abstinens, present participle of abstinēre
Date: 14th century
1 : voluntary forbearance especially from indulgence of an appetite or craving or from eating some foods
2 a : habitual abstaining from intoxicating beverages b : abstention from sexual intercourse
? ab?sti?nent \-nənt\ adjective
? ab?sti?nent?ly adverb

[SvenBent]

Off topic:

Hi Christian.
would still love to experiment with your alternativ RZM version

[Piotr Tarsa]

Rugxulo:
Probably he meant 'absent'

[joerg]

please excuse me for my bad english
it should be "absent" - thank you piotr

but sadly - christian seems not to have time for us yet!

best regards

[Rugxulo]

please excuse me for my bad english
it should be "absent" - thank you piotr

Sorry, didn't mean to pick on you, please don't take offense, I just thought it was a little funny.