exe prefilter quick comparison

[evg]

Hello all,

this is my first post to the forum. I'm no native english speaker so please bear
with me on bogus expression.

I've been playing around with the various exe prefilters floating around and have compiled a list of random compressors (fast/slow lz, bwt, ppm, cm) on the different filters for comparision.
Hope you find it usefull in some way.

Best regards
evg

              lzma    lpaq8   ccm     rzm     cmm4    paq8px
ffm.exe   [1] 2799620 2476898 2591466 2640073 2504964 1964021
ffm.bcj2  [2] 2719724 2479808 2596009 2648954 2585510 2012975
ffm.e8e9  [3] 2737546 2507203 2660452 2720321 2588577 2012470
ffm.e8flt [4] 2734668 2500968 2701028 2769142 2648842 1999812
ffm.flt32 [5] 2656717 2404250 2510738 2584191 2501921 1960591
ffm.paq8  [9] 2735918 2465710 2723637 2764968 2566167 1992513

                bcm     bliz    lzpm    ppmd    gzip    bzip2
ffm.exe       3251631 3178168 2990558 3058165 3604936 3523534
ffm.bcj2      3375638 3289914 3011673 3120777 3556903 3560015
ffm.e8e9      3201514 3285531 3060813 2989902 3525368 3452216
ffm.e8flt     3199186 3309636 3012545 2992000 3523933 3446092
ffm.flt32     3154370 3128719 2945082 2933253 3456748 3374660
ffm.paq8      3200374 3303402 3054876 2991850 3522724 3448906

               lzma   lpaq8  ccm    rzm    cmm4   paq8px
lame.exe   [6] 202672 189295 195561 202773 188040 153905
lame.bcj2  [7] 198986 191878 197494 204679 195297 158696
lame.e8e9      197820 196032 200395 208787 193795 158207
lame.e8flt     197843 192899 202378 211278 197348 155018
lame.flt32 [8] 195280 187195 193832 201052 190704 154989
lame.paq8      197552 189421 202279 210932 192859 154853

               bcm    bliz   lzpm   ppmd   gzip   bzip2
lame.exe       235744 232437 216387 221182 236207 252260
lame.bcj2      239878 241272 220907 223339 234645 255407
lame.e8e9      230723 239358 221498 215009 229400 245980
lame.e8flt     230678 241840 216448 215150 229526 245834
lame.flt32     231442 233216 215774 215279 228702 246896
lame.paq8      230696 241292 216551 215020 229177 246285

[1] ffmpeg svn-r18427 win32, size 9,807,360 bytes
[2] 7z 4.65 stored, size 9.813.285 bytes
[3] e8e9 unknown version from C, Mattern, same size as 1
[4] e8flt v2 from E, Shelwin, same size as 1
[5] flt32 unknown version from E, Shelwin, 7z stored 10,002,976 bytes

[6] lame 3.98.2 win32, size 503,808 bytes, same size for e8e9 and e8flt
[7] 7z stored, 504,401 bytes
[8] 7z stored, 511,975 bytes
[9] paq8px v31 stored, 9,810,573 bytes

Compressors and settings:
lzma 4,65 default, ~128mb; lpaq8 6, 198mb; ccm 1,30c 4, 274mb;
rzm 0,70h default, 258mb; cmm4 0,2b 64, 260mb
paq8px v31 -6, 233mb; bcm 0,7 default, ~50mb;
bliz 0,24b 10mb, ~60mb; lzpm 0,16 ex ~700mb; ppmd j -o16 -m256
gzip 1,3,12 -9, ~1mb; bzip 1,0,5 -9, ~7mb

Edit: Updated paq8px exe transform as suggested by Jan

[Jan Ondrus]

You can test also e8/e9 filter from paq8px.
"paq8px -0 x.exe" will transform your x.exe file.

x.exe.zip - 554503 bytes
x.exe.paq8px.zip - 536169 bytes

[evg]

You can test also e8/e9 filter from paq8px.
"paq8px -0 x.exe" will transform your x.exe file.

x.exe.zip - 554503 bytes
x.exe.paq8px.zip - 536169 bytes

Thanks a lot Jan, updated first post to include paq8px, somehow I have forgotten that one

[Shelwien]

I'd add http://www.compression.ru/ds/disasm32.rar
and http://shelwien.googlepages.com/durilca2_002a.rar (using -t3 -l options)

[Simon Berger]

But you should change the exe header to 0 to disable double detection and usage of more then one filter.
Won't paq8 at least detect it as an exe file again?

[evg]

I'd add http://www.compression.ru/ds/disasm32.rar
and http://shelwien.googlepages.com/durilca2_002a.rar (using -t3 -l options)

Thanks for the suggestions, I'll add those tomorrow.

But you should change the exe header to 0 to disable double detection and usage of more then one filter.
Won't paq8 at least detect it as an exe file again?

I don't have detailed logs, I'll recheck that. Tho I think I can remember that paq used the exe filter on already processed files.

Best regards
evg

[Jan Ondrus]

But you should change the exe header to 0 to disable double detection and usage of more then one filter.
Won't paq8 at least detect it as an exe file again?

No, paq8 won't detect transformed exe as exe-block and transformation is not applied again.
But exeModel is not used -> I think this is reason why paq8 has worse compression when applied on transformed file.

    // Detect EXE if the low order byte (little-endian) XX is more
    // recently seen (and within 4K) if a relative to absolute address
    // conversion is done in the context CALL/JMP (E8/E9) XX xx xx 00/FF
    // 4 times in a row.  Detect end of EXE at the last
    // place this happens when it does not happen for 64KB.

[evg]

Updated the table with disasm32 and durilca.
As the table gets quite confusing I decided to add it as attachment (excel, csv, html). Also includes logs for paq.

No, paq8 won't detect transformed exe as exe-block and transformation is not applied again.
But exeModel is not used -> I think this is reason why paq8 has worse compression when applied on transformed file.

True for paq compressing paq-0, still detects exe chunks in the other transforms. BTW how about writing the progress indicator to stderr? It would be more easy to produce logs that way.

Thanks again for the suggestions