CCMx Benchmark, compress on stored .TAR/.7z/.RAR

[Raymond_NGhM]

CCMX 1.30a

1st Test,

Memory mode:0

List of files:

5,371,438 LZTurbo.asm
193,353 LZTurbo.exe.nml.txt
2,350 LZTurbo.exe.str.txt
191,440 _bss.bin
2,676 _data.bin
9,968 _rdata.bin
-----------------------------
5,771,225 bytes Total


TAR 5.50 MB (5,776,896 bytes) --> TAR.CCMX 271 KB (278,495 bytes) good
RAR 5.50 MB (5,771,535 bytes) --> RAR.CCMX 280 KB (286,939 bytes)
7Z 5.50 MB (5,771,482 bytes) --> 7Z.CCMX 280 KB (286,895 bytes)
-----------------------------------------------------------------
2nd Test,

Memory mode:3

List of files:

08/04/2004 12:56 AM 8,384,000 shell32.dll
08/04/2004 12:56 AM 4,874,240 wmp.dll
08/04/2004 12:56 AM 4,274,816 nv4_disp.dll
08/04/2004 12:56 AM 3,003,392 mshtml.dll
08/04/2004 12:56 AM 2,940,928 wmploc.dll
08/04/2004 12:56 AM 2,897,920 xpsp2res.dll
08/04/2004 12:56 AM 2,804,224 msi.dll
08/04/2004 12:56 AM 2,113,536 dxdiagn.dll
08/04/2004 12:57 AM 2,105,344 wmvcore.dll
08/04/2004 12:56 AM 2,067,968 cdosys.dll
08/04/2004 12:56 AM 1,888,992 ati3duag.dll
08/04/2004 12:56 AM 1,737,856 mtxparhd.dll
08/04/2004 09:27 AM 1,712,128 GdiPlus.dll
08/04/2004 12:56 AM 1,708,032 netshell.dll
08/04/2004 12:56 AM 1,689,088 d3d9.dll
08/30/2002 01:00 PM 1,677,312 wmvcore2.dll
08/04/2004 12:56 AM 1,580,544 sfcfiles.dll
08/04/2004 12:56 AM 1,507,356 msjet40.dll
08/30/2002 01:00 PM 1,501,696 diskcopy.dll
08/04/2004 12:56 AM 1,483,264 shdocvw.dll
08/04/2004 12:56 AM 1,435,648 query.dll
08/04/2004 12:56 AM 1,428,480 msvidctl.dll
08/04/2004 12:56 AM 1,392,671 msvbvm60.dll
08/30/2002 01:00 PM 1,355,776 msvbvm50.dll
08/04/2004 12:56 AM 1,294,336 dsound3d.dll
08/04/2004 12:56 AM 1,287,680 quartz.dll
08/04/2004 12:56 AM 1,281,536 ole32.dll
08/04/2004 12:56 AM 1,251,840 comsvcs.dll
08/04/2004 12:56 AM 1,236,480 msxml3.dll
08/04/2004 12:56 AM 1,227,264 dx8vb.dll
08/04/2004 12:56 AM 1,192,960 mmcndmgr.dll
08/04/2004 12:56 AM 1,179,648 d3d8.dll
08/04/2004 12:56 AM 1,134,592 wuaueng.dll
08/04/2004 12:56 AM 1,119,744 wmsdmoe2.dll
08/04/2004 12:56 AM 1,082,368 esent.dll
08/04/2004 12:56 AM 1,053,696 danim.dll
08/04/2004 12:56 AM 1,050,624 wmnetmgr.dll
08/04/2004 12:56 AM 1,028,096 mfc42.dll
08/04/2004 12:56 AM 1,024,000 mfc42u.dll
08/04/2004 12:56 AM 1,016,832 browseui.dll
08/04/2004 12:56 AM 1,001,472 wmvdmoe2.dll
08/04/2004 12:56 AM 994,304 msgina.dll
08/04/2004 12:56 AM 984,576 syssetup.dll
08/04/2004 12:56 AM 983,552 kernel32.dll
08/04/2004 12:56 AM 983,552 setupapi.dll
08/04/2004 12:56 AM 949,248 msdtctm.dll
02/28/2003 06:26 PM 947,472 msjava.dll
08/04/2004 12:56 AM 937,984 winbrand.dll
08/04/2004 12:56 AM 896,512 wmspdmoe.dll
08/04/2004 12:56 AM 884,736 msimsg.dll
08/04/2004 12:56 AM 875,008 netplwiz.dll
08/04/2004 12:56 AM 870,784 ati3d1ag.dll
08/04/2004 12:56 AM 858,624 tapi3.dll
08/04/2004 12:56 AM 831,519 mswdat10.dll
08/04/2004 12:56 AM 825,344 d3dim700.dll
08/04/2004 12:56 AM 809,984 wmvdmod.dll
08/04/2004 12:56 AM 792,064 comres.dll
08/04/2004 12:56 AM 764,928 winntbbu.dll
08/04/2004 12:56 AM 759,296 wmsdmod.dll
08/04/2004 12:56 AM 755,200 ir50_32.dll
08/04/2004 12:56 AM 733,696 qedwipes.dll
08/04/2004 12:56 AM 723,456 userenv.dll
08/04/2004 12:56 AM 721,920 lsasrv.dll
08/04/2004 12:56 AM 713,728 opengl32.dll
08/04/2004 12:56 AM 713,216 sxs.dll
08/04/2004 12:56 AM 708,096 ntdll.dll
08/04/2004 12:56 AM 701,440 msxml2.dll
08/04/2004 12:57 AM 695,296 drmv2clt.dll
08/04/2004 12:56 AM 678,400 inetcomm.dll
08/04/2004 12:56 AM 670,720 wmadmoe.dll
08/04/2004 12:56 AM 660,992 mqqm.dll
08/04/2004 12:56 AM 657,920 rasdlg.dll
08/04/2004 12:56 AM 656,384 wininet.dll
08/03/2004 10:59 PM 655,360 mstscax.dll
08/04/2004 12:56 AM 640,000 dbghelp.dll
08/04/2004 12:56 AM 628,224 catsrvut.dll
08/04/2004 12:56 AM 622,080 netcfgx.dll
08/04/2004 12:56 AM 619,008 dx7vb.dll
08/04/2004 12:56 AM 616,960 advapi32.dll
08/04/2004 12:56 AM 614,912 h323msp.dll
08/04/2004 12:56 AM 614,429 mswstr10.dll
08/04/2004 12:56 AM 611,328 comctl32.dll
08/04/2004 12:56 AM 601,088 urlmon.dll
08/04/2004 12:56 AM 597,504 crypt32.dll
08/04/2004 12:56 AM 596,992 wsecedit.dll
08/30/2002 01:00 PM 590,336 d3dramp.dll
08/04/2004 12:56 AM 589,312 wiashext.dll
08/04/2004 12:56 AM 586,240 mlang.dll
08/04/2004 12:56 AM 581,120 rpcrt4.dll
08/04/2004 12:56 AM 577,024 user32.dll
08/04/2004 12:56 AM 566,784 gpedit.dll
08/04/2004 12:56 AM 562,176 qedit.dll
08/04/2004 12:56 AM 560,640 printui.dll
08/04/2004 12:56 AM 553,472 oleaut32.dll
08/04/2004 12:56 AM 552,989 msrepl40.dll
08/04/2004 12:56 AM 549,376 shdoclc.dll
08/04/2004 12:56 AM 540,160 comuid.dll
08/04/2004 12:56 AM 537,088 msftedit.dll
08/04/2004 12:56 AM 530,432 mstime.dll
08/04/2004 12:56 AM 526,848 p2psvc.dll
08/04/2004 12:56 AM 517,632 mqsnap.dll
08/04/2004 12:56 AM 516,768 ativvaxx.dll
08/04/2004 12:56 AM 512,029 msexch40.dll
08/04/2004 12:56 AM 506,368 msxml.dll
08/04/2004 12:56 AM 501,248 clbcatq.dll
08/04/2004 12:56 AM 498,205 dxmasf.dll
08/04/2004 12:56 AM 488,448 ntmsmgr.dll
08/04/2004 12:56 AM 484,864 wmspdmod.dll
08/04/2004 12:56 AM 473,600 shlwapi.dll
08/04/2004 12:56 AM 471,552 mqutil.dll
08/04/2004 12:56 AM 463,360 wiadefui.dll
08/04/2004 12:56 AM 457,728 certmgr.dll

08/03/2004 11:20 PM 2,180,992 ntoskrnl.exe
08/03/2004 10:59 PM 2,056,832 ntkrnlpa.exe
08/04/2004 12:56 AM 1,298,432 dxdiag.exe
08/04/2004 12:56 AM 1,200,128 ntbackup.exe
08/04/2004 12:56 AM 815,104 mmc.exe
08/04/2004 12:56 AM 602,624 autoconv.exe
08/04/2004 12:56 AM 588,800 autochk.exe
08/04/2004 12:56 AM 580,608 autofmt.exe
08/04/2004 12:56 AM 502,272 winlogon.exe
---------------------------------------------------
Total 121 File(s) 128MB (134,307,806 bytes)

TAR 128 MB (134,376,448 bytes)--> TAR.CCMX 33.7 MB (35,343,759 bytes)
RAR 128 MB (134,313,012 bytes)--> RAR.CCMX 34.2 MB (35,871,236 bytes)
7Z 128 MB (134,309,584 bytes)--> 7Z.CCMX 33.6 MB (35,320,256 bytes) good

RAR/7Z headers has smaller size than TAR.
But in 7Z:
7Z combined lists of archive, like Path/Folders/Filenames/CRC32/...
& compress in one place then save as in EOF. that's affect good achive
in compression ratio.

WinRK 3.0.3 Store Mode, work same as 7z but it was bug that show
CRC error during in test time.

Well done, thanks Christian M.

[Intrinsic]

Hi,
Possible to try this same test but using QFC? Was mainly aimed at older archivers who's filename storage within the archive wasn't that efficient, but have had some good results with modern ones after playing with the different filename sorting options.

http://www.geocities.com/jadoxa/qfc/index.html

[Bulat Ziganshin]

Possible to try this same test but using QFC?

i recommend to use freearc for the same puposes. its much better:
* allows to configure any compression algorithms called as external executable
* provides prepared settings for ccm, lpaq and so on
* allows to run algorithms sequentially (such as bcj+delta+ccm)
* compress data and directory independently using independent algorithms
* allows to easily test data after compression
* logs all the timings/sizes to logfile
* allows to sort files in any order

imho, its really hard to find reasons not to use it. the only problem is that it doesnt automatically multithread such compression

[Raymond_NGhM]

Hi, Intrinsic
New test with QFC and add best sort for matching,

1st Test
Memory mode:0

Default sorted files:

TAR 5.50 MB (5,776,896 bytes)--> TAR.CCMX 271 KB (278,495 bytes)
RAR 5.50 MB (5,771,535 bytes)--> RAR.CCMX 280 KB (286,939 bytes)
.7Z 5.50 MB (5,771,482 bytes)--> .7Z.CCMX 280 KB (286,895 bytes)
QFC 5.50 MB (5,771,385 bytes)--> QFC.CCMX 272 KB (279,019 bytes)

Sorted Name+Ext.

TAR.CCMX Same as default
RAR.CCMX Same as default
.7Z.CCMX Same as default
QFC.CCMX Same as default
---------------------------------------
2nd Test,
Memory mode:3

Default sorted files:

TAR 128 MB (134,376,448 bytes)--> TAR.CCMX 33.7 MB (35,343,759 bytes)
RAR 128 MB (134,313,012 bytes)--> RAR.CCMX 34.2 MB (35,871,236 bytes)
QFC 128 MB (134,310,810 bytes)--> QFC.CCMX 34.1 MB (35,792,646 bytes)
.7Z 128 MB (134,309,584 bytes)--> .7Z.CCMX 33.6 MB (35,320,256 bytes)

Sorted by Name+Ext.

TAR.CCMX Same as default (sorted only by name)
.7Z.CCMX Same as default (7Zip sort only by name, it can't support name+ext)
RAR.CCMX 33.6 MB (35,318,073 bytes)
QFC.CCMX 33.6 MB (35,314,364 bytes) Best

Finally I think names+ext is finely

[toffer]

I dont know how ccm detects executable data. But if it does detection based on the file header, tarring everything together, will bypass the exe filter.

Its really better to use freearc:

* allows to run algorithms sequentially (such as bcj+delta+ccm)

If im not wrong this is 7zips x86 filter.

[Bulat Ziganshin]

it was just an example. for ccm with it's built-in filter it may even decrease compression. i don't know how ccm exe filter works but think that it should be ok with tarred files because it's typical usage of ccm

[Intrinsic]

Cheers Raymond, QFC does pretty good in that last case.

As a random bit of maybe (un)interesting info, using the option to Encrypt file names(and setting a password of course or it won't encrypt them) in RAR i have achieved better compression on some datasets that having no password/no encrypted file names, this happened on very very few occaisons though and only made a few tenths of a % difference, and in general will increase the size of your archives.