FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)

[Biozynotiker]

the drawback is that srep executable will become larger

This should not be a problem, imho.

Currently i don't use srep with stdin/out to minimizing the "disk trashing", because this is the only way for me to check the return codes form every step of the compression job (rar wrapper).
A 1-Pass solution could be useful in many situations.

[Biozynotiker]

>Or compressing same kind of iso's (e.g. all windows7, 2008r2, winpe, versions and variants) ~60gig to 3gig.
amazing!

It's not that amazing than it's look like!

Win7, 2008 using over 90% the same files (at the same service pack level).
All files are compressed into *.wim files on that isos. The *.wim files are non solid archives.
So even in compressed state, the files are mostly the same.

I think exdupe can do the same "magic" here.

[Bulat Ziganshin]

yes, it's what i mean. the magic is multi-gigabyte range deduplication, so srep, exdupe, or Shelwien's program should perform pretty the same

[BetaTester]

Update on the SREP, means a new version of FreeArc?

[qqbenko]

BetaTester : http://freearc.org/Download-Alpha.aspx

[Bulat Ziganshin]

i've set up Mercurial repository: http://source.freearc.org:8002/freearc/, although URL may be changed later

[sourena]

hi all
compressing FreeArc-0.666-win32.exe using FreeArc-0.666-win32.exe(installed on win7) using gui ,original file size : 7,938,425 bytes
compressed size using maximum(ccm) : 8,022,826 bytes
why is that? is it already compressed? with what program/algorithm?

[Bulat Ziganshin]

algo=lzma, program=nsis installer

[qqbenko]

Hmmm.. Maybe a SREP 3.0 (x64) bug ?


Config : Ph II x6 ,8 GB RAM ,W7 x64.

Edit : Ok. Thanks for the answer.

[Bulat Ziganshin]

i don't remember exactly, but probably seen it sometimes too. i think it's problem in win7 mem allocation routines

[Bulat Ziganshin]

SREP 3.01 (March 4, 2012)

• speed up the 32-bit executable by calculating 32-bit hash (instead of 64-bit); it may hurt speed/compression on very large files

[Skymmer]

Its curious what "very large" means here

[Bulat Ziganshin]

i've tried it on LostPlanets2 archive lp2.pcf: 22,069,494,174 bytes

-m1:
srep64: 7,284,431,814 bytes in 151 seconds
srep32 3.0: 7,284,431,814 bytes in 285 seconds
srep32 3.01: 7,284,443,650 bytes in 159 seconds


-m3:
srep64: 7,009,388,668 bytes in 273 seconds
srep32 3.0: 7,009,388,668 bytes in 428 seconds
srep32 3.01: hasn't finished in 2 hours

probably, it's because reduced 32-bit hash produced too much collisions so srep keeps trying false match candidates. I should collect more stats to understand the situation (why who can help me by doing this work?)

the stats are required to understand whether the same problem possible in -m1 mode. If -m1 is guaranteed to work fast, i can incorporate into srep32 both matchfinders so that new one automatically used in -m1 mode. It's especially important since freearc 0.80 is expected to include 32-bit srep -m1 engine, so faster hashing will substantially inprove its speed

[neit]

Hello everyone,

I'm currently making place on my hhd by compressing some huge files with FreeArc GUI (GTA4 games files at the moment).
Here's the log:

D:\Apps\FreeArc\bin>FreeArc a -mx -ld1600m -k -s -dpF:\Steam_backup\common -- F:\Steam_backup\common\grand.theft.auto.iv.arc grand theft auto iv\
FreeArc 0.666 Using additional options: --logfile=D:\Apps\FreeArc\freearc.log --workdir=F:\
Creating archive: F:\Steam_backup\common\grand.theft.auto.iv.arc using rep:1567440kb+exe+delta+tempfile+lzma:177mb:normal:bt4:128, $obj => rep:1567440kb+delta+tempfile+lzma:177mb:normal:bt4:128, $text => dict:128mb:80%:l8192:m400:s100+lzp:160mb:92%:145:h23:d1mb+ppmd:16:384mb, $wav => tta, $bmp => mm+grzip:8mb:m1:l2048:h15:a
Memory for compression 1811mb, decompression 1539mb, cache 256kb
Compressed 5,384 files, 16,542,222,691 => 14,279,881,060 bytes. Ratio 86.3%
Compression time: cpu 8966.70 secs, real 6515.14 secs. Speed 2,539 kB/s

86.3% of ratio is not bad and I'd like to give a try with SREP.
So I extracted srep and srep64 to the bin directory and edited my arc.ini (Future-LZ optimization).

Now, how can I tell FreeArc GUI to use srep instead of rep ?
I suppose i have to modify some other lines in arc.ini, but since I'm a noob in compression stuff I prefer asking you the right way to do.

Hope my English isn't so bad and thank you for helping me.

[JangoFatXL]

You can do this with latest alpha: http://freearc.org/Download-Alpha.aspx

[neit]

You can do this with latest alpha: http://freearc.org/Download-Alpha.aspx

Thanks.
Here's the result with -mc:rep/srep

Memory for compression 434mb, decompression 352mb, cache 16mb
Compressed 5,384 files, 16,542,222,691 => 14,323,260,810 bytes. Ratio 86.5%
Compression time: cpu 1950.12 secs, real 1649.27 secs. Speed 10,030 kB/s

And the best compression I had (Worms Reloaded)

Memory for compression 434mb, decompression 352mb, cache 16mb
Compressed 2,333 files, 3,557,700,857 => 505,451,701 bytes. Ratio 14.2%    
Compression time: cpu 313.72 secs, real 309.53 secs. Speed 11,494 kB/s

Extracted 2,333 files, 505,450,869 => 3,557,700,857 bytes. Ratio 14.2%     
Directory 22,239 => 71,303 bytes. Ratio 31.1%
Extraction time: cpu 34.98 secs, real 67.20 secs. Speed 52,942 kB/s

[Bulat Ziganshin]

finally shared archive i've used a lot for SREP testing: http://freearc.org/download/testdata...rc.pcf.srep.7z

[Biozynotiker]

probably, it's because reduced 32-bit hash produced too much collisions so srep keeps trying false match candidates. I should collect more stats to understand the situation (why who can help me by doing this work?)

I've done some tests, but I can't reproduce similar problems.
I tried the 32bit version on my 64bit os, does this matter?
Does this behavior only occur on 32bit systems?

Btw, could this be a problem of the -a option? I always use -a1 to save memory.

[Bulat Ziganshin]

-a: possible, please try without it. if you can download abovementioned 4gb file, you will get the exact 22 gb file on which i found the problem. i'm also working on 64-bit win7, and have 16gb ram, of those 4.5 gb is RAM disk. amount of RAM remaining may be extremely important for reproducing this test (and -a affects this)

[Biozynotiker]

I can easily reproduce the srep (32bit v3.01) behavior with you lost planet test file. It doesn't matter witch -a value is set.
I tested it also with other big files (200gb database) and i have not seen such behavior.

The first 12gb form the lost planet test file can be reduced by srep to about 6gb and its done very fast.
The last last 10gb can be reduced to 1gb and the progress is very slow.

I cut the last 10gb out of the file, and have the exact same behavior, so this is not always happend to "big" files only,
but maybe only on very high "compressable" files.

[jimbow]

Hi Bulat,
Do you have plan to add directory support to SREP? It would nice to have this feature like exdupe. I prefer SREP because files usually compress better than after using exdupe.
Thanks

[Bulat Ziganshin]

no, i plan to add SREP inside FreeArc instead: http://code.google.com/p/freearc/issues/detail?id=235

and in the latest FreeArc 0.67 you can enable using of external SREP just by enabling the checkbox