FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)

[Skymmer]

Somehow I didn't tracked the last versions of SREP since 2.00 version. A lot of work have been done in later versions, so thanks to expanded weekend, I hope to play with it these days.
Thanks Bulat!

[m^2]

How does it behave when the computer has 512 MB RAM?

[Bulat Ziganshin]

How does it behave when the computer has 512 MB RAM?

it's enough to compress files up to 5-10 gb

[Black_Fox]

I believe he meant how does the "-mem75%-600mb" behave with 512 MB RAM

[m^2]

I believe he meant how does the "-mem75%-600mb" behave with 512 MB RAM

Yep.

[Bulat Ziganshin]

the same as -mem0. after calculation, the mem value is forced into 70mb...max(size_t) range

of course, if client computers may have 512mb ram, you shouldn't compress with lzma:512mb anyway

[m^2]

Trying to compile Tornado with Clang I got:

tornado/EntropyCoder.cpp:690:38: error: default arguments cannot be added to an
out-of-line definition of a member of a class template
TCounter<type> :: TCounter (unsigned _n = NUM)

[Bulat Ziganshin]

fill bugreport at Clang

[Matt Mahoney]

Visual C++ and Mars don't like this code either, even though g++ and Borland say it's OK.

template <class T>
struct A {
  int f(int x);
};

template <class T>
int A<T>::f(int x=0) {return x;}

int main() {
  A<char> a;
  return a.f();
}

[m^2]

Clang is correct. C++ [dcl.fct.default]p6 says:

Except for member functions of class templates, the default arguments in a
member function definition that appears outside of the class definition are
added to the set of default arguments provided by the member function
declaration in the class definition. Default arguments for a member function of
a class template shall be specified on the initial declaration of the member
function within the class template.

[jimbus80]

I have FreeArc installed and I have downloaded Srep 0.296. If I want to compress with freearc using new Srep 0.296, where should I place "srep.exe" file or how should I do it?

I'll try to explain myself. For example, I've seen above the command "arc a archive -m=srep:mem75%-600mb:f+lzma:512mb", I have tried it (typing at freearc\bin folder) and it works, but I am pretty sure my freearc is not using Srep 0.296, but an older version. How should I do it? And for descompression?

Thanks.

[Bulat Ziganshin]

executable files are looked in the PATH. just find where old srep.exe placed and replace it with the new one

btw, previous srep versions doesn't support -mem75%-600mb option, so either new srep was called in your experiment or options were not passed at all. i suggest you to start with installing latest freearc (march 18) and adding/replacing [External compressor:srep] section in arc.ini with recommended one

[Bulat Ziganshin]

TCounter<type> :: TCounter (unsigned _n = NUM)

thank you foir piinting this out. i've moved default to declaration site, new sources are pushed to svn

[jimbus80]

I have not found any srep.exe in my computer.

Now I've copied all the new srep*.exe files (0.296) to "C:\Program Files (x86)\FreeArc\bin" and replaced [External compressor:srep] section in arc.ini with recommended one. I hope the new FreeArc 0.67 uses them with the command "arc a archive -m=srep:mem75%-600mb:f+lzma:512mb".

I would like to get the best compression. I hope that's the right way to get it.

[m^2]

thank you foir piinting this out. i've moved default to declaration site, new sources are pushed to svn

Thanks, I'll try it some time later.

[jimbus80]

When I use the command "arc a archive -m=srep:mem75%-600mb:f+lzma:512mb" with FreeArc 0.67 and Srep 0.296 I always get an error:

- If arc.ini is the original one, the error is "Invalid option: -mem75f"
- If arc.ini is the recommended one, the error is "Arc.exe has stopped working. Close the program" or "write error (disk full?) in compression algorithm srep:mem75f"

[Skymmer]

When I use the command "arc a archive -m=srep:mem75%-600mb:f+lzma:512mb" with FreeArc 0.67 and Srep 0.296 I always get an error:

You need to use the double percent symbol. I mean:
-m=srep:mem75%%-600mb:f+lzma:512mb
Hope it will help.

By the way. Does anybody tried the stdin\stdout mode of SREP 2.96 ? It works incorrectly for me. For example.
Just tried to compress the enwik9 with

-ep1 -lc- -di+$ -ma- -r -s; -mt2 -msrep296p+lzma:a1:mfbt4:d32m:lc3:fb64

The result is:

FreeArc 0.67 (March 18 2011) creating archive: D:\Test_enwik9_pipe.arc
Started: 0.00 secs
Compressing 1 file of 1,000,000,000 bytes: 0.03 secs
  Using srep296p+lzma:32mb:normal:bt4:64
  Memory for compression 369mb, decompression 32mb
Compressing 0 bytes with srep296
  0.0%1457 mb, -m3 -l512 -c256 -a4 -nommap
Compression ratio: 496683520 -> 493647254: 99.39%. Cpu 41.607 mb/sec, real 2.046  99.9%

Errorlevel=0

  Solid block compression results
    srep296p: 493,647,255 bytes in 248.224 seconds
    lzma:32mb:normal:bt4:64: 108,591,782 bytes in -1.000 seconds

  Writing directory: 248.34 secs
  Found 1 directory names: 248.34 secs
  Directory written: 248.3
Compressed 1 file, 1,000,000,000 => 108,591,782 bytes. Ratio 10.8%
Compression time: cpu 360.06 secs, real 248.34 secs. Speed 4,027 kB/s
All OK

Well, nice to see that durilca kingsize have been beaten but seriously...

Compression ratio: 496683520 -> 493647254: 99.39%. Cpu 41.607 mb/sec, real 2.046  99.9%

Thats obviously wrong. It happens on both x86\x64 OSes and different compiles. Can anybody confirm?

[Biozynotiker]

Can anybody confirm?

I'm using srep 2.96 (alpha) stdin/out with rar 4.01, no error at all. (only srep -m3 switch testet).

[Bulat Ziganshin]

Biozynotiker, you probably use only stdin or stdout od SREP a time, while inside freearc it used as a filter

jimbus80, you probably use it in cmd file, in which case you should double the % sign

Skymmer, it was reported by egor23 and discussed in russian forum. i don't even know whether it's a fa or srep bug

[jimbus80]

No matter whether I use a cmd file with double %% or a single one using the command line, when I use "-m=srep:mem75%%-600mb:f+lzma:512mb" I always get an error and it closes. Most of the times, I do not get an error if I use -m=srep+lzma:256mb. My computer has Windows 7 64-bit and 4 GB RAM.

[Skymmer]

Just performed a couple of trials with piping and got interesting results. The command line is:

lzma d enwik8.lzma -so | srep - - | lzma e enwik8_new2.lzma -a0 -d27 -fb32 -si -eos

srep is 2.96 and lzma is actually the modified x64 version from Addons dir. The results is:

-m3 49668352 -> 49634662
-m2 49668352 -> 49655964
-m1 100000000 -> 99924456

So. -m1 processing the whole file but the resulting file seems to be corrupted, because LZMA reports

LZMA-FreeArc-x64 9.12 beta : Igor Pavlov : Public domain : 2010-03-24
*** VERSION MODIFIED FOR FREEARC, NOT COMPATIBLE WITH ORIGINAL ***
Decoder error

And the output file is only 67 108 875 bytes.

[Bulat Ziganshin]

please try with -nommap

[Skymmer]

Still no luck with -nommap option. Compression goes fine, but "decoder error" on decompression.
And by the way, the -nommap option was always automatically implied in all tests with arc with srep 2.96 plugged via arc.ini in stdin\out mode.

[jimbus80]

If I compress several files using the command "-m=srep:mem75%%-600mb:f+lzma:512mb", what command should I use to decompress them? I get errors when I use "arc.exe x -m=srep:mem75%%-600mb:f+lzma:512mb", "arc.exe x -m=srep", etc.

[m^2]

Just "arc -x archive.arc"

[jimbus80]

Of course I tried with arc x archive.arc (it is the most simple command) and it did not work either. In fact, it did not finish the job: I get a 0 bit file and the decompression task does not finish; it remains working endlessly.

[Bulat Ziganshin]

arc x archive.arc should do it, with help of srep.exe. can you provide program output?

[jimbus80]

More tries always using a ".cmd" or ".bat" file:

1. I compress a file with the command "arc.exe a result_file -m=srep+lzma:256mb source_file".

2. I compress a file with the command "arc.exe a result_file -m=srep source_file".

In both cases (but a little more in the first one), the compression is amazing: the result file is much smaller than using only 7-zip or FreeArc.

The problem comes when I try to decompress those files with the command "arc.exe x result_file.arc" inside a ".bat" or ".cmd" file. Then, in any case (file compressed with the command -m=srep+lzma:256mb or -m=srep), FreeArc says, after a while, "Processed 100.0%", but:

1. The ".bat/.cmd" does not continue to the next line (= does not finish): it remains stuck there, showing "Processed 100.0%".

2. The decompressed result file KEEPS GROWING its size endlessly (if its original size was 100 MB, it keeps growing: 108, 116, 124...), until I close the window manually (the oversized file keeps saved into the hard drive, it does not disappear).

3. Arc.exe and srep.exe keep loaded in memory until I close the cmd window manually.

I am using Srep 0.296 (tried 32 and 64 bit files), FreeArc 0.67 (with the recommended Srep for arc.ini modification) under Windows 7 64-bit.

I don't know what else I can provide, but I'll be glad to do it if you tell me exactly.

Regards.

Edit: Bulat, I've made some pictures to clarify what I am saying. I've uploaded to http://www.mediafire.com/?kz6bxkuaa213j91

[Bulat Ziganshin]

srep 2.9x has bug in stdin-to-stdout compression mode. please try this arc.ini section:

[External compressor:srep]
packcmd = srep {options} $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = srep -d {options} - - <stdin> <stdout>

sorry - you became alpha tester

ps: compression ratio is amazing just because SREP ignores part of input data - read Skymmer's report above

[jimbus80]

Yes, you must be right, because now the results are worse using Srep than using only 7zip or FreeArc :-)