FreeArc

[thometal]

Is the 1GB dic option not exclusive for a 64-bit version ?
I couldn't find the mem. requirements for LZMA with ht4 matchfinder in the 7z docs.. do you mean the HC4 instead off HT4 ?
LZMA with 1GB dictionary would require about 1GB *7.5 = 7,5 GB RAM (HC4 match finder) according to the 7zip docs....

From Freearc hp news section:

May 15 2008: new FreeArc 0.50 alpha

lzma: changed default settings; :h parameter allows to change size of hash; new HT4 matchfinder allows to use dictionary up to 1gb

so far as I do renember HT4 uses 1.5*dictionary size and is a bit faster as bt4 but offers also a little worse compression. I think <5%.


damn to slow..... but thx for the command line Bulat

TT

[pat357]

>arc a a -m=lzma:1g -di -lc-

FreeArc 0.50 alpha (June 29 2008)
Creating archive: a.arc using lzma:1gb:normal:32
Memory for compression 1792mb, decompression 1024mb, cache 1mb
Compressing 2 files, 1.459.783.680 bytes. Processed 0.0%

Thx Bulat !

[pat357]

From Freearc hp news section:

May 15 2008: new FreeArc 0.50 alpha

lzma: changed default settings; :h parameter allows to change size of hash; new HT4 matchfinder allows to use dictionary up to 1gb

so far as I do renember HT4 uses 1.5*dictionary size and is a bit faster as bt4 but offers also a little worse compression. I think <5%.

Thanks for the info !!
It seems that this LZMA options are only available in Freearc ; 7zip LZMA doesn't know the h parameter for the size of the hashtable
or the "new" HT4 matchfinder. (tested 7z 4.58 and latest v4.59a4)

[Bulat Ziganshin]

of course. i'm still using lzma 4.43 with my little updates (abovementioned+improved memory allocation that avoids memalloc problems with 128mb dict (with bt4) on 2gb boxes and even allows to use 256mb dictionary with bt4 on 4gb ones)

[thometal]

@pat our knowledge is right =P)

Bug:

If I cancel a decompressions process Freearc exits completely after pressing on Yes, with an GTK error: "gtk2hs_closure_marshal: uncaught_exception".

My Sys:
Win xp home 32bit
2gb ram
Core Duo T2400

Lastest Freearc.

Is this the right place to summit bugs?

TT

[pat357]

of course. i'm still using lzma 4.43 with my little updates (abovementioned+improved memory allocation that avoids memalloc problems with 128mb dict (with bt4) on 2gb boxes and even allows to use 256mb dictionary with bt4 on 4gb ones)

Nice to see you are making improvements to LZMA !
Does the d=256m:mf=bt4 on 4GB boxes also require a 64 bit OS ? I think it can never work on 32-bit systems... Am I correct ?
On my Vista Prem. box / 4gb RAM

arc a -mlzma:bt4:256m -di+$ -lc- test dat.tar
FreeArc 0.50 alpha (June 23 2008) creating archive: test.arc
Started: 0.00 secs
Compressing 1 file of 379.715.584 bytes: 0.01 secs
  Using lzma:256mb:normal:bt4:32
  Memory for compression 2688mb, decompression 256mb
ERROR: Can't allocate memory required for (de)compression in lzma:256mb:normal:bt4:32

.
Hm, 2688mb needed to compress
Will there also be freearc (64bit) for Win64 available ?

[Bulat Ziganshin]

right, only 64-bit windows provides 32-bi programs full 4gb space

i can't make 64-bit version since my compiler doesn't support this (and part of compression methods too). so we are limited to 4gb for the next 12 months or so

[Bulat Ziganshin]

thometal: it's the right place, i know about this bug, plan to fix it in next alpha

[IsName]

As for CMM, Bulat says he need config options. The feature of using external compressors was implemented in v0.40, and its description is still missing in the English docs. So I put this documentation secton here.

config file arc.ini
...
Sections [External compressor]
These sections are used to describe the compression algorithms and preprocessing using external programs, such as PAQ and CCM. Examples of these sections:

[External compressor:precomp] 
mem = 2 
packcmd = precomp -o$$arcpackedfile$$.tmp $$arcdatafile$$.tmp 
unpackcmd = precomp -o$$arcdatafile$$.tmp -r $$arcpackedfile$$.tmp

[External compressor:jpg] 
mem = 10 
packcmd = packjpg $$arcdatafile$$.jpg 
unpackcmd = packjpg $$arcdatafile$$.pjg 
datafile = $$arcdatafile$$.jpg 
packedfile = $$arcdatafile$$.pjg

The title of the section after ':' sets new algorithm name. packcmd and unpackcmd describe the commands for packing and unpacking, respectively. packedfile and datafile - the names of packed and unpacked files, respectively, which are used by both commands; default is $$arcpackedfile$$.tmp and $$arcdatafile$$.tmp. cmem and dmem describe the amount of memory used in packing and unpacking (if they are the same, you can set both sizes using mem).

You can describe several versions / variations of one program, if they have the same call format:

[External compressor:ccm123,ccmx123,ccm125,ccmx125] 
mem = 276 
packcmd = {compressor} c 4 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp 
unpackcmd = {compressor} d $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

In this line {compressor} in the description replaced by the compression method name, that is equivalent to following description:

[External compressor:ccm123] 
mem = 276 
packcmd = ccm123 c 4 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp 
unpackcmd = ccm123 d $$arcpackedfile$$.tmp $$arcdatafile$$.tmp 
  
[External compressor:ccmx123] 
mem = 276 
packcmd = ccmx123 c 4 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp 
unpackcmd = ccmx123 d $$arcpackedfile$$.tmp $$arcdatafile$$.tmp 
…

When using compressors, whose packed data format varies from version to version (as the CCM), it's recommended to rename executables according to version number (ccm123.exe, ccmx125.exe…) and describe them in arc.ini in the same way as above. Then add in the section [Compression methods] alias on the latest version of:

[Compression methods] 
ccm = ccm125 
ccmx = ccmx125

Example of the above algorithms to determine a new method of compression in arc.ini:

[Compression methods] 
super = precomp+ccm/$jpg=jpg

Now use the option -m=super and voila!

In file arc.ini in the program package you can find other examples describing external compressors and using them to improve compression modes -m#p/-max.

[Bulat Ziganshin]

thank you. but please put it to the wiki http://haskell.org/haskellwiki/FreeArc

just add one more item to the list with page name FreeArc/xxx and then go to the new page and edit it

[Bulat Ziganshin]

i plan to finish 0.45 docs, autotranslate it to English as i did before and then put to some online editor like Zoho to allow everyone (with my permission) to edit it

if you know which online editor is better fot this purpose - please tell here

[ostkamp]

Hello Bulat,

I also sent this by email but never got a response, thus I am retrying here (it also is very hard to register to this forum, as the registration confirmation email was never received by two of my email accounts).

I tried out the latest FreeArc 0.50alpha with OpenSuSE 10.3 Linux.

I extracted the FreeArc tarball into a new directory /tmp/blubber and
after installation tried to pack this with FreeArc. Here is what happens:

$ arc a blubber.arc blubber
FreeArc 0.50 alpha (June 23 2008) creating archive: blubber.arc

There were 1 warning(s)
ERROR: Prelude.chr: bad argument

I am wondering what happens here?

IIRC you said 'arc' should be nearly command line compatible with RAR; and
with RAR this is a completely valid command which recursively packs the
directory.

Furthermore with the slightly different command for original RAR

$ rar a blubber.rar blubber/

(note the trailing slash) nothing would be archived (because the directory
contains only a subdirectory and no regular files) while

$ arc a blubber.arc blubber/

archives everything (so it does what is expected without the trailing
slash).

So it appears we see one bug and another incompatibility regarding command
line options.

Can this be fixed, please?

I also wonder how I can let 'arc' tell what it is doing like with RAR.
Currently it does not tell which files it is currently archiving, testing
etc. Is there a way to get the list that RAR provides?

Is there a Solaris version of FreeArc?

Thanks in advance & best regards

Guido

[Bulat Ziganshin]

i'm really sorry. i'm too lazy sometimes

1. fixed in my "internal" version: http://www.haskell.org/bz/arc1.arc . can you please test it too?

2. i don't want to fix this incompatibility because it doesn't have too much usage. this syntax for archiving whole dir used in Total Commander plugin and rather common among other archivers

3. -i2. have you read docs?

4. no. if you will try to build it yourself (read readme.txt in sources archive) i will be gald to help you. you don't need to install gtk2hs as far as we build console version

[IsName]

thank you. but please put it to the wiki http://haskell.org/haskellwiki/FreeArc

just add one more item to the list with page name FreeArc/xxx and then go to the new page and edit it

I didn't succeed in creating a new page.
Edit: seems it doesn't like [ and ] symbols in page name, even encoded.

[ostkamp]

Hi Bulat,

i'm really sorry. i'm too lazy sometimes

1. fixed in my "internal" version: http://www.haskell.org/bz/arc1.arc . can you please test it too?

2. i don't want to fix this incompatibility because it doesn't have too much usage. this syntax for archiving whole dir used in Total Commander plugin and rather common among other archivers

3. -i2. have you read docs?

4. no. if you will try to build it yourself (read readme.txt in sources archive) i will be gald to help you. you don't need to install gtk2hs as far as we build console version

1. unfortunately the fix does not work for me. Here is the result:

$ ls -l /usr/local/bin/arc
-rwxr-xr-x 1 root root 2610572 Jul 14 18:50 /usr/local/bin/arc
$ md5sum /usr/local/bin/arc
be147d9477641256a2f843f7a369d8a0 /usr/local/bin/arc
$ /usr/local/bin/arc a blubber.arc blubber
FreeArc 0.50 alpha (June 28 2008) creating archive: blubber.arc

There were 1 warning(s)
ERROR: Prelude.chr: bad argument

2. OK

3. Sorry, failed to read the docs.

4. I gave it a try today, but I had a very hard time with the GHC Haskell stuff. When I finally had installed all dependencies, a badly built ncurses library package prevented the execution of the Haskell compiler. After I solved this, the stdint.h internally needed was not around on the Solaris 9 Sparc box. I worked around that as well. When I came to start compilation of FreeArc, that bombed out because of the i486 related compiler options and after removing them, failed on the Environment.cpp stuff. Most of the data you are querying there is either not present or needs to be retrieved by different methods. Is there a way to work with hardcoded settings, that compile anywhere?

5. While checking for 1. I found another error with your private version. When calling 'arc' without parameters, the process goes into an endless loop after printing the help. Can you check?

Regards

Guido

[Bulat Ziganshin]

I didn't succeed in creating a new page.

you just ned to create link to unexisting page and when you follow it, you are prompted to edit it. can you please try again?

[Bulat Ziganshin]

Guido,

1. well, seems that it's another problem. i will look into it, of course, but not in a few days. i don't even have access to linux ATM

4. about ghc problems - you can try to ask in maillist or IRC_channel if you are ready to go deeper. about Environment - you can create new section here and hardcode your settings. f.e., define FREEARC_SOLARIS define and use it to distinguish your platform, put -DFREEARC_SOLARIS together with -DFREEARC_UNIX in your makefiles. then i will look into integrating your changeds into common codebase

5. see 1

[pat357]

I did some testing with Arc 30-06-2008 and it bumped out on a few occasions.

Here is what i got :

1. the dir "2bmp" contains 3 bmp files without .BMP extension.

Arc a -mx --> crash (got "Arc.exe stopped working.." message)
arc a -mx -di+$ -i2 2bmp_tst1003_m7b .\2bmp\*

FreeArc 0.50 alpha (June 30 2008) creating archive: 2bmp_tst1003_m7b.arc
Started: 0.00 secs
Found 3 files: 0.00 secs
Sorted 3 files: 0.00 secs
Joined filelists, 3 files: 0.00 secs
   1(71) ["default","default","default","default","default","default","default","default","default","default","default",
"default","default","default","default","default","default","default","default","default","default","default","default",
"default","default","default","default","default","default","default","default","default","default","default","default",
"default","default","default","default","default","default","default","default","default","default","default","default",
"default","default","default","default","default","default","default","default","default","default","default","default",
"default","default","default","default","default","default","default","default","default","default","default","default"]

   4(49) ["default","default","default","default","default","default","default","default","default","default","default",
"default","default","default","default","default","default","default","default","default","default","default","default",
"default","default","default","default","default","default","default","default","default","default","default","default",
"default","default","default","default","default","default","default","default","default","default","default","default",
"default","default"]
   5(43) ["default","default","default","default","default","default","default","default","default","default","default",
"default","default","$text","default","default","default","$text","$text","$text","default","$text","$text","default","d
efault","default","default","default","default","default","default","default","default","default","default","default","d
efault","default","default","default","default","default","default"]
Compressing 3 files of 121.642.496 bytes: 0.13 secs
  Using exe+rep:117mb+delta+lzma:117mb:max:bt4:128
  Memory for compression 1397mb, decompression
  Compressing ./2bmp/1
  Compressing ./2bmp/4                                                   64.8%

Arc a -m7b --> same crash

Arc a mrep:117mb+delta --> crash

arc a -mexe+rep:117mb+delta -di+$ -i2 2bmp_tst1005 .\2bmp\*
FreeArc 0.50 alpha (June 30 2008) creating archive: 2bmp_tst1005.arc
Started: 0.00 secs
Found 3 files: 0.00 secs
Sorted 3 files: 0.01 secs
Joined filelists, 3 files: 0.01 secs
Compressing 3 files of 121.642.496 bytes: 0.01 secs
  Using exe+rep:117mb+delta
  Memory for compression 157mb, decompression
  Compressing ./2bmp/1
  Compressing ./2bmp/4                                                   71.0%

So it seems rep en delta are causing the crash.

arc a -mdelta+rep:117mb -di+$ -i2 2bmp_tst1007 .\2bmp\* --> OK !!

FreeArc 0.50 alpha (June 30 2008) creating archive: 2bmp_tst1007.arc
Started: 0.00 secs
Found 3 files: 0.00 secs
Sorted 3 files: 0.01 secs
Joined filelists, 3 files: 0.01 secs
Compressing 3 files of 121.642.496 bytes: 0.01 secs
  Using delta+rep:117mb
  Memory for compression 157mb, decompression
  Compressing ./2bmp/1
  Compressing ./2bmp/4
  Compressing ./2bmp/5                                                   99.9%
  Solid block compression results (8.861 seconds)
    delta: 122.178.620 bytes in 7.457 seconds
    rep:117mb: 119.884.125 bytes in 1.404 seconds

  Writing directory: 10.49 secs
  Found 1 directory names: 10.49 secs
  Directory written: 10.4
Compressed 3 files, 121.642.496 => 119.884.125 bytes. Ratio 98.5%
Compression time 8.86 secs, speed 13.728 kB/s. Total 10.50 secs
All OK

arc a -mrep:117mb ---> OK
arc a -mdelta ---> OK
arc a -mrep:80mb+delta ---> OK
arc a -mdelta+rep:117mb ---> OK
arc a -mrep:117mb+delta ---> crash

It seems that arc only crashes if rep > certain value if followed by delta : no crash on rep:80mb+delta and crashes on rep:117mb+delta.
When switching delta and rep --> no crash.
These crashes do not occur when using the Jun 23 version.
Any idea ?

2. Another problem that I had using Arc 30-06-2008 :
arc a -mx -di+$ tst5 .\jpg\*

FreeArc 0.50 alpha (June 30 2008) creating archive: tst5.arc
Started: 0.00 secs
Found 31 files: 0.00 secs
Sorted 31 files: 0.01 secs
Joined filelists, 31 files: 0.01 secs
   A10.jpg(8) ["$compressed","$compressed","$compressed","$compressed","$compressed","$compressed","$compressed","$compr
essed"]
   DSCN0810.JPG(22) ["default","$compressed","$compressed","$compressed","$compressed","$compressed","$compressed","$com
pressed","$compressed","$compressed","$compressed","$compressed","$compressed","$compressed","$compressed","$compressed"
,"$compressed","$compressed","$compressed","$compressed","$compressed","$compressed"]
"default","$compressed","$compressed","$compressed","$compressed","$compressed","$compressed","$com
pressed","$compressed","$compressed","$compressed","$compressed","$compressed","$compressed","$compressed","$compressed"
,"$compressed","$compressed","$compressed","$compressed","$compressed","$compressed"]
 ........
   Earth.jpg(7) ["$compressed","$compressed","$compressed","$compressed","$compressed","$compressed","$compressed"]
Compressing 31 files of 164.849.454 bytes: 0.57 secs

  Using exe+rep:158mb+delta+lzma:158mb:max:bt4:128
  Memory for compression 1828mb, decompression  1.0%
ERROR: Can't allocate memory required for (de)compression in lzma:158mb:max:bt4:128

Arc tries to compress with a 158mb dict /bt4 mf and bumps out of memory, what doesn't surprise me on a 32 bit system...
This problem happens only with the 30-06 version, not with the 23-06 version.

3. The auto-file-type detection causes .JPG files to be identified as $compressed instead of $jpg or $jpgsolid. This makes that arc is compressing the .JPG using the wrong method.
Is there a way around this ? (besides -ma- of course)

I'm not sure whether these are "known" issues, but I thought it would be helpful input for developing the next freearc version.

System : C2Q E6600 / 4GB RAM / Vista Prem. (32 bit)

--edit --
for 1. :
arc a -mrep:80mb+delta ---> OK


--- edit--
I've tried to isolate the crash and here is what I found :

arc a -mdelta tst 1 ---> crash

The file 1 is a cutted part from the 2nd image in (1).
I've uploaded it here :
(file is 7 MB)

http://rapidshare.com/files/13092005...elta_crash.rar

Can someone try this on your system to test ?
You need arc 30-06-2008 to test ; arc 23-06-2008 works ok on this file.

[pat357]

As for CMM, Bulat says he need config options. The feature of using external compressors was implemented in v0.40, and its description is still missing in the English docs. So I put this documentation secton here.

[External compressor:jpg]
mem = 10
packcmd = packjpg $$arcdatafile$$.jpg
unpackcmd = packjpg $$arcdatafile$$.pjg
datafile = $$arcdatafile$$.jpg
packedfile = $$arcdatafile$$.pjg

Example of the above algorithms to determine a new method of compression in arc.ini:

[Compression methods]
super = precomp+ccm/$jpg=jpg

Now use the option -m=super and voila!

Unfortunately this doesn't work as expected because the default auto-file-type detection will classify all JPG files as type "$compressed" instead of "$jpg".
This way all .JPG files all processed in the same way as .RAR, .ZIP,... files.

Would it be possible to correctly detect JPG by the file-header ?
The same holds for .BMP files...
When I switch off auto-file-type detection (-ma- switch), it works as long as the files have the correct extension.

[Intrinsic]

-m5 mode on SFC(http://www.maximumcompression.com/da...ull_testset.7z) files produces smaller archive than -mx, i've attached the log files for you, for both -ma- and auto filetype determination.

Also, the compressed size output from cmd line and also in the GUI version reports the wrong file size, the input combined size is correct though.
In the -m5 log you'll see that it says the output file size is: 11,777,626 bytes, but when i view the properties of the file it shows: 11,778,183 bytes. I'd guess it's measuring the compressed size of the input data as opposed to the full file size with headers etc?

Hopefully these logs are of some use to you.

[John]

Wich are the best settings for a multimedia data? (game)

[Intrinsic]

Unfortunately this doesn't work as expected because the default auto-file-type detection will classify all JPG files as type "$compressed" instead of "$jpg".
This way all .JPG files all processed in the same way as .RAR, .ZIP,... files.

Would it be possible to correctly detect JPG by the file-header ?
The same holds for .BMP files...
When I switch off auto-file-type detection (-ma- switch), it works as long as the files have the correct extension.

Actually i found when using a custom method the file type detection seems to be ignored and stuff is just fed into precomp and then mashed with ccm. At least that's what i found when i setup some custom ones for testing earlier. In fact you can completely remove the above $jpg part of that custom method as it makes no difference. If you didn't have precomp then i'd guess it would be used when using -ma- but haven't tried so can't say for sure.

[Intrinsic]

Wich are the best settings for a multimedia data? (game)

If time isn't a factor, you may as well just use -max John. The auto file type recognition isn't the hottest thing atm though, so you can add -ma- to the command line to disable it and you can get potentially much better compression most of the time. Esp when there are lots of media files for which precomp etc can work with.


Oh and for Bulat:

How about this idea i had: As default use the groups file for filetype, and when you find a file that isn't in that groups file do your auto detection on it?

[thometal]

hi, i found some issues under Ubuntu 8.04 32bit in installationmode "make install":

If I try to compress around 16 gb's the arcdatafile become bigger and bigger, after a while FreeArc shuts down without errormsg and without to complete the compressionprozess. Is there an way to disable the usage of the arcdatafile, without loss of compressionsrate?

Also the history of nearly all dropdown menu's doesn't work.

The compressiondialog changes the size dynamicly by path and filename length, sometimes the windows is larger than screensize.

In the compression settingsdialog the text goes out of the window, maximize doesn't work.


THX

TT

[pat357]

Actually i found when using a custom method the file type detection seems to be ignored and stuff is just fed into precomp and then mashed with ccm. At least that's what i found when i setup some custom ones for testing earlier. In fact you can completely remove the above $jpg part of that custom method as it makes no difference.

When you check the arc.groups (from the distro), you'll notice that $jpg group is empty, it doesn't contain any extension. The .JPG files are in the group $jpgsolid.
This means that $jpg should be replaces by $jpgsolid (or change the arc.groups) to make it work.
For example :

precomp+ccm/$jpgsolid=jpg    (jpg's processed by packjpg)
precomp+ccm/$jpgsolid=jpgsolid (jpg's processed by precomp)

both work if you use them together with the -ma- switch.

[pat357]

Oh and for Bulat:
How about this idea i had: As default use the groups file for filetype, and when you find a file that isn't in that groups file do your auto detection on it?

Sounds like a great idea to me. This could be an excellent workaround until the auto-file-type detection is more reliable.

[John]

I compressed 10gb's of data, compressing works well, but at 35% of unpack/test i get a error:

Error reading or writing data in delta

I used the gui with the option -mx
Mi Pc have 2gb's of ram

Any clue?

[pat357]

I compressed 10gb's of data, compressing works well, but at 35% of unpack/test i get a error:
I used the gui with the option -mx
Mi Pc have 2gb's of ram
Any clue?

There are 2 settings for -mx : one using 192 mb to decompress and the other using 1gb. Which one did you use ?
If you used 1gb, try to pack again with the -mx 192mb method : 1gb RAM could be a problem on a 2gb RAM system.
Also close unneeded programs before unpacking/testing.

[John]

I just used

arc a -mx -mt -r test

You mind to do..?

arc a -mx:192mb -mt -r test

[pat357]

It seems that I can't browse in directories inside archives using the GUI.
I 've created an archive containing only one directory with files there.
When I try to browse it using the GUI, I only see the directory, but clicking on the directory doesn't show any files.
Is there something wrong with my setup ? Has anyone the same problem ?

Using Totalcmd + plugin works for browsing but not for unpacking a single file : I can only unpack all files/dirs.