Some comments on Tornado

[m^2]

I'll start with a bug report.

-l# -- length of hash row (1..9999), default 4

With 9999 Tornado crashes.
It seems that this value has to be a power of 2.

Also, some modes could be tweaked a bit.
From my tests:
-c2 provides much better compression than -c1 with very low speed penalty. I suggest changing this in -1 mode.
-c4 provides better efficiency than c3 from mode -3 up. I think that at least -4 should get this.
-12 -s2 -l256 is always much faster and usually better than the default -s1 -l512.

[Bulat Ziganshin]

>With 9999 Tornado crashes.

thanks, will fix sometime

-1 mode is optimized for speed. btw, note that speed difference may differ between cpus and data being compressed

>-c4

that you mean by eficiency? afair, on texts -c4 doesn't provide any substantial compression improvementr oover -c3 while makes program several percents slower even in -5

>-12 -s2 -l256

these settings are used in -11 mode -s2 is better than -s1 and i don't remember why -s1 used in -12 mode. probably i just forget to change this setting

[m^2]

>With 9999 Tornado crashes.

thanks, will fix sometime

-1 mode is optimized for speed. btw, note that speed difference may differ between cpus and data being compressed

I use 1 CPU, but 6 different test sets. Also text and compressed ones. The biggest measured speed difference was 6.4%. The smallest size gain - 10.0%. This was HDD->HDD, but my HDD is fast and CPU rather slow so usually the speed difference will be even lower.

>-c4

that you mean by eficiency? afair, on texts -c4 doesn't provide any substantial compression improvementr oover -c3 while makes program several percents slower even in -5

I know 2 efficiency measures, one is maximumcompression.com and the other I developed myself, you can see it in my testing reports.
In all tests with both measures c4 scored better in -3 and -4.

>-12 -s2 -l256

these settings are used in -11 mode -s2 is better than -s1 and i don't remember why -s1 used in -12 mode. probably i just forget to change this setting

OK, let me suggest also updating Tornado in FreeArc. You don't use s2 there at all.

[Bulat Ziganshin]

1. well, i measured pure cpu time, without i/o. i will consider your suggestion when time permits

2. may be. on my tests on old box (1ghz) it looked natural choice. again, i should recheck it again

3. i think that tor:9..tor:11 modes use -s2, it's just silent i can't support this parameter explicitly since it will break backward compatibility (it's drawback of FA archive format)

[m^2]

1. well, i measured pure cpu time, without i/o. i will consider your suggestion when time permits

2. may be. on my tests on old box (1ghz) it looked natural choice. again, i should recheck it again

Nice, thanks

3. i think that tor:9..tor:11 modes use -s2, it's just silent i can't support this parameter explicitly since it will break backward compatibility (it's drawback of FA archive format)

All I know is

// Preconfigured compression modes
PackMethod std_Tornado_method[] =
    //                 row hashsize  caching buffer parser  hash3 shift update
    { {  0, STORING,     0,       0, 0,      1*mb,  0     , 0 ,     0,  999 }
    , {  1, BYTECODER,   1,   16*kb, 0,      1*mb,  GREEDY, 0 ,     0,  999 }
    , {  2, HUFCODER,    1,   16*kb, 0,      2*mb,  GREEDY, 0 ,     0,  999 }
    , {  3, HUFCODER,    2,   64*kb, 0,      4*mb,  GREEDY, 0 ,     0,  999 }
    , {  4, HUFCODER,    2,    4*mb, 1,      8*mb,  GREEDY, 0 ,     0,  999 }
    , {  5, ARICODER,    4,    8*mb, 1,     16*mb,  LAZY  , 12,     0,  999 }
    , {  6, ARICODER,    8,   16*mb, 1,     32*mb,  LAZY  , 13,     0,  999 }
    , {  7, ARICODER,   16,   32*mb, 1,     64*mb,  LAZY  , 14,     0,    4 }
    , {  8, ARICODER,   32,   64*mb, 1,    128*mb,  LAZY  , 15,     0,    3 }
    , {  9, ARICODER,   64,  128*mb, 1,    256*mb,  LAZY  , 16,     0,    2 }
    , { 10, ARICODER,  128,  256*mb, 1,    512*mb,  LAZY  , 17,     0,    2 }
    , { 11, ARICODER,  256,  512*mb, 1,   1024*mb,  LAZY  , 18,     0,    1 }
    , { 12, ARICODER,  512, 1024*mb, 1,    768*mb,  LAZY  , 19,     0,    1 }
    };

[Bulat Ziganshin]

now it's

    //                        row hashsize  caching buffer parser  hash3 shift update
    { {  0, STORING,   false,   0,       0, 0,      1*mb,  0     ,   0,    0,  999 }
    , {  1, BYTECODER, false,   1,   16*kb, 0,      1*mb,  GREEDY,   0,    0,  999 }
    , {  2, HUFCODER,  false,   1,   16*kb, 0,      2*mb,  GREEDY,   0,    0,  999 }
    , {  3, HUFCODER,  false,   2,   64*kb, 0,      4*mb,  GREEDY,   0,    0,  999 }
    , {  4, HUFCODER,  true,    2,    4*mb, 1,      8*mb,  GREEDY,   0,    0,  999 }
    , {  5, ARICODER,  true,    4,    8*mb, 1,     16*mb,  LAZY  ,   1,    0,  999 }
    , {  6, ARICODER,  true,    8,   16*mb, 1,     32*mb,  LAZY  ,   1,    0,  999 }
    , {  7, ARICODER,  true,   16,   32*mb, 1,     64*mb,  LAZY  ,   1,    0,    4 }
    , {  8, ARICODER,  true,   32,   64*mb, 1,    128*mb,  LAZY  ,   1,    0,    3 }
    , {  9, ARICODER,  true,   64,  128*mb, 2,    256*mb,  LAZY  ,   2,    0,    2 }
    , { 10, ARICODER,  true,  128,  256*mb, 2,    512*mb,  LAZY  ,   2,    0,    2 }
    , { 11, ARICODER,  true,  256,  512*mb, 2,   1024*mb,  LAZY  ,   2,    0,    1 }
    , { 12, ARICODER,  true,  512, 1024*mb, 1,    768*mb,  LAZY  ,   1,    0,    1 }

[m^2]

One more thing about -L parameter:
It should be mentioned that maximum value depends on hash size and i.e. for 16KB (-1 mode) it's 4096.

Also, I found a comment in the sources about HT5 matchfinder...I feel that it would be much better than current -12. I'm aware that size would grow, but IMO it's worth it.

[Bulat Ziganshin]

1. where should i mention it? is the small program's help?
2. i actually implemented basic version of ht5..ht7 but it need more work to make them really useful. OTOH in lzma ht5..ht7 are really useful since ht4 provides arcghives ~2% larger than bt4 (in *max* mode)

[m^2]

1. where should i mention it? is the small program's help?

"1..hashsize/4" doesn't take much space.

2. i actually implemented basic version of ht5..ht7 but it need more work to make them really useful. OTOH in lzma ht5..ht7 are really useful since ht4 provides arcghives ~2% larger than bt4 (in *max* mode)

Very nice, thanks.

[Bulat Ziganshin]

1. but harder to understand. his is not a real problem, overall (imho)

[m^2]

Feature suggestion: automatic memory usage reduction. I just found that tor -12 was compressing a 1.5 GB file for ~30-35 hours. Didn't finish. For comparison, -10 needed 7 minutes.

[Bulat Ziganshin]

may be. i consider tornado.exe as demo program, it lacks a lot of features for real compressor (data integrity checking, header)

[m^2]

may be. i consider tornado.exe as demo program, it lacks a lot of features for real compressor (data integrity checking, header)

Yes, but this can really block other things...

[Bulat Ziganshin]

but why you use compression modes which isn't supported by your computer??

[m^2]

but why you use compression modes which isn't supported by your computer??

1. No documentation mentions system requirements.
2. It always worked. Even in a case when file size + hash size slightly exceeded my memory size (when you substract the part used by OS - exceeded significantly), I didn't notice slowdown bigger than expected ~4x compared to -10. This suggested that Tor limits itself.