That awesome slug is back!

[LovePimple]

Slug v1.26b has been released.

I uploaded Slug 1.26b. It comes with small improvements in both, speed and ratio. So, let the hard-disks glow.

Download


Thanks Chris!

Mirror: Download

[LovePimple]

Quick test...

A10.jpg > 845,286
AcroRd32.exe > 1,894,351
english.dic > 1,072,010
FlashMX.pdf > 3,894,215
FP.LOG > 1,082,120
MSO97.DLL > 2,304,324
ohs.doc > 902,533
rafale.bmp > 1,272,522
vcfiu.hlp > 824,752
world95.txt > 808,176

Total = 14,900,289 bytes


ENWIK8 > 35,106,390 bytes

Compression Time: 8.314 Seconds

Decompression Time: 8.311 Seconds

[Christian]

Thanks LovePimple!

Now, I'll wait for some benchmarks to show up.

[LovePimple]

Slug 1.26c compression speed is slightly slower than 1.26b on my AMD Sempron 2400+ machine.

[Christian]

Thanks for the info. Can you please check your PMs, LovePimple...

[LovePimple]

Here are the results you asked for. Could not send them by PM because the PM service complained about the message being 'too long'.

Edit: <link removed>

[joerg]

First Test on ATHLON-XP 512 MB RAM
---------------------------------------
compress a oracle dump-file (Text-File)
---------------------------------------
RINGS13 648331264 bytes to 26356464 bytes time= 75.22 s
-------------------------------------------------------------------------------
RINGS15C 5 648331264 bytes to 26580793 bytes time= 77.32 s.
-------------------------------------------------------------------------------
THOR096A 648331264 bytes to 79322888 bytes time= 51.544s SPEED: 12.00 MB/sec
-------------------------------------------------------------------------------
Slug 1.26b 633136.00 kb -> 50514.93 kb (7.98%) time= 36 s
Slug 1.26c 633136.00 kb -> 50514.93 kb (7.98%) time= 68 s
-------------------------------------------------------------------------------
should means
slug 1.26b ist the fastest compressor on this system
is a lot faster then 1.26c
bets allways thor and the other compressors
(in other tests the same tendency on this system)

slug version 1.26c has no fortune on this system

RINGS15C needs more then twice the time of slug 1.26b
but has a really better compression too

[nimdamsk]

Where can i get Slug 1.26b to test?

[joerg]

try the website from christian

http://christian.martelock.googlepages.com/index.htm

sorry now i see version 1.26b is no more there

[Christian]

Thanks joerg. I really do appreciate your effort, but sadly your test does not help me much. Speed test have to be done very thoroughly - multiple runs, maybe even reboots between the different runs - process times are interesting, too...

[Christian]

Where can i get Slug 1.26b to test?

It's still there - just take the link from 1.26c and replace the c with a b.

First Test on ATHLON-XP 512 MB RAM

What Athlon-XP do you have and which OS are you using? The great difference between 'b' and 'c' is really strange. Is the test hard disk very fragmented? Did you run the tests more than one time?

[Christian]

If someone does have some time, I'd be interested in benchmark numbers for these versions:

[removed]

The data-set should be at least 1GB larger than your RAM-size and NOT to nul but to a real file. Additionally, it'd be useful to do at least two runs. Wall times, process times (e.g. using timer from Igor) and your system setup (CPU,RAM,HDD,OS) would be great.

Thanks a lot!

EDIT: Without feedback I can only optimize for my system - and that's what I did. There'll be 1.27 tomorrow, maybe the final 1.2+ - anyway, I hope it'll work quite well for everyone.

[LovePimple]

That's stopped the benchmarkers dead in their tracks!

[Black_Fox]

I tried compressing 3,5GB tar archive from an external disk to internal one, but it seems USB is too slow... if results can be trusted, 1.26 and 1.26b were the fastest (wall time 334 s), 1.26c the slowest (377 s) and 1.27 somewhere inbetween (353 s). Will repeat tests later on internal disk only...
CPU: vanilla Athlon 64 X2 3800+, L1: 64+64 kB/core, L2: 512kB/core

[LovePimple]

There'll be 1.27 tomorrow, maybe the final 1.2+ - anyway, I hope it'll work quite well for everyone.

Thanks Chris!

Mirror: Download

[Christian]

Will repeat tests later on internal disk only...

Thanks BF - I'm looking forward to the results. *fingers crossed* But no matter what, 1.27 is the last version for the time being.

[Black_Fox]

Tests for compression within one disk are finished. Times were very consistent this time (wall times were +- 2 seconds):

SLUG 1.26
Kernel Time  =    10.984 = 00:00:10.984 =   3%
User Time    =   202.093 = 00:03:22.093 =  62%
Process Time =   213.078 = 00:03:33.078 =  66%
Global Time  =   322.016 = 00:05:22.016 = 100%

SLUG 1.26b
Kernel Time  =    12.062 = 00:00:12.062 =   3%
User Time    =   204.312 = 00:03:24.312 =  62%
Process Time =   216.375 = 00:03:36.375 =  66%
Global Time  =   327.484 = 00:05:27.484 = 100%

SLUG 1.26c
Kernel Time  =    14.140 = 00:00:14.140 =   3%
User Time    =   209.421 = 00:03:29.421 =  57%
Process Time =   223.562 = 00:03:43.562 =  61%
Global Time  =   365.781 = 00:06:05.781 = 100%

SLUG 1.27
Kernel Time  =    15.812 = 00:00:15.812 =   3%
User Time    =   211.921 = 00:03:31.921 =  53%
Process Time =   227.734 = 00:03:47.734 =  57%
Global Time  =   396.391 = 00:06:36.391 = 100%

[Christian]

Thanks a lot for testing, BlackFox! And many thanks to Matt for testing Slug and RZM!

But no matter what, 1.27 is the last version for the time being.

Surprise, there's now Slug 1.27b and it's not a bugfix.
It comes with a small compression improvement. Good news is, the ratio improvement is for free - at least on my system and over at Metacompressor's live-benchmarking.

For now I won't tweak IO anymore - it's boring and without enough feedback (only 4 people reported proper feedback) it's just like tapping in the dark. Btw., I tried IO with a reader- and a writer-thread, but no luck. Results did not improve at all on my system.

[LovePimple]

Surprise, there's now Slug 1.27b and it's not a bugfix.

Thanks Chris!

Mirror: Download

[m^2]

BUG:
Mistake in a command line:

slug f ".\projects.tar" ".\a.b"

Slug tries to create infinite archive.

[Christian]

There's a new experimental version of slug available - actually, it's a rewrite because I plan to open the source in the future. Some things have changed:

-a tiny bit slower on my C2D, maybe ~10%
-no extra handling of already compressed data
-better compression on most files

If you have some time spare, feel free to check for any errors (except wrong commandline usage ).

[m^2]

There's a new experimental version of slug available - actually, it's a rewrite because I plan to open the source in the future. Some things have changed:

-a tiny bit slower on my C2D, maybe ~10%
-no extra handling of already compressed data
-better compression on most files

If you have some time spare, feel free to check for any errors (except wrong commandline usage ).

That's a news!
Thanks a lot, I'm on it.

[m^2]

Quick results:

TCUP:
slug	  120218388 13.235s
slug X 116258883 ~19s

Bookstar:
slug   14320218 1.468s
slug X 13881376 ~1.6s

[Christian]

Quick results:

TCUP:
slug	  120218388 13.235s
slug X 116258883 ~19s

Bookstar:
slug   14320218 1.468s
slug X 13881376 ~1.6s

Thanks m^2! Interesting results. Probably, I'll have to add handling for hardly compressible data parts at a later time. Damn it, this will bloat the code by another couple of lines.

Is the file TCUP available somewhere?

[m^2]

Thanks m^2! Interesting results. Probably, I'll have to add handling for hardly compressible data parts at a later time. Damn it, this will bloat the code by another couple of lines.

Is the file TCUP available somewhere?

Normally no.
Really, I haven't decided whether I do better by testing by publicly available data or private one - because of the rule to tweak on one data and verify on another.
But you can get the installer from www.tcup.eu. I use version 4.1.

[Christian]

Small update: I reintroduced compressed data handling. There are no other changes.

Slug should ratio-wise be on par with "thor e4" while being a lot faster.

[Simon Berger]

How is the compressed data handling working? Lets say we have a tar file with a mix of files compressed with your ccm and very good compressable text files.

[m^2]

TCUP:
115933595 ~15
Bookstar:
13881376 ~1.6
And one of my not yet ready test sets, XSOS, virtual machine image, 442515968 B:
slug 69986283 10.141
thor e4 67513532 38.312
slug X 67232526 ~12

[Christian]

How is the compressed data handling working? Lets say we have a tar file with a mix of files compressed with your ccm and very good compressable text files.

Well, the algo detects hardly compressible parts in the tar file on the fly. When such a part is detected, a faster compression mode is selected. While in control and processing the data, the faster mode decides at which point it makes sense to switch back.

The faster mode is just huffman coding plus some rolz book keeping.

[Simon Berger]

Well, the algo detects hardly compressible parts in the tar file on the fly. When such a part is detected, a faster compression mode is selected. While in control and processing the data, the faster mode decides at which point it makes sense to switch back.

Thanks. Yes the switching to faster compression mode isn't this hard (except a good system to mark these points) but I wondered about if it switch back again.