Zhuff - fast compression

[Cyan]

Zhuff, what's in this name ?
no, i know it sounds like a beer, but,
it's a new compression software, aiming at the "faster than your HDD" category. Zhuff is basically LZ4 and Huff0 merged together, both being very fast.

It ends up with something which is pretty close in term of compression ratio to tornado -2 (from Bulat), or thor e2 (from Oscar Garcia).
That typically means compressed result within 2% of gzip fastest mode but 3 to 4 times faster.

You can have a try at the first released version here :
http://phantasie.tonempire.net/pc-co...on-t99.htm#155

[Edit] a few benchmarks numbers (specifically targeting high-speed compressors) are graphically represented here :
http://phantasie.tonempire.net/pc-co...rk-t96.htm#149

Regards

[Lasse Reinhold]

How about adding a -bench flag like fastlz, quicklz, etc. has which times it in-memory?

Because on enwik8 I find that quicklz level 1 is like twice as fast as zhuff, when I compress to NUL.

[Cyan]

You right, Lasse.
"-bench" code is already existing from LZ4, it can be added to Zhuff. It will be present for next release

There is no competition with QuickLZ L1, which is, hands down, much much faster than Zhuff.
Zhuff can merely be an interesting competitor to QuickLZ L2, at best.

If you look for QuickLZ L1 competition, according to the graph, this should point towards LZ4 (on text) or LZP2 (on binaries).
Even then, QuickLZ L1 keeps an important memory advantage, which is extremely positive on older systems (with 256KB of L2 cache).
LZP2 and LZ4 are both very aggressive on L2 memory usage, which is no problem for modern Core2, but an issue for older and embedded systems.

The comparison was made using external "timer" program from Igor Pavlov.
As you know, this is not exactly equivalent to "in-memory" compression, due to kernel and I/O management considerations.
On extremely fast compression programs, such times are very significant (they are even predominant for LZ4 binaries decompression speed for instance).
So you will not find the same "speed ratio", but the ranking should remain more or less identical.

"-bench" in-memory test compression is already available for LZ4 & LZP2, should you wish to test & compare them.

Regards

[Fu Siyuan]

Seems you have achieved your goal in some perspect. It performs better than other fastest compressors. I hope you can keep on improving compression ratio , since I personally don't think it's a big difference between the speed of 100M/s and 50M/s.

[Matt Mahoney]

Zhuff made the Pareto frontier. http://mattmahoney.net/dc/text.html#3845

[Fu Siyuan]

Zhuff made the Pareto frontier. http://mattmahoney.net/dc/text.html#3845

Hi Matt, did you moved the server of LTCB? I can't get the website from everywhere since last week!

[Sportman]

Nice speed round lzturbo 0.95 -21

Input file: mixed text/html 468,911,946 bytes

archiver - comp. - decomp. - output size

lzbw1 -c - 2.119 sec - 1.966 sec - 144,713,203 bytes
6pack -1 - 2.275 sec - 1.680 sec - 139,072,287 bytes
tor64 -1 - 1.862 sec - 1.781 sec - 137,042,038 bytes
tor -1 - 1.998 sec - 1.681 sec - 137,042,038 bytes
lzp2 -c - 1.822 sec - 1.135 sec - 135,782,067 bytes
lz4 -c - 2.078 sec - 1.010 sec - 133,106,322 bytes
6pack -2 - 2.465 sec - 1.621 sec - 131,850,543 bytes
lzop -1 - 3.226 sec - 1.401 sec - 129,854,790 bytes
qpress64 -L1 - 1.088 sec - 0.864 sec - 128,863,821 bytes
qpress -L1 - 1.251 sec - 0.899 sec - 128,863,821 bytes
qpress64 -L2 - 1.766 sec - 0.970 sec - 119,646,911 bytes
qpress -L2 - 1.994 sec - 0.971 sec - 119,646,911 bytes
flzp c - 11.070 sec - 4.975 sec - 118,887,402 bytes
blz c - 8.458 sec - 4.515 sec - 118,218,756 bytes
blzpack c - 8.463 sec - 4.522 sec - 118,218,756 bytes
thor e1 - 2.197 sec - 3.591 sec - 116,910,904 bytes
tor64 -2 - 2.580 sec - 2.049 sec - 115,934,010 bytes
tor -2 - 2.371 sec - 2.147 sec - 115,934,010 bytes
lzturbo -11 -p1 - 1.877 sec - 0.857 sec - 115,810,797 bytes
qpress64 -L3 - 4.924 sec - 0.640 sec - 114,413,518 bytes
qpress -L3 - 5.950 sec - 0.654 sec - 114,413,518 bytes
slug (1.1b) c - 2.142 sec - 2.205 sec - 110,454,826 bytes
lzturbo -21 -p1 - 2.317 sec - 1.387 sec - 107,620,170 bytes
zhuff - 2.872 sec - 1.873 sec - 102,998,491 bytes
nz64 a -cf -m32m - 2.833 sec - 2.696 sec - 102,896,415 bytes
nz64 a -cf -m3072m - 2.925 sec - 2.860 sec - 102,703,037 bytes
thor e2 - 3.921 sec - 3.828 sec - 102,094,072 bytes
nz64 a -cF -m32m - 3.487 sec - 4.244 sec - 100,196,944 bytes
lzturbo -12 -p1 - 2.948 sec - 0.861 sec - 98,769,225 bytes
lzturbo -31 -p1 - 2.993 sec - 2.276 sec - 98,312,761 bytes
arc a -m1 -mt1 - 3.952 sec - 4.233 sec - 93,107,475 bytes
tor -3 - 3.819 sec - 2.702 sec - 93,107,246 bytes
tor64 -3 - 3.931 sec - 2.786 sec - 93,107,246 bytes
7z a -tzip -mx=1 - 14.304 sec - 4.226 sec - 90,896,342 bytes
lzturbo -22 -p1 - 3.310 sec - 1.343 sec - 90,303,788 bytes
lzturbo -14 -p1 - 11.776 sec - 0.722 sec - 89,093,062 bytes
rar a -m1 -mt1 - 11.033 sec - 5.417 sec - 89,041,066 bytes
tor -4 - 5.185 sec - 2.591 sec - 88,157,726 bytes
thor e3 - 4.837 sec - 4.715 sec - 88,082,692 bytes
lzturbo -13 -p1 - 5.870 sec - 0.847 sec - 87,460,457 bytes
packet -m1 -s0 - 32.145 sec - 6.832 sec - 85,858,554 bytes
7z a -tzip -mx=5 - 101.356 sec - 4.062 sec - 84,916,872 bytes
lzturbo -32 -p1 - 4.155 sec - 2.048 sec - 84,851,279 bytes
thor e5 - 13.074 sec - 3.898 sec - 84,323,324 bytes
nz64 a -cd -m32m - 12.279 sec - 4.347 sec - 83,083,855 bytes
bzp c - 13.124 sec - 15.883 sec - 81,797,783 bytes
slug c - 4.084 sec - 4.179 sec - 81,271,283 bytes
lzturbo -23 -p1 - 6.357 sec - 1.238 sec - 79,848,534 bytes
sx c - 3.871 sec - 4.064 sec - 78,081,595 bytes
tor -5 - 9.115 sec - 3.230 sec - 76,810,271 bytes
thor e4 - 9.924 sec - 4.683 sec - 76,796,244 bytes
rings c 1 - 35.431 sec - 24.846 sec - 76,430,445 bytes
nz64 a -cF -m3072m - 3.475 sec - 3.943 sec - 73,929,221 bytes
packet -m6 -s1 - 45.789 sec - 3.591 sec - 73,882,589 bytes
nz64 a -cD -m32m - 21.091 sec - 4.620 sec - 72,949,069 bytes
nz64 a -cd -m1024m - 14.509 sec - 4.307 sec - 71,599,109 bytes
lzturbo -53 - 13.798 sec - 6.303 sec - 70,929,995 bytes
sr3a c - 24.801 sec - 27.903 sec - 70,056,938 bytes
tor-small -6 - 12.270 sec - 2.884 sec - 68,514,998 bytes
rar a -m3 -mt1 - 59.772 sec - 4.503 sec - 67,871,142 bytes
bliz c - 94.938 sec - 92.824 sec - 65,985,193 bytes
tor64 -6 - 12.202 sec - 2.670 sec - 61,983,701 bytes
tor -6 - 13.477 sec - 2.760 sec - 61,983,701 bytes
tor-small -7 - 16.777 sec - 2.629 sec - 60,773,103 bytes
arc a -m2 -mt1 - 12.862 sec - 4.348 sec - 59,710,118 bytes
nz64 a -cD -m1024m - 35.419 sec - 4.396 sec - 58,472,128 bytes
bcm64 -b64 - 172.990 sec - 110.962 sec - 53,167,803 bytes
bcm -b64 - 183.035 sec - 120.493 sec - 53,167,803 bytes
arc a -m3 -mt1 - 34.962 sec - 7.341 sec - 52,971,667 bytes
ccm 0 - 129.776 sec - 144.121 sec - 52,625,854 bytes
flashzip a -m0 -c7 -b8 - 13.099 sec - 12.333 sec - 50,367,657 bytes
tor -7 - 29.681 sec - 2.344 sec - 48,704,015 bytes
arc a -m4 -mt1 - 84.605 sec - 6.987 sec - 46,999,974 bytes
rzm c - 436.699 sec - 13.647 sec - 44,687,747 bytes
tor -8 - 43.564 sec - 2.311 sec - 44,679,646 bytes
arc a -m5 -mt1 - 258.452 sec - 6.552 sec - 43,841,053 bytes
tor -9 - 71.377 sec - 2.265 sec - 43,778,860 bytes
arc a -m6 -mt1 - 256.719 sec - 6.113 sec - 39,287,193 bytes
flashzip a -m1 -c7 -b8 - 120.831 sec - 11.193 sec - 39,048,321 bytes
arc a -m7 -mt1 - 266.536 sec - 5.955 sec - 36,986,583 bytes

[Lasse Reinhold]

You right, Lasse.
If you look for QuickLZ L1 competition, according to the graph, this should point towards LZ4 (on text) or LZP2 (on binaries).
Regards

Oh yeah, sorry, I somehow managed to get the graph wrong

[Matt Mahoney]

Hi Matt, did you moved the server of LTCB? I can't get the website from everywhere since last week!

Is the Great Firewall blocking my site? I am attaching a copy of http://mattmahoney.net/dc/text.html if that link doesn't work. But the site is working.

[Fu Siyuan]

Is the Great Firewall blocking my site? I am attaching a copy of http://mattmahoney.net/dc/text.html if that link doesn't work. But the site is working.

Thanks!

A small suggestion : I've noticed you updated the 7zip 9.04. I think the core of it, LZMA, is the flagship implementation of asymmtric algorithms. But you made very little concern on it. The same as WinRAR. Their top benchmarks are all PPM derivations. So IMHO, it's better to emphasize the original algorithms, or if I made a text detector and added a PPM in my program, should I got a top bench too?

[Bulat Ziganshin]

ppm provides better compression on text files. otoh, i agree - there are lot of programs incorporating ppmd, are we need to list them all? lzma results is what really interesting, and probably 7zip's optimized bzip2/deflate too

[Cyan]

Thanks for testing Matt & Sportman.

Minor update, to signal the availability of in-memory benchmark option present in version v0.2 :
http://phantasie.tonempire.net/pc-co...on-t99.htm#155

@Fu : no worry, i'm not done with compression yet I agree there is still some improvement possible before reaching HDD speed...

[Sportman]

This time a bigger test file with less text :-)

Input file: ubuntu virtual disk 2,029,003,264 bytes

archiver - comp. - decomp. - output size

tor64 -1 - 9.491 sec - 8.590 sec - 965,586,761 bytes
lzbw1 -c - 10.716 sec - 12.178 sec - 954,479,930 bytes
6pack -1 - 12.684 sec - 8.632 sec - 942,320,358 bytes
6pack -2 - 12.533 sec - 7.840 sec - 912,522,836 bytes
qpress64 -L1 - 5.274 sec - 4.320 sec - 890,275,943 bytes
lz4 -c - 11.144 sec - 4.050 sec - 887,936,589 bytes
lzop -1 - 20.050 sec - 6.402 sec - 853,574,263 bytes
lzp2 -c - 9.527 sec - 6.033 sec - 849,523,354 bytes
qpress64 -L2 - 10.280 sec - 4.811 sec - 840,563,603 bytes
blz c - 39.656 sec - 22.026 sec - 833,037,803 bytes
thor e1 - 11.537 sec - 22.117 sec - 832,547,018 bytes
lzturbo -11 -p1 - 9.291 sec - 5.591 sec - 832,114,668 bytes
tor64 -2 - 11.693 sec - 9.761 sec - 820,447,766 bytes
qpress64 -L3 - 26.462 sec - 3.183 sec - 805,842,686 bytes
slug (1.1b) c - 11.866 sec - 14.340 sec - 803,510,813 bytes
lzturbo -21 -p1 - 11.720 sec - 7.585 sec - 792,745,170 bytes
thor e2 - 20.789 sec - 20.850 sec - 759,937,908 bytes
zhuff - 16.185 sec - 9.679 sec - 748,833,819 bytes
nz64 a -cF -m32m - 23.127 sec - 39.795 sec - 728,523,980 bytes
lzturbo -31 -p1 - 16.227 sec - 13.923 sec - 724,280,811 bytes
lzturbo -12 -p1 - 14.619 sec - 4.985 sec - 722,233,616 bytes
nz64 a -cf -m32m - 15.845 sec - 17.005 sec - 706,137,898 bytes
nz64 a -cf -m3072m - 16.573 sec - 17.542 sec - 706,137,795 bytes
thor e3 - 26.431 sec - 25.644 sec - 693,032,348 bytes
lzturbo -41 -p1 - 26.315 sec - 27.565 sec - 687,823,404 bytes
lzturbo -22 -p1 - 17.640 sec - 7.126 sec - 684,230,503 bytes
lzturbo -13 -p1 - 38.762 sec - 5.523 sec - 681,280,664 bytes
7z a -tzip -mx=1 - 79.998 sec - 25.762 sec - 678,499,274 bytes
arc a -m1 -mt1 - 22.935 sec - 24.072 sec - 666,658,727 bytes
tor64 -3 - 23.447 sec - 16.523 sec - 666,658,494 bytes
rar a -m1 -mt1 - 61.845 sec - 28.385 sec - 645,558,095 bytes
thor e5 - 85.341 sec - 21.678 sec - 643,203,576 bytes
lzturbo -32 -p1 - 23.558 sec - 12.378 sec - 643,089,642 bytes
slug c - 23.621 sec - 25.887 sec - 640,741,260 bytes
lzturbo -23 -p1 - 43.344 sec - 7.415 sec - 640,574,922 bytes
tor64 -4 - 26.070 sec - 15.662 sec - 629,857,814 bytes
thor e4 - 67.724 sec - 30.822 sec - 604,805,468 bytes
sx c - 25.115 sec - 30.160 sec - 604,530,208 bytes
lzturbo -33 -p1 - 58.925 sec - 11.614 sec - 603,979,127 bytes
nz64 a -cF -m3072m - 27.200 sec - 38.494 sec - 599,064,897 bytes
tor64 -5 - 50.572 sec - 20.866 sec - 585,445,615 bytes
nz64 a -cd -m32m - 41.538 sec - 15.708 sec - 559,550,909 bytes
nz64 a -cd -m1024m - 68.741 sec - 15.805 sec - 541,589,726 bytes
tor64 -6 - 89.563 sec - 19.754 sec - 540,676,151 bytes
nz64 a -cD -m32m - 81.674 sec - 20.432 sec - 521,571,789 bytes
flashzip a -m0 -c7 -b8 - 119.687 sec - 101.685 sec - 521,560,038 bytes
tor64 -7 - 201.126 sec - 18.612 sec - 517,654,911 bytes
nz64 a -cD -m1024m - 143.278 sec - 19.799 sec - 499,672,886 bytes

[Cyan]

Since it was requested by a few users, here is a new version of Zhuff, featuring :

- All compression modes into a single binary
- Multi-threading support
- Pipe mode support
- Context Menu integration
- Directory compression, with shar from Shelwien

It can be downloaded here.

[Matt Mahoney]

Nice improvement in compression. http://mattmahoney.net/dc/text.html#3099

[Cyan]

Thanks for testings Matt.

Btw, it seems LTCB speed pareto frontier (underlined figures) directly compares algorithm's speed measured on vastly different hardware specs.
Would that sound sensible to compare only speeds that are measured (and verified) on identical hardware, while other speeds could still be provided "as information" ?

[Matt Mahoney]

Well, the speeds and Pareto frontier are just provided as "information" anyway because I rank by size only. I actually report the speed of compression "systems" that include both the hardware and software. I want to allow programs that might need to run on a GPU, distributed network, supercomputer, custom chips, quantum computer, or whatever, because maybe that's what it takes for a top ranking. I want to allow all possible methods, not just those that will run on my laptop.

So if you have a fast computer, you might want to benchmark it yourself and send me the results.