19th January 2016, 16:29
My examples like lz4 & lz5
Sorry, I'm not a profi programmer and not an English speaker.
I've read about lz4 & lz5 by Yann Collet and downloaded his examples from Github (lz4.exe & lz5.exe). I added several improvements in lz77 algorithm and wrote Win32 example, which shows compression ratio on ~ 4% better than lz5 with simple hash.
My program uses simple hash and hash table on 1M cells like lz5.exe (as I saw it). I've compared my program with lz5 in simple hash mode on a 800Kb plain text file and a 8.3Mb exe file.
I want to know is anyone interested in this info?
Remark: my example stores compressed data in raw format.
Last edited by lz77; 19th January 2016 at 16:36.
The Following 3 Users Say Thank You to lz77 For This Useful Post:
Bulat Ziganshin (19th January 2016),Cyan (19th January 2016),Stephan Busch (19th January 2016)
19th January 2016, 16:45
Continue your interesting investigations, and at some point, you will find your results in good enough shape to be shared.
Always check your compressed stream can be decoded with appropriate source code.
There is always a risk that compressed format proved not decodable if done later on.
Also : lz5 is work from @inikep.
19th January 2016, 19:43
...and on side note, -dev branch of LZ5 comes with a little surprise: it comes with a bit better ratio, especially on highest levels, so it also happens to be moving target . One can get even better ratio on large chunks of data by increasing sliding window size (but it forces one to keep larger buffer in memory and does not works for small chunks of data). Or by tweaking levels for better match finding (but it quickly gets slow, while gain is quite small).
And IIRC, LZ5 is all about 1 << 22 window, which is 4Mb, not 1Mb (technical limit is 1 << 24). It also uses somewhat more complicated (vs LZ4) stream encoding, using "huffman like" codes (without doing any huffman of course). It gives more flexibility while trying not to lose speed too much. Overall LZ5 is somewhat slower than LZ4, but still comparable with most of "fast LZ" kind of things in general. While it can reach zlib-like ratios. In some selected cases it can actually beat zlib due to larger window, which looks funny to my taste, granted decompression speed is like 2-3x faster.
Last edited by xcrh; 19th January 2016 at 19:51.
Reason: forum dislikes "shift left" operator and screwed message up :(
22nd January 2016, 10:57
Thanks, I have a few more questions:
1. Can I make money on such things? Or its just for fame/fun? I "live" in a bad region and have income only from Internet. So I'm looking for remote work.
2. What development environment for C (with integrated debugger) is most suitable for the development of such programs?
By the way: Jan 19 I received Chinese fanless PC. I think, in this store prices are cheapest:
17th February 2016, 15:23
My small test lile LZ4 is available for download, see http://encode.ru/threads/2452-My-sma...by-Yann-Collet
My test like lz5 will be soon.
Tags for this Thread