I am new here and it seems to be good forum about compression, so I 'm asking here.
Currently I'm preparing for my diploma thesis and one of its part is to develop my own (database-like) file format to store data. The bytes (better say "table-columns") in this file are compressed and I need to achieve very good compression with good speed. Let consider following test case:
I have binary test file which has 592 MB uncompressed. If I use ZLIB, it will be compressed to 190 MB in 30 seconds. Speed is good, but compression isn't. BZip2 (or using some BWT implementation) and LZMA are too slow.
Then, I have found source code for GRZipII and using ST4 + MTF + Arithmetic coding (according to source, it does RLE too), the file was compressed to 100 MB in 1.5 minute. And it's very good, but there's no documentation and neither one comment in GRZip source code, so I can't use this code.
So I have tried to find implementation for each method. Now I use ST2 + MTF + RLE + range coder. I get 120 MB in 1 minute. I'm trying to find any solution if it's possible to use some other method to compress it more. I found library OpenBWT which contains test for some RLE0 + "Simple Structured Coding Model" and it outputs very good filesize. But it doesn't output any compressed data. Also decoder isn't there.
And I can't find any other implementation of RLE0 nor "Simple Structured Coding Model". ST2 and Range coder can be found here - http://www.compressconsult.com/. Using any other ST or arithmetic coding doesn't help (time and also filesize is bigger)
Could you help me, please? Thank you!