QuickLZ ZIP - new zip/deflate library

[Lasse Reinhold]

Hope this fills a gap - fast zip compression: http://www.quicklz.com/zip.html

[Bulat Ziganshin]

it's great idea!

[Nania Francesco]

how does it work?

[Bulat Ziganshin]

because Lasse doesn't answer, i will explain how i may make this yourself:

just use fast LZ serach engine (such as quicklz and encode found literals/matches using static huffman tree (STATIC_TREES)

[Lasse Reinhold]

Hi, sorry for late reply

Yeah, Bulat is entirely right

Unfortunatly the deflate format is not designed for speed, so it's not very fast (~58 MB/s compared to 148 MB/s for QuickLZ, for some test files on one of my pc's).

Lasse

[Bulat Ziganshin]

you can try to reuse bit i/o from tornado, may be it will be faster?

[Lasse Reinhold]

Cool, will look at at

What I'm doing now is just

void out(unsigned int data, unsigned int len)
{
unsigned int *dst = dst2 + (bitoffset >> 3);
*dst |= data << (bitoffset & 0x7);
bitoffset += len;
}

where dst2 is a (public) bit-offset pointer. Requires destination memory to be initial zero'ed out.

[Lasse Reinhold]

Oh, dst2 was the byte base pointer and bitoffset is bit offset pointer. But the solution sucks Will look at yours soon.

[LovePimple]

Hope this fills a gap - fast zip compression: http://www.quicklz.com/zip.html

Thanks Lasse!

[Bulat Ziganshin]

Lasse Reinhold
yes, your way isn't fastest possible

[Matt Mahoney]

quicklz_zip.exe failed on enwik8. I decompressed with unzip (InfoZIP 5.52) and the output was the right size (100 MB) but differed from the original. I also get this message:

C:
es>unzip x8.zip
Archive: x8.zip
inflating: enwik8
error: invalid compressed data to inflate

[Lasse Reinhold]

Yikes. I'll fix it later - currently busy with my 1.30 release.

[LovePimple]

Will you be working on 'QuickLZ ZIP' now that QuickLZ v1.30 final has been released?

[Lasse Reinhold]

Either that, or a QuickLZ 1.40. I'm not sure yet.

[LovePimple]

I hope you get time to fix the problem that Matt reported.

[nimdamsk]

How should i compile QuickLZ 1.30 final? I've got these errors:

g++.exe quicklz.c -oq0.exe
quicklz.c:23: error: 'size_t' does not name a type
quicklz.c:24: error: 'size_t' does not name a type
quicklz.c:25: error: 'size_t' does not name a type
quicklz.c:26: error: 'size_t' does not name a type

cl quicklz.c
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.10.3052 for 80x86
Copyright (C) Microsoft Corporation 1984-2002. All rights reserved.

quicklz.c
quicklz.c(23) : error C2061: syntax error : identifier 'qlz_decompress'
quicklz.c(23) : error C2059: syntax error : ';'
quicklz.c(23) : error C2059: syntax error : 'type'
quicklz.c(24) : error C2061: syntax error : identifier 'qlz_compress'
quicklz.c(24) : error C2059: syntax error : ';'
quicklz.c(24) : error C2059: syntax error : 'type'
quicklz.c(25) : error C2061: syntax error : identifier 'qlz_size_decompressed'
quicklz.c(25) : error C2059: syntax error : ';'
quicklz.c(25) : error C2059: syntax error : 'type'
quicklz.c(26) : error C2061: syntax error : identifier 'qlz_size_compressed'
quicklz.c(26) : error C2059: syntax error : ';'
quicklz.c(26) : error C2059: syntax error : 'type'

Thanks.

[Lasse Reinhold]

Hi nimdansk,

To compile the quick.exe project on the website using VC, first comment out the COMPRESSION_LEVEL flag in quicklz.c and define STREAMING_MODE to 960000 (if you want it to be compatible with the demo project on the website).

Now go to the VCquick directory and compile with cl /Ox quick.c

You can also compile the other C demo projects. For example, go to the C directory and compile with cl /Ox block_compress.c or cl /Ox block_decompress.c, etc.

I'm not sure how to define the size_t type in g++, it varies between compilers.

I'll probably add some notes to the manual about compiling soon.

[Lasse Reinhold]

Hmm, seems like the back slash characters got removed in my post (in the directory paths).

[nimdamsk]

Hum... I realised that there is no main() function in quicklz.c thus it couldn't be compiled

I compiled quick.c, but is it possible to make different versions for -0, -1 etc.? Compiling quicklz.c i get the same error as before.

I have:
>g++ -v
Reading specs from E:/Dev-Cpp/bin/../lib/gcc/mingw32/3.4.2/specs
Configured with: ../gcc/configure --with-gcc --with-gnu-ld --with-gnu-as --host=mingw32 --target=mingw32 --prefix=/mingw --enab
le-threads --disable-nls --enable-languages=c,c++,f77,ada,objc,java --disable-win32-registry --disable-shared --enable-sjlj-exc
eptions --enable-libgcj --disable-java-awt --without-x --enable-java-gc=boehm --disable-libgcj-debug --enable-interpreter --ena
ble-hash-synchronization --enable-libstdcxx-debug
Thread model: win32
gcc version 3.4.2 (mingw-special)

C:>q
QuickLZ 1.30. Copyright 2006-2007 Lasse Mikkel Reinhold.

Compiled with:
COMPRESSION_LEVEL using levels.h wrapper
STREAMING_MODE 960000
test_rle defined
speedup_incompressible defined
memory_safe not defined

[Lasse Reinhold]

Hi nimdansk,

There's an older version of the demo project here which compiles with single compression level: http://www.quicklz.com/quick130beta.c

For g++, try locating typedef.h and including it. size_t should be defined there.

[Lasse Reinhold]

I have fixed a bug in QuickLZ ZIP so that I can now compress enwik8 (and other files):

http://www.quicklz.com/zip.html

[LovePimple]

Thanks Lasse!

[Lasse Reinhold]

I have released source if anyone should care - www.quicklz.com/zip.html

Just 355 lines for a stand alone zip/deflate library

[LovePimple]

Thanks Lasse!