Okay! A new version featuring improved exe compresison is here!
quad102a.zip (26 KB)
Thanks! At least something to break today's depression...
Thanks Ilia!Originally Posted by encode
Ditto!Originally Posted by Black_Fox
Quad 1.02a improves on its predecessor by 170,475 bytes in my test.
Some files were inflated by few bytes (XM, MP3, PDF, PSD, BMP, PPS, SAVE).
Performance on EXE and DLL files respectively:
original - 2,232,320 - 3,432,448
quad 1.01a - 693,331 - 1,519,479
quad 1.02a - 624,926 - 1,410,564
Possibly, all things with auto detection hurts compression is some cases. This exe one can produce tiny inflate on some binary data, on ASCII text files this exe filter has no effect.Originally Posted by Black_Fox
You're true, it's better with auto anyway. I like simplicity of your compressors' UI - only e|d, infile, outfile parameters.
Could it be possible to look after filetype headers, so additional filtering is applied only on files exe-filter is made for?
It is possible. For example, PIM archiver looks for file header and if needed enables a special fitler.Originally Posted by Black_Fox
For EXE, DLL, OCX files we can look for "MZ" at the beginning. Cirrently I will keep auto, since this approach is effective for TAR and other files with x86 code (Linux ELF executables, etc.).
Also I have an idea about to add some simple header to the QUAD file.
Current QUAD file header:
<uncompressed size>, 4 bytes
<compressed data>, X bytes
As the example, improved header:
"quad" magic, 4 bytes
flags or transformType, 1 byte
<uncompressed size>, 4 bytes
<compressed data>, X bytes
With flags we can define anything, including:
And extra reserved flag for future header extensions.
I we dont plan any additional features inside QUAD files, we can define just "transformType" field to enable or disable any transformations. For example, the x86 transformer can be enabled by default - since it uses auto detection, but if user wish, user can disable it. But we must have a lots of filters first...
Thank you Johan!Originally Posted by Johan
143 EXE files in one tar-file 327 Mb (largest file - 18 Mb, smallest - 0.5 Mb; from my "crogram files") :
NAME - TIME compress/decompress - Result size:
quad 1.02a - ( compress in 21 min 25 sec, decompress in 2 min 14 sec ) - 138 Mb
WinRAR 3.62 - (max 4Mb solid) - ( compress in 6 min 17 sec, decompress in 38 sec) - 128 Mb
7-zip 4.43b - (Ultra 32 Mb solid) - ( compress in 9 min 00 sec , decompress in 49 sec ) - 110 Mb
WinZip 11 - (Optim. for best compr.) - ( compress in 6 min 55 sec , decompress in 6 min 51 sec ) - 140 Mb
Pim 1.25b - ( compress in 6 min 03 sec , decompress in 5 min 13 sec ) - 143 Mb
Huh, at least it is not a flawless victory from the world's best compressors! Quad is holding punch! If seriously:
WinZip 11 – is a lame usage of the stolen PPMd compression engine.
WinRAR additionally to the E8/E9 transformer has stuff like table/delta transformer, which can make the difference.
7-Zip is King of the World!!! Nuff said!
Well, mess with the best die like the rest... But if you compare Quad to others, keeping in mind memory usage/decompression speed (and to be honest the size of source code, which is even about a few times smaller compared to the LZMA decoder...)
I is here!
Test for TC 5.1 dev7 - ( compress in 31 min 16 sec , decompress in 7 min 45 sec ) - 120 Mb.
Note on Quad. Currently, I use a brute-force approach to get the best possible compression from this engine. For example, if I change Flexible Parsing (Max mode) to the Lazy Evaluation (Normal mode), compression will be about 4x-8x or even more times faster with really tiny compression loss. (I'm just focusing on decompression speed only)
Dont forget the stolen Wavpack!Originally Posted by encode
I nearly fell off my chair when I saw this! I understand why you wrote it though.Originally Posted by encode
Quad is showing very good performance even at this early stage in its development. It should be a truly awesome compressor when the final is released!
And for Pimple 1.43b result is:
Pimple 1.43b - (Maximum 128 Mb) - ( compress in 33 min 56 sec , decompress in min sec ) - 129 Mb.
I hope Quad will be improved in final release.
Currently Quad stays in active development stage. Allso I'm thinking about:
1. Improved LZ output coding - i.e. to eliminate the "A10.jpg" problem. Since here we use arithmetic coder, with just slightly improved model we will be able to compress this JPEG file. To do so we must use some kind of SSE technique, probably like in LZPXJ:
encode(freqUnder + (s * h), freq + h, totFreq + (N * h));
Anyway, currently I'm consulting with some people about that.
2. Table and/or Delta preprocessor, of course with auto detection. I will try to find a fast and simple solution that able to improve compression of EXE files. For example, try to Enable/Disable some options before compressing files with WinRAR:
Enable 32-bit executable (Pentium) compression
Enable Delta compression
and see how compression changes.
How about adding a feature that stores a file in the archive without compression if the compressed file size is bigger than original?
Currently, I am working on the compression engine. The storing files without a compression are not a compression related trick. Moreover, what if a TAR archive will already contain a compressed file somewhere in the middle?Originally Posted by Black_Fox
So anyways, currently I am looking into the best PPM sources (PPMD+, PPMZ, PPMd, ...) to find some additional modeling tricks and ideas...
it wont find already compressed part in the middle of the file, but if the tar archive contents were already compressed entirely, you could get some kBs out of it...Originally Posted by encode
edit: or maybe you could add some precomp routines... when the source code gets available
Of course, it will be better if the engine will be more stable on already compressed data...
How about adding small GUI for Quad? Like Dark or better...
Currently, I am focused on compression engine only. For example, I am experimenting with "Full Exclusions" – a well-known trick for the PPM. Note the most PPM coders (PPMd, PPMZ, HA) uses such trick. With this feature, Quad shows some improvements on ALL files. On text files, the difference is small, and on binary and random data, we can find a nice improvement.Originally Posted by Squxe
The GUI and adding Quad support for PIM is somewhere far in future...
Some testing results:
three files form the SFC:
A10.jpg - as a hard-to-compress file
acrord32.exe - as a binary file
world95.txt - as a text file
Modifications of Quad:
Quad 1.02a - current public version
Quad 1.03a-fb - uses full blending
Quad 1.03a-fx,x1.0 - uses full exclusions, model update speed x1.0
Quad 1.03a-fx,x1.5 - uses full exclusions, model update speed x1.5
Quad 1.03a-fx,x2.0 - uses full exclusions, model update speed x2.0
Quad 1.03a-fb: 831,899 bytes
Quad 1.03a-fx,x1.0: 841,869 bytes
Original: 842,468 bytes
Quad 1.03a-fx,x1.5: 844,833 bytes
Quad 1.03a-fx,x2.0: 848,317 bytes
Quad 1.02a: 855,819 bytes
Quad 1.03a-fb: 1,505,736 bytes
Quad 1.03a-fx,x2.0: 1,511,596 bytes
Quad 1.03a-fx,x1.5: 1,515,523 bytes
Quad 1.03a-fx,x1.0: 1,523,136 bytes
Quad 1.02a: 1,524,108 bytes
Original: 3,870,784 bytes
Quad 1.03a-fx,x1.0: 611,203 bytes
Quad 1.03a-fx,x1.5: 612,146 bytes
Quad 1.02a: 613,119 bytes
Quad 1.03a-fx,x2.0: 613,470 bytes
Quad 1.03a-fb: 618,052 bytes
Original: 2,988,578 bytes
I think the best one is the "Quad 1.03a-fx,x1.5". Quad with full blending becomes noticeable slower, update speed x1.0 is badly performs on binary data, and x2.0 performs badly on textual data. So, The x1.5 is in the middle. Also, if at all, the speed of Quad with Full Exclusions is not affected and in ALL cases this trick gives compression gain (especially on small and/or hard compressible and binary data).
worms2.tar - Worms 2 Demo game
Quad 1.03a-fb: 8,760,794 bytes
Quad 1.03a-fx,x1.5: 8,872,624 bytes
Quad 1.02a: 8,884,476 bytes
Original: 17,097,728 bytes
mptrack.exe - ModPlug Tracker software
Quad 1.03a-fx,x1.5: 511,209 bytes
Quad 1.03a-fb: 514,184 bytes
Quad 1.02a: 516,120 bytes
Original: 1,159,172 bytes
How much memory does Quad require now?
Exactly the same amount!Originally Posted by LovePimple
Excellent! Results are impressive!Originally Posted by encode
A baseline PPM has potential codespace wastage. This new trick eliminates such problem. Note with higher order PPM such trick is able to kill the speed, but at low orders, it is possible to keep the same speed. Also, note the speed is slightly "affected" (negligible milliseconds) only on small files, with large files speed is the same. In addition, the compression improvement is not depended from the file size and usually it leads to the ~10...15 KB at max. But since we don't loose the speed why not to include such trick?! Anyway, I will continue experimenting...
I have download redline.tar.quad but i am not able to extract it please tellme how i extract this file
How about public release of quad 1.03a (for testing purposes)?
Simply follow next steps:Originally Posted by Anonymous
1. Download the FIRST version of Quad (1.01a)
2. Unzip quad.exe to some folder, for example to drive C:
3. Place the redline.tar.quad to the same place as quad.exe
4. Run the Command Prompt. (Start->Accessories->Command Prompt)
5. Change the default directory (Cocuments and SettingsUser>) to the Quad one. To do so, type:
Cocuments and SettingsUser>cd ..
Cocuments and Settings>cd ..
6. Now, to unpack the Quad file, type this command:
C:>quad d redline.tar.quad redline.tar
7. After unpacking finished, use your favorite archiver to unpack the TAR file!
Hope this helps!