Page 1 of 3 123 LastLast
Results 1 to 30 of 66

Thread: FP8 (= Fast PAQ8)

  1. #1
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts

    FP8 (= Fast PAQ8)

    FP8 (= Fast PAQ is based on paq8px_v69 compressor. It has removed some models so it is faster on default data. It is only little bit slower than paq8pf (at least on my machine) while providing better compression ratio.
    Special data (wav, bmp, jpeg) are compressed with original model (with same speed and ratio as paq8px)

    speed:
    - paq8px_v69 = 50.8s
    - paq8pf = 6.0s
    - fp8 = 6.6s

    compression ratio (SFC -7):
    - paq8px_v68 = 8764929 bytes
    - paq8pf = 9402940 bytes
    - fp8 = 9167064 bytes
    Attached Files Attached Files

  2. #2
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Thanks Jan! Going to test it soon.

  3. #3
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    Quote Originally Posted by Skymmer View Post
    Thanks Jan! Going to test it soon.
    Can you please compare it with paq8pf and nanozip best mode?

  4. #4
    Member Wladmir's Avatar
    Join Date
    Apr 2010
    Location
    Brazil
    Posts
    18
    Thanks
    0
    Thanked 0 Times in 0 Posts

    FP8 even faster

    Jan Ondrus, you could develop a new version of FP8 without using the compression model special data (wav, bmp, jpeg) of paq8px. That would make the FP8 even faster.

  5. #5
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    520
    Thanks
    188
    Thanked 166 Times in 74 Posts
    Quote Originally Posted by Wladmir View Post
    Jan Ondrus, you could develop a new version of FP8 without using the compression model special data (wav, bmp, jpeg) of paq8px. That would make the FP8 even faster.
    I think this is what paq8pf is doing and that this actually is the reason why FP8 was created: To give better compression ratio than paq8pf on special data, but still offering a speedup on default data.

    So, have a look at paq8pf instead if you're looking for best speed on all kind of data.

    EDIT: It seems that paq8pf has been stopped because of FP8 and the executable attachment has been removed, but I'm sure you can get a copy from someone (for example Jan).
    Last edited by schnaader; 5th May 2010 at 20:12.
    http://schnaader.info
    Damn kids. They're all alike.

  6. #6
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    Quote Originally Posted by Wladmir View Post
    Jan Ondrus, you could develop a new version of FP8 without using the compression model special data (wav, bmp, jpeg) of paq8px. That would make the FP8 even faster.
    Yes, but using default model for special data is not good idea (it would mean bad compression ratio for such data). I think special models should/can be made faster instead (at least as fast as default model in current version). This can be done by reducing number of contexts/predictions used by models.

  7. #7
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    Quote Originally Posted by schnaader View Post
    I think this is what paq8pf is doing
    I don't think it is.

    Quote Originally Posted by schnaader View Post
    and that this actually is the reason why FP8 was created: To give better compression ratio than paq8pf on special data, but still offering a speedup on default data.
    The reason was: Make paq version with same or simalar speed as paq8pf but with better compression ratio and with source code available.

    EDIT:
    paq8pf.exe
    Last edited by Jan Ondrus; 5th May 2010 at 20:51.

  8. #8
    Member Wladmir's Avatar
    Join Date
    Apr 2010
    Location
    Brazil
    Posts
    18
    Thanks
    0
    Thanked 0 Times in 0 Posts

    comparisons

    I made comparisons at the highest supported by my computer. (1 GByte DDR2).

    method | level | size
    none | 0 | 163.340 bytes
    paq8px | -7 | 80.64s | 33.086bytes
    fp8 | -7 | 9.77s | 35.840bytes
    NanoZip | nz_cm | 1.96s | 37.190bytes
    paq8pf | -7 | 8.92s | 37.370 bytes
    paq9a | -8 | 3.19s | 38.730bytes

  9. #9
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,491
    Thanks
    730
    Thanked 655 Times in 351 Posts
    Quote Originally Posted by Wladmir View Post
    I made comparisons at the highest supported by my computer.
    what about lpaq? it may be faster and compress better than paq9

  10. #10
    Member Wladmir's Avatar
    Join Date
    Apr 2010
    Location
    Brazil
    Posts
    18
    Thanks
    0
    Thanked 0 Times in 0 Posts

    sse2

    Jan Ondrus, could you create the version "fp8_v1_sse2.exe"? The version "paq8px_v69_sse2.exe" was quick on my computer.

  11. #11
    Member
    Join Date
    Mar 2009
    Location
    Prague, CZ
    Posts
    60
    Thanks
    27
    Thanked 6 Times in 6 Posts
    Nice speedup without loosing too much.

    i tried compressing ~3.5 GB (2 files), and it was ok, but the info displayed at the end was not:

    ....
    1/2 Filename: cc.tar.001 (1887436800 bytes)
    ....
    Compressed from 1887436800 to 1267174912 bytes.
    2/2 Filename: cc.tar.002 (1559366144 bytes)
    ....
    Compressed from 1559366144 to -1267174961 bytes.
    ....
    Total -848164352 bytes compressed to -1 bytes.
    Time 50797.76 sec, used 1515851009 bytes of memory

  12. #12
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    161
    Thanks
    53
    Thanked 13 Times in 9 Posts
    Quote Originally Posted by mhajicek View Post
    Nice speedup without loosing too much.

    i tried compressing ~3.5 GB (2 files), and it was ok, but the info displayed at the end was not:

    ....
    1/2 Filename: cc.tar.001 (1887436800 bytes)
    ....
    Compressed from 1887436800 to 1267174912 bytes.
    2/2 Filename: cc.tar.002 (1559366144 bytes)
    ....
    Compressed from 1559366144 to -1267174961 bytes.
    ....
    Total -848164352 bytes compressed to -1 bytes.
    Time 50797.76 sec, used 1515851009 bytes of memory
    The problem was that you tried compressing more than 2GB. PAQ Compressors can handle files and folders with a total of 2GB or less space

  13. #13
    Member
    Join Date
    Mar 2009
    Location
    Prague, CZ
    Posts
    60
    Thanks
    27
    Thanked 6 Times in 6 Posts
    moisesmcardona:
    I know about this limit, but it is imho a single file limit (these files are immediately skipped), not limit of total amount of data to be compressed, at least my 2,5GB archive (3,5GB dec.) works perfectly. I just reported the incorrect numbers diplayed in console window, probably some counters are not designed for so big archives.
    Last edited by mhajicek; 18th May 2010 at 12:22.

  14. #14
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,254
    Thanks
    305
    Thanked 775 Times in 484 Posts
    That problem is fixed in zpaq and zp. There is no file size limit.

  15. #15
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts

    new version

    fp8_v2 version with minor changes in model compared to fp8_v1
    Attached Files Attached Files

  16. #16
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,254
    Thanks
    305
    Thanked 775 Times in 484 Posts
    Nice improvement on the Silesia corpus. BTW I had to compress ooffice on the D: drive because of the g++ tmpfile() bug. It only happens when .exe is detected.

    http://mattmahoney.net/dc/silesia.html

  17. #17
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,254
    Thanks
    305
    Thanked 775 Times in 484 Posts
    #8 on LTCB. I decided to give it its own entry since it is a fork of paq8. http://mattmahoney.net/dc/text.html#1544

  18. #18
    qqwertyy
    Guest
    I found bug in fp8* code or in g++/glibc or in my g++ options knowledge :

    0)my Linux box

    0a)x86_64:
    Linux mobilestar 2.6.32.59 #1 SMP Mon Mar 19 20:27:43 GMT 2012 x86_64 Intel(R) Pentium(R) Dual CPU T2370 @ 1.73GHz GenuineIntel GNU/Linux
    gcc-g++-4.2.3-x86_64-1
    glibc-2.7-x86_64-12ttt

    0b)i686:
    Linux mobilestar 2.6.32.59 #1 SMP PREEMPT Mon Mar 19 16:48:50 UTC 2012 i686 Intel(R) Pentium(R) Dual CPU T2370 @ 1.73GHz GenuineIntel GNU/Linux
    gcc-g++-4.2.3-i486-1
    glibc-2.7-i486-12_slack12.1


    1)

    1a)x86_64(pure):
    g++ -O2 -static -fPIC -DUNIX -DNOASM -fomit-frame-pointer fp8.cpp -o fp8.x86_64

    1b)i686:
    g++ -O2 -static -march=i686 -DUNIX -DNOASM -fomit-frame-pointer fp8.cpp -o fp8.i686


    2)after successful compression on pure i686 or pure x86_64 via command:
    fp8_v2 -6 Modes

    i have to different files Modes.fp8 and some members of Modes.fp8 cannot be unpacked
    on the system with architecture different architecture:

    2a) unpacking x86_64 archive (packed on x86_64 via x86_64 executable ) on i686 via i686 executable :
    http://pastebin.com/rY0vKnzH

    2b) unpacking i686 archive (packed on i686 via i686 executable ) on x86_64 via x86_64 executable :
    http://pastebin.com/6G54Rj9p

    2c) unpacking i686 archive (packed on i686 via i686 executable ) on x86_64 via the same i686 executable :
    http://pastebin.com/Khyp7MYT

    The same history with fp8_v1 (but on the members "./Modes/PSKdesc.htm" and "./Modes/Throb1X.png")

    P.S.
    I`m sorry for my broken english.

  19. #19
    qqwertyy
    Guest
    http://zalil.ru/33069253

    c9a90eb6b7ff09b175340c8ce1ea87fb *fp8-bug.zip

    Test files set:

    http://zalil.ru/33069268

    9c79ef0f6bf5e133aaabb8f81f2a6e66 *Modes.7z

  20. #20
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    It is known that different compiles produce different archives when compressing WAV/audio files in paq8px and fp8 programs. It is because fp8 and paq8px contain wavmodel by Florin Ghido which uses Floating-point arithmetic for its matrix computations. I was unable to rewrite it with Integer Arithmetic while providing same compression ratio. Maybe someone more educated can do it.
    Attached Files Attached Files

  21. #21
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,046
    Thanks
    165
    Thanked 874 Times in 437 Posts
    I tried adding this - http://nishi.dreamhosters.com/u/double.inc
    and compression really drops when __int64 is used for internal representation (and a little when float is used too),
    while it stays the same with double.
    But I think it should be possible to add manual handling of floating point there, and get a working integer port.

  22. #22
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    860
    Thanks
    442
    Thanked 169 Times in 80 Posts
    Hey Jan,

    I have done decompression tests on my testsets. 10 of 11 sets could be decompressed properly (MD5 checksums matched) but the sources.tar.fp8 could not be decompressed error-free. Can you please have a look on it?

  23. #23
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts

    new version

    bug fix in jpegmodel
    should be a little slower with a little better compression ratio.
    Attached Files Attached Files

  24. #24
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,254
    Thanks
    305
    Thanked 775 Times in 484 Posts
    Improved compression ratio on the Silesia corpus. http://mattmahoney.net/dc/silesia.html

  25. #25
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by Jan Ondrus View Post
    should be a little slower with a little better compression ratio.
    Jan, thanks for new version.
    Quick and dirty test on 30 jpeg files with total size of 75 873 544 bytes. Performed on i5-2500K @ 3700 MHz.
    Code:
    original   -------     75 873 544
    fp8_v2     657.281     57 015 208
    fp8_v3     671.625     57 029 037
    Last edited by Skymmer; 15th May 2012 at 19:31. Reason: typos

  26. #26
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,254
    Thanks
    305
    Thanked 775 Times in 484 Posts
    Compression is also improved on LTCB from .1544 to .1532 using 8% more time. It is still in 8'th place.
    http://mattmahoney.net/dc/text.html#1532

  27. #27
    Member
    Join Date
    Jun 2012
    Location
    Deutschland
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts
    a small note: I've tried to build a library out of fp8_v2 to be able to use in own programs, but without success. when I compress a file and in the same run decompress the same file it's not depacked correctly. I've changed many static variables to use dynamic allocated ones and reinitialised the global variables, but with this configuration I've got bufferover- or underruns. I've tried running only the decompressor with a valid compressed file and it generated the correct depacked file, but I've still got bufferoverruns.
    I might have introduced them, but I think you doesn't get the debug-message because most of your buffers are static ones and they are not checked by the debug runtime.
    and even after using dynamic alloced structures and filling the buffers with zeros the depacked content was only correct (only when compression the file before trying to decompress the file) for the first 20bytes and afterwards all was lost... but perhaps this is also the problem why different compiled programs will generate different compressed versions or your problems with the doubles. perhaps the compressed output is dependent on garbage in the memory. has anyone tried to compress a file on one os and decompress the file on a different os?

    I might have introduced the errors with my changes, so be careful with my findings...

  28. #28
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,046
    Thanks
    165
    Thanked 874 Times in 437 Posts
    it may be related to the already mentioned floating-point problem.
    If encoder and decoder are compiled separately and in different environments, its quite possible that they won't be compatible.
    Try using strict fp options for compiler.

  29. #29
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    I want to congratulate the author of this fantastic compressor that you can still improve a lot. I just wanted to say that I should simplify the code to get better in less time and that the road taken in later versions is perhaps not the right one!
    Good job !
    Best Regards !

  30. #30
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,254
    Thanks
    305
    Thanked 775 Times in 484 Posts
    The variables are static because they need to retain their values between function calls. If you are trying to make it thread safe, then you have a lot of work to do.

Page 1 of 3 123 LastLast

Similar Threads

  1. PAQ8 - Download Page
    By Jan Ondrus in forum Data Compression
    Replies: 7
    Last Post: 7th October 2010, 21:14
  2. Inline assembly routines for paq8
    By Shelwien in forum Data Compression
    Replies: 24
    Last Post: 26th August 2009, 22:22
  3. deflate model for paq8?
    By kaitz in forum Data Compression
    Replies: 2
    Last Post: 6th February 2009, 21:48
  4. PAQ8 tests
    By kaitz in forum Forum Archive
    Replies: 4
    Last Post: 17th January 2008, 15:03
  5. PeaZip v1.3 now with PAQ8 support!
    By LovePimple in forum Forum Archive
    Replies: 29
    Last Post: 9th February 2007, 16:58

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •