Results 1 to 26 of 26

Thread: PAQ8O released!

  1. #1
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by Matt Mahoney
    Andreas Morphis has made some improvements to the .bmp model in paq8n. The new program is paq8o. There is also a paq8osse.exe which requires SSE2 support (I think Pentium 4 or higher), which is a little faster, but archive compatible with paq8o.exe. Some tests with rafale.bmp:

    634,547 66 sec with paq8n -5
    551,665 78 sec with paq8o -5, 72 sec with paq8osse -5
    (beats WinRK)

    Non .bmp data should compress about the same as paq8n. I will run more tests but I expect paq8osse to be about 10% faster than paq8n on most data other than .bmp. The speedup is due to compiling with Intel C++.
    http://cs.fit.edu/~mmahoney/compression/

    Mirror: Download PAQ8O + bug fix

  2. #2
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    erll, after Z they can use russian dictionary

    and rafale isn't typical bmp. at least with my program it doesn't use MM compression

  3. #3
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by Bulat Ziganshin
    erll, after Z they can use russian dictionary
    Good idea!

  4. #4
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quick (-7) test...

    Test Machine: AMD Sempron 2400+

    PAQ8O compressed bliss.bmp to 288,994 bytes in 61.25 seconds.

  5. #5
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    Quote Originally Posted by Bulat Ziganshin
    and rafale isnt typical bmp. at least with my program it doesnt use MM compression
    True, it is actually a 16 bit image that is "sign extended" at the wrong end. All the green values end with bits 000 or 111. All the red and blue values end with 0000 or 1111. If you set all the extra low bits to 0 (last 2 of green, last 3 of red and blue), then it improves the image quality and makes it more compressible like this:

    635,470 rafale.bmp.paq8l
    623,957 out.bmp.paq8l

    I havent tried it with paq8o yet.

  6. #6
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    It also works for paq8osse -5

    551,665 rafale.bmp.paq8n
    542,222 out.bmp.paq8n

    Here is the program to drop the low bits of rafale.bmp to out.bmp

    // drop low bits of rafale.bmp -> out.bmp
    #include <stdio.h>
    int main() {
    FILE* in=fopen("rafale.bmp", "rb");
    if (!in) {perror("rafale.bmp"); return 0;}
    FILE* out=fopen("out.bmp", "wb");
    for (int i=0; i<54; ++i) putc(getc(in), out); // copy header
    int c;
    while ((c=getc(in))!=EOF) {
    putc(c&0xf8, out); // blue
    putc(getc(in)&0xfc, out); // green
    putc(getc(in)&0xf8, out); // red
    }
    return 0;
    }

    Here is the program to put them back. The output is identical to rafale.bmp.

    // put back low bits of rafale.bmp (out.bmp -> rafale2.bmp)
    #include <stdio.h>
    int main() {
    FILE* in=fopen("out.bmp", "rb");
    if (!in) {perror("out.bmp"); return 0;}
    FILE* out=fopen("rafale2.bmp", "wb");
    for (int i=0; i<54; ++i) putc(getc(in), out); // copy header
    int c;
    while ((c=getc(in))!=EOF) {
    putc(c+((c>>3)&1)*7, out); // blue
    c=getc(in);
    putc(c+((c>>2)&1)*3, out); // green
    c=getc(in);
    putc(c+((c>>3)&1)*7, out); // red
    }
    return 0;
    }

  7. #7
    Member
    Join Date
    Jan 2007
    Location
    Moscow
    Posts
    239
    Thanks
    0
    Thanked 3 Times in 1 Post
    Matt, you are the true hacker Thanks!

    P.S. If you'll change both "fc" and "f8" masks to "70" you'll get funny surrealistic picture, something like heat scanning And this works for any input picture

  8. #8
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks for the info Matt!

  9. #9
    Member
    Join Date
    May 2008
    Location
    HK
    Posts
    160
    Thanks
    4
    Thanked 25 Times in 15 Posts
    Quote Originally Posted by Matt Mahoney
    it is actually a 16 bit image that is "sign extended" at the wrong end. All the green values end with bits 000 or 111. All the red and blue values end with 0000 or 1111.
    for 16bit bmp bitmaps files, compression=3 (BI_BITFIELDS) and 3 bitmasks are placed in palette area.
    so you should check the bit masks instead.

  10. #10
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    rafale.bmp is stored as 24 bit image. There is no compression (value is 0) and no palette. I know it was originally a jpeg image, then a wallpaper on Werner's computer. When you make a jpeg a wallpaper, Windows converts it to .bmp. I think at some point it was converted to a 16 bit image, and then to 24 bit, but I don't know how. I have not seen this kind of artifact in any other images.

    There is a smaller version of the original at http://www.defenseindustrydaily.com/cat/lockheed-m artin/page/33/
    but I have not found the original.

  11. #11
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    Quote Originally Posted by LovePimple
    PAQ8O compressed bliss.bmp to 288,994 bytes
    btw:
    winrar compresses bliss.bmp to 382 895 bytes (382 822 internally, without headers). you must select force text compression (ppm), set model order to three and select force truecolor bitmap compression in advanced options.

  12. #12
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    PAQ8O v2 has been released. PAQ8O v2 is basically the same as the unofficial bug fix version which was available from my home page a short time after PAQ8O v1 was first released.

    http://cs.fit.edu/~mmahoney/compression/

    Mirror: Download PAQ8O v2


    The bug fix version is still available from my home page.

  13. #13
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    I checked that LovePimple's paq8o bug fix is compatible with paq8o v2. Both versions of paq8osse.exe also work on my Athlon-64 3500+ and both are also compatible with paq8o.

  14. #14
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks for testing Matt!

  15. #15
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    We now wait for PAQ8P!

  16. #16
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    872
    Thanks
    457
    Thanked 175 Times in 85 Posts
    I am currently running PAQ8oSSE2 v2 on my reference system.
    This is the 6th try to compress the bmp testset, but unfortunately my vista64 shuts down with bluescreen after 10 hours. the resulting archive size is 277 MB. Is there a way to check if PAQ8 did finish compressing before bluescreen incident? I mean, how can I verify that all bits of my bmp.tar are inside PAQ8o archive (apart from starting decompression)? Does PAQ8o give a warning or calculates CRC if archive is broken (because of my PC crash)?

    I don't know if the crash was caused by PAQ8o, so I don't want to
    produce panic.

  17. #17
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    Quote Originally Posted by LovePimple
    We now wait for PAQ8P!
    paq with lpc model for wave files would be good

    theres over 10 mb of wav data on mfc testset. with lpc model paq should be able to shave one or two megabytes out of compressed file.

  18. #18
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Good idea!

  19. #19
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    for the interested audience:
    http://sourceforge.net/projects/powerpaq/

    Best regards!

  20. #20
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    Quote Originally Posted by Stephan Busch
    Is there a way to check if PAQ8 did finish compressing before bluescreen incident?
    Not really without decompressing, but if the output file size is a multiple of 4096 then the file was probably not closed. Is the output the same size each time the computer crashes? Also, does it crash on paq8o (not sse)? It was compiled with g++ instead of Intel.

  21. #21
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    872
    Thanks
    457
    Thanked 175 Times in 85 Posts
    The crashes were produced by weak web connection and not by PAQ8oSSE. The tests are finished now and I will put the results online this night. I decompressed the file to verify and there were no errors. As expected, PAQ8o now leads bitmap compression with a winning entry sending WinRK to 2nd place.
    If only a WAV filter could be included, PAQ8o would become this years number one. Is there also a chance to add ECM filter for disc image files (.IMG, .BIN)?

  22. #22
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    301
    Thanks
    26
    Thanked 22 Times in 15 Posts
    i second the ECM filter..and the wav filter ..

  23. #23
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    872
    Thanks
    457
    Thanked 175 Times in 85 Posts
    what means 'I second' ?, Maadjordan

  24. #24
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    301
    Thanks
    26
    Thanked 22 Times in 15 Posts
    i vote for your suggestion (ecm,wav) filters in paq8

  25. #25
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    872
    Thanks
    457
    Thanked 175 Times in 85 Posts
    cool

    Let's see if our voice is heard by master Matt

  26. #26
    Member
    Join Date
    Sep 2007
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by LovePimple
    Non .bmp data should compress about the same as paq8n. I will run more tests but I expect paq8osse to be about 10% faster than paq8n on most data other than .bmp. The speedup is due to compiling with Intel C++.
    There is a minor speedup other than compiling with Intel C++. Training is skipped if there is no error (err=0). The speedup depends on the randomness of the data being compressed. For text/log files paq8o is up to 8% faster than paq8n (both compiled with Mingw with the same compiler options).

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •