Page 1 of 2 12 LastLast
Results 1 to 30 of 35

Thread: Paq8o7

  1. #1
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    139
    Thanked 198 Times in 108 Posts
    New:
    Detection of BMP 4,8,24 bit and PGM 8 bit images before compress
    On non BMP,PGM,JPEG data mem is lower
    Fixed bug in BMP 8-bit detection in other files like .exe.

    Mem is lower only when BMP,JPEG or PGM models ar not in use, like when compressing calgary.tar

    TIFF is not detected so it is a bit like bug.

    Results:
    Program_______File________InSize___OutSize__Time_M em(bytes)
    paq8o6.exe -3 calgary.tar_3152896 -> 665684__166.31__63025039
    paq8o7.exe -3 calgary.tar_3152896 -> 665684_166.09__55767899
    paq8o6.exe -3 A10.jpg_____842468 -> 648002__86.91__75686392
    paq8o7.exe -3 A10.jpg_____842468 -> 647999__89.73__69088984
    paq8o6.exe -3 lena.bmp____786486 -> 411067__53.44__63025046
    paq8o7.exe -3 lena.bmp____786486 -> 411067__52.03__59186242

    Source:
    http://code.google.com/p/paq8/source
    KZo


  2. #2
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks KZo!

    Mirror: Download

  3. #3
    Member
    Join Date
    May 2008
    Location
    CN
    Posts
    45
    Thanks
    0
    Thanked 0 Times in 0 Posts
    no binary?

  4. #4
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Binary can be found in SVN in directory "trunk", along with source files

  5. #5
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    paq8o7 crashed on enwik8. I tried

    paq8o7 -8 enwik8

    After 1 or 2 seconds it pops up a window "paq8o7.exe has encountered a problem and needs to close" and outputs just a header for enwik8.paq8o7. It also crashes with -7, -6, -5, but not -4. It runs OK on smaller files (enwik7). I tried recompiling with g++ but it has the same problem. I am testing on a Athlon-64 3500+ in WinXP SP2 (32 bit mode) with 2 GB memory.

  6. #6
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    301
    Thanks
    26
    Thanked 22 Times in 15 Posts
    may be the mem reduction for general data stream is the cuase..? KZO will tune it i'm sure.. still he is doing a great job..

  7. #7
    Member
    Join Date
    May 2008
    Location
    CN
    Posts
    45
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Black_Fox
    I found the binary as you mentioned
    thanks

  8. #8
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    139
    Thanked 198 Times in 108 Posts
    Quote Originally Posted by Matt Mahoney
    paq8o7 crashed on enwik8. I tried
    I will test the problem after work and school, if i have time.
    i think the bug is in detect
    KZo


  9. #9
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    ...
    char pgm_buf[32];
    ...
    pgm_buf[pgm_ptr++]=c; // buffer overflow: pgm_ptr can be >= 32!
    ...

    can be fixed with this line:
    if (pgm_ptr>=32) pgm=pgm_ptr=pgmw=pgmh=pgmc=pgmcomment=0;

    http://www.mytempdir.com/2040807
    link contains also minor improvements to jpeg-model and jpeg detection.

  10. #10
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    139
    Thanked 198 Times in 108 Posts
    Quote Originally Posted by Jan Ondrus
    ...
    char pgm_buf[32];
    ...
    pgm_buf[pgm_ptr++]=c; // buffer overflow: pgm_ptr can be >= 32!
    ...

    can be fixed with this line:
    if (pgm_ptr>=32) pgm=pgm_ptr=pgmw=pgmh=pgmc=pgmcomment=0;

    http://www.mytempdir.com/2040807
    link contains also minor improvements to jpeg-model and jpeg detection.
    Uploded this fixes and updates to SVN
    Source:
    http://code.google.com/p/paq8/source
    KZo


  11. #11
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Jan, and KZo!

    Mirror: Download

  12. #12
    Member
    Join Date
    Jul 2008
    Posts
    54
    Thanks
    0
    Thanked 0 Times in 0 Posts

  13. #13
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    139
    Thanked 198 Times in 108 Posts
    Hm enwiki8 still has problems.
    Last night i did not test it, did not have time. Only uploded fixes by Jan.
    KZo


  14. #14
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    I got an email from Niels Fr?hling who seems to have found the bug independently in the PGM detection code. I haven't looked at it myself. Quote:

    But here is a real bug:

    // Detect .pgm image
    if ((buf0&0xFFFFFF)==0x50350A) pgm=i; //possible 'P5 '
    if (pgm && (i-pgm)==1 && c==0x23) pgmcomment=1; //pgm comment
    //not tested without comment
    if (pgm && !pgmcomment && c==0x20 && !pgmw && pgm_ptr) {
    pgm_buf[pgm_ptr++]=0;
    pgmw=atoi(pgm_buf);
    pgm_ptr=0;
    }
    ...

    These lines:

    pgm_buf[pgm_ptr++]=0;

    grow unbound on "pgm_buf", funny enough in enwik _IS_ a 'P5 ...' somewhere (the H&K MP5) after the first 10MB, so testing paq8o7 on enwik8 crashes on my machine.
    Anyway it's not nice to not check for overflow.

  15. #15
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    Localized problematic PGM header in enwik8:
    http://www.mytempdir.com/2041476

    Quote Originally Posted by KZo
    Hm enwiki8 still has problems.
    You updated only source file in resposiotry. There is still old version of exe file!

  16. #16
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    139
    Thanked 198 Times in 108 Posts
    Quote Originally Posted by KZo
    Only uploded fixes by Jan
    Quote Originally Posted by Jan Ondrus
    Quoting: KZo
    Hm enwiki8 still has problems.

    You updated only source file in resposiotry. There is still old version of exe file!
    Binary was old. Sorry for making a mess.

    EDIT:
    How to update memory usage:
    <u>"level: -0 = store, -1 -2 -3 = faster (uses 35, 48, 59 MB)"
    "-4 -5 -6 -7 -8 = smaller (uses 133, 233, 435, 837, 1643 MB)"</u>

    If no model is used then memory is less used, but when all models are used then it is larger or leave it as is.
    KZo


  17. #17
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    We got there in the end!

    Mirror: Download

  18. #18
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Matt, have you tried using OpenMP directives in paq8 for autoparalellization? The latest version of GCC 4.2.2 comes with built in OpenMP support which could maybe help performancewise for those with multi-core (which is probably 1/2 of people nowadays).

  19. #19
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    My results:

    orbitalsniper.tar.paq8o6
    orbitalsniper.tar 2517504 -> BMP 32x1088 BMP 640x480 617163
    2517504 -> 617203
    Time 221.61 sec, used 246291987 bytes of memory

    orbitalsniper.tar.paq8o7
    2517504 -> BMP(8-bit) 800x600BMP 32x1088 BMP(8-bit) 800x600 BMP(8-bit) 4x570BMP 640x480 613044
    2517504 -> 613084
    Time 144.20 sec, used 245632255 bytes of memory

  20. #20
    Member
    Join Date
    Apr 2010
    Location
    El Salvador
    Posts
    43
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Hahobas
    Matt, have you tried using OpenMP directives in paq8 for autoparalellization?
    I tried OpenMP, gets slower. As there is no autoparallelization (except if you use PGI Workstation parallel optimizing compiler), but you have to mark regions to execute in parallel, I tried both - loop and function parallelization.
    Both fail because of significant thread-initialization and inter-thread synchronization of the caches/memory of the neuronal model.
    Also the models are applied sequential, you would to have to develop another model-paradigma, which seperates getMixerContext()/applyMixerContext() ...
    Good luck.

  21. #21
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    I tested paq8o7, no problems. I noticed it also improves JPEG compression.

    Mirrored to http://cs.fit.edu/~mmahoney/compression/

  22. #22
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    One question:

    PAQX VS LPAQX ?

    Lpaq is very faster!

  23. #23
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    paq8o7j with improved JPEG model:

    file / uncompressed / paq8o7 -6 / paq8o7j -6
    a10.jpg / 842468 / 642092 / 638203
    dscn3974.jpg / 1114198 / 827713 / 827071

    http://www.mytempdir.com/2045367

  24. #24
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    139
    Thanked 198 Times in 108 Posts


    So it beats Stuffit 11...

    Uploaded to SVN version paq8o8


    Source,binary:
    http://code.google.com/p/paq8/source
    KZo


  25. #25
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Jan, and KZo!

    Mirror: Download

  26. #26
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    139
    Thanked 198 Times in 108 Posts
    Made HTML version of PAQ8 test results

    http://code.google.com/p/paq8/wiki/PAQ8testResults
    KZo


  27. #27
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks KZo! Nice little chart!

  28. #28
    Member
    Join Date
    May 2008
    Location
    CN
    Posts
    45
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Paq8o8's ratio is less than Paq8o7?
    D:dl>paq8o7 a.jpg
    Creating archive a.jpg.paq8o7 with 1 file(s)...
    a.jpg 313584 -> JPEG 600x845 212747
    313584 -> 212774
    Time 17.72 sec, used 253391980 bytes of memory

    Close this window or press ENTER to continue...


    D:dl>paq8o8 a.jpg
    Creating archive a.jpg.paq8o8 with 1 file(s)...
    a.jpg 313584 -> JPEG 600x845 213085
    313584 -> 213112
    Time 18.17 sec, used 253391980 bytes of memory

    Close this window or press ENTER to continue...

  29. #29
    Member
    Join Date
    May 2008
    Location
    CN
    Posts
    45
    Thanks
    0
    Thanked 0 Times in 0 Posts
    D:dl>paq8o6 a.jpg
    Creating archive a.jpg.paq8o6 with 1 file(s)...
    a.jpg 313584 -> JPEG 600x845 212984
    313584 -> 213011
    Time 16.56 sec, used 270999436 bytes of memory

    Close this window or press ENTER to continue...


    D:dl>paq8o5 a.jpg
    Creating archive a.jpg.paq8o5 with 1 file(s)...
    a.jpg 313584 -> JPEG 600x845 213803
    313584 -> 213830
    Time 12.59 sec, used 286294716 bytes of memory

    Close this window or press ENTER to continue...


    D:dl>paq8o4 a.jpg
    Creating archive a.jpg.paq8o4 with 1 file(s)...
    a.jpg 313584 -> JPEG 600x845 214228
    313584 -> 214255
    Time 13.16 sec, used 267437244 bytes of memory

    Close this window or press ENTER to continue...

  30. #30
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Matt Mahoney
    paq8o8 by KZ, Oct. 23, 2007, improves JPEG compression further.
    Quote Originally Posted by l1t
    Paq8o8s ratio is less than Paq8o7?
    Which means that there are thousands of files, which got compressed better and tens of files (incuding yours ), that got compressed worse.

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •