Results 1 to 14 of 14

Thread: paq8pxpre

  1. #1
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts

    paq8pxpre

    Combined source from
    paq8o8pre.cpp v2 aka prepaq v2 file (pre)compressor
    using precomp.dll (C) 2007/08 Christian Schneider
    and
    paq8px_v67 file compressor/archiver. Release by Jan Ondrus, Nov. 5, 2009

    Code:
                 paq8o8pre v2 -7 paq8px_v67 -7 paq8pxpre -7
    A10.jpg               638165        637116       637143  (+27)
    acrord32.exe          910902        895521       895544  (+23)
    english.dic           385584        384202       384193  (-9)
    FlashMX.pdf          1821452       3554092      1588181  (-1965911)
    fp.log                264893        254154       254062  (-92)
    mso97.dll            1293710       1261349      1261481  (+132)
    ohs.doc               483033        482511       482153  (-358)
    rafale.bmp            551488        528475       528504  (+29)
    vcfiu.hlp             401874        395307       395273  (-34)
    world95.txt           358516        351914       351983  (+69)
    
    total                7109617       8744641      6778517  (-1966124)
    Attached Files Attached Files
    Last edited by Jan Ondrus; 10th January 2010 at 10:53. Reason: added some test results

  2. #2
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    160
    Thanks
    51
    Thanked 13 Times in 9 Posts
    Thanks!

    BTW, I had to compile the source again since the included exe was asking for "libgcc_s_dw2-1.dll".

    I include here my compiled version.
    Attached Files Attached Files

  3. #3
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by Jan Ondrus View Post
    Combined source from
    paq8o8pre.cpp v2 aka prepaq v2 file (pre)compressor
    using precomp.dll (C) 2007/08 Christian Schneider
    and
    paq8px_v67 file compressor/archiver. Release by Jan Ondrus, Nov. 5, 2009
    Impressive! Thanks Jan

    moisesmcardona, also thanks to you for static compile!

    But sad news is that I probably found an issue. For some reasons PAQ8pxPre refuse to process one test file with lenght of 4 244 176 896 bytes. Some other smaller files are accepted but not this one. The message is:
    Code:
    E:\Test3>paq8pxpre -0 -slow Test.dat
    Test.dat: not found, skipping...
    Nothing to compress
    
    DIR
    23.06.2009  05:42            43 008 libgcc_s_dw2-1.dll
    20.12.2007  17:59           134 952 packjpg_dll.dll
    09.01.2010  12:55           146 944 paq8pxpre.exe
    15.01.2008  10:59            87 040 precomp.dll
    15.01.2010  13:22     4 244 176 896 Test.dat
    20.07.2005  11:48            35 840 zlib1.dll
    Can anybody confirm it ?

  4. #4
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    Quote Originally Posted by Skymmer View Post
    Impressive! Thanks Jan

    moisesmcardona, also thanks to you for static compile!

    But sad news is that I probably found an issue. For some reasons PAQ8pxPre refuse to process one test file with lenght of 4 244 176 896 bytes. Some other smaller files are accepted but not this one.

    Can anybody confirm it ?
    I think there is some file size limit in all paq versions (2^31 = 2 GB).
    See code below.

    Code:
    // Same as expand() except fname is an ordinary file
    int putsize(String& archive, String& s, const char* fname, int base) {
      int result=0;
      FILE *f=fopen(fname, "rb");
      if (f) {
        fseek(f, 0, SEEK_END);
        long len=ftell(f);
        if (len>=0) {
          static char blk[24];
          sprintf(blk, "%ld\t", len);
          archive+=blk;
          archive+=(fname+base);
          archive+="\r\n";
          s+=fname;
          s+="\n";
          ++result;
        }
        fclose(f);
      }
      return result;
    }

  5. #5
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    515
    Thanks
    182
    Thanked 163 Times in 71 Posts
    Quote Originally Posted by Jan Ondrus View Post
    I think there is some file size limit in all paq versions (2^31 = 2 GB).
    This is true, and even if there wasn't, the Precomp DLL is from version 0.3.8 that perhaps also had a 2 GB limit, at least it gets slower after 4 GB. When merging PAQ and the next Precomp version, I'll try to change the PAQ code so it supports files larger than 2 GB - has this been done by someone else in another PAQ version already?

    Anyway, changing the limit even in the standard PAQ versions shouldn't be that hard - just change every file position variable from int (32 bit) to long long (64 bit), handle 64-bit values correctly everywhere (PAQ header, file size display) and use fgetpos/fsetpos instead of fseek (if needed).
    http://schnaader.info
    Damn kids. They're all alike.

  6. #6
    Member [IO]'s Avatar
    Join Date
    Jan 2010
    Location
    England
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I've been toying with the PAQ routines for a few years now (inbetween other projects). My own variants are probably faster because i'm one of the "fanatical optimising freaks".

    The work ALL you guys are doing here is GREAT - thank you

    Just tested "moisesmcardona"'s binary (VC++ ? So PEiD tells me) (I don't use that other compiler) against a binary I compiled with my Intel compiler and thought I would share the results with you;

    Code:
    D:\_PAQwerk\paq8pxpre>paq8pxpre -1 gccout Overlord_Title.mp3
    Creating archive gccout.paq8pxpre with 1 file(s)...
    
    1/1  Filename: Overlord_Title.mp3 (5494863 bytes)
    Precompressing: 100.0% - Time: 764 ms
    Block segmentation:
     0           | default   |   5494893 bytes [0 - 5494892]
    Transformed from 5494863 to 5494893 and compressed to 4609971 bytes.
    
    Total 5494863 bytes compressed to 4610015 bytes.
    Time 221.66 sec, used 37597191 bytes of memory
    
    D:\_PAQwerk\paq8pxpre>p8 -1 intelout Overlord_Title.mp3
    Creating archive intelout.paq8pxpre with 1 file(s)...
    
    1/1  Filename: Overlord_Title.mp3 (5494863 bytes)
    Precompressing: 100.0% - Time: 655 ms
    Block segmentation:
     0           | default   |   5494893 bytes [0 - 5494892]
    Transformed from 5494863 to 5494893 and compressed to 4609971 bytes.
    
    Total 5494863 bytes compressed to 4610015 bytes.
    Time 196.18 sec, used 37597189 bytes of memory
    I used no parameters on compiling, and the old ASM routines in the package. Interestingly, the IC++ binary is 230Kb, whilest compiling it also warned me about the fopen (it's a bit security paranoid) possible holes.

    I could'nt test the other binary as I don't use, nor ever will, that compiler - but it would be good to see the speed difference.

    Hopefuly I shall get time later today to actually browse the source and see if there is any obvious room for speed improvements.

  7. #7
    Member [IO]'s Avatar
    Join Date
    Jan 2010
    Location
    England
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Using my TAQ8.asm file and my pregenerated tables (see the includes in the /OVL folder);

    Code:
    D:\_PAQwerk\PAQ8PX~1>paq8pxpre -1 intel_IO_out Overlord_Title.mp3
    Creating archive intel_IO_out.paq8pxpre with 1 file(s)...
    
    1/1  Filename: Overlord_Title.mp3 (5494863 bytes)
    Precompressing: 100.0% - Time: 717 ms
    Block segmentation:
     0           | default   |   5494893 bytes [0 - 5494892]
    Transformed from 5494863 to 5494893 and compressed to 4609971 bytes.
    
    Total 5494863 bytes compressed to 4610015 bytes.
    Time 171.13 sec, used 37588993 bytes of memory
    
    D:\_PAQwerk\PAQ8PX~1>fc /B intel_IO_out.paq8pxpre gccout.paq8pxpre
    Comparing files intel_IO_out.paq8pxpre and GCCOUT.PAQ8PXPRE
    FC: no differences encountered
    As you can see, binary exacts to the previous precompiled version.

    Enclosed is the sources & binary used - I hope this helps development.
    Attached Files Attached Files

  8. #8
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    Quote Originally Posted by [IO] View Post
    As you can see, binary exacts to the previous precompiled version.

    Enclosed is the sources & binary used - I hope this helps development.
    Your version crashes for me.

    This is not correct replacement for llog function:
    Code:
    #ifdef GENTABS
      if (x>=0x1000000)
        return 256+ilog(x>>16);
      else if (x>=0x10000)
        return 128+ilog(x>>8);
      else
        return ilog(x);
    #else
      return ilogtable[x];
    #endif

  9. #9
    Member [IO]'s Avatar
    Join Date
    Jan 2010
    Location
    England
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    No idea why it crashes - works fine on my Vista (32bit) home premium, but feel free to try a recompile.

    As for the ilog function - again, feel free to manipulate it anyway you see fit. Just hope that those things help you produce something better, thats all.

  10. #10
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by [IO] View Post
    Just tested "moisesmcardona"'s binary (VC++ ? So PEiD tells me) (I don't use that other compiler) against a binary I compiled with my Intel compiler and thought I would share the results with you;
    Thanks for compile! As for moisesmcardona's compile... Judging the linker version (2.56) and sections structure I think its GCC.

    Quote Originally Posted by Jan Ondrus View Post
    Your version crashes for me.
    Runs OK on my system. AMD64 4000+, XP Pro SP3.

  11. #11
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    I take my words back. After more serious testing it appeared that [IO] compile also crashes on my system. I've used my TestBed.tar file and seems that IO's compile crashes on 5_HEROES.MOD file from there. At least on it.

  12. #12
    Member
    Join Date
    May 2009
    Location
    Europe
    Posts
    67
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Skymmer View Post
    seems that IO's compile crashes on 5_HEROES.MOD file
    Same Here. XP 32 bits SP3 E8400

    Looks promising though

  13. #13
    Member [IO]'s Avatar
    Join Date
    Jan 2010
    Location
    England
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    On some data, it crashes on mine too. Or rather it's the precomp.dll which crashes. Possibly some issue between compilers or I forgot to set a flag during compile or something simple which is my fault. Might even be something in the ASM routine. Feel free to recompile and retest - be interesting to know the speed-results on that small hack.

    The essential idea was to demonstrate the advantage of "pre-generated tables" (lookups) over generating "on-the-fly". It's something i've used in all my PAQ variants in the last few years as standard - this is the first i've heard of any crashing. Probably my bad for not checking with the DLL files.

    Aside of that, there are numerous "minor" tweaks for performance advantages that i've integrated, aside of the obvious CPU based ones - but i'll leave them for another day - working code is obviously more important.

  14. #14
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Jan!

    Here are my usual builds for paq8pxpre...

    NOTE: _speed_optimised is now renamed _alt.
    Attached Files Attached Files

Similar Threads

  1. Alternative paq8pxpre builds
    By M4ST3R in forum Download Area
    Replies: 0
    Last Post: 20th January 2010, 18:51

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •