Results 1 to 9 of 9

Thread: paq8pxv - virtual machine

  1. #1
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    138
    Thanked 198 Times in 108 Posts

    paq8pxv - virtual machine

    paq8pxv_v3
    Code:
    • jpeg model cfg (paq7 model)
    jpeg.cfg:
    Code:
    • +2 new context (paq7 model + 2 contexts from paq8a)
    • fix hash table lookup, now compression as expected.
    This is based on PAQ8PXD_V62 and PAQ8PXD_V17v2 ( https://encode.ru/showthread.php?p=47706#post47706 )
    First version (v1) ( https://encode.ru/threads/1464-Paq8p...ll=1#post59098 )
    Original attempt here ( https://encode.ru/threads/1464-Paq8p...ll=1#post42973 ) ( attempt to mix VM from fpaqvm (vm/jit) to paq8pxd_v16 )


    Code:
    File list (18 bytes)
    Compressed from 18 to 22 bytes.
    
    
    1/1  Filename: mill.jpg (7132151 bytes)
    Block segmentation:
     0           | jpeg      |   7132151 [0 - 7132150]
    
    
     Segment data size: 14 bytes
    
    
     TN |Type name |Count      |Total size
    -----------------------------------------
      3 |jpeg      |         1 |   7132151
    -----------------------------------------
    Total level  0 |         1 |   7132151
    
    
    jpeg      stream(1).  Total 7132151
    Compressed model from 16490 to 13387 bytes
    Compressing jpeg    stream(1).  Total 7132151
    JPEG 4912x3264 Stream(1) compressed from 7132151 to 5635767 bytes
     Segment data compressed from 14 to 17 bytes
     Total 7132151 bytes compressed to 5635792 bytes.
    Time 422.71 sec, used 641 MB (672399440 bytes) of memory
    squeezechart:
    PAQ7 (24.12.2005) -5 Matt Mahoney 57,1 5636813




    Source:
    https://github.com/kaitz/paq8pxv


    It is what it is, nothing more.
    Attached Files Attached Files
    KZo


  2. The Following 4 Users Say Thank You to kaitz For This Useful Post:

    Darek (5th February 2019),Mike (4th February 2019),moisesmcardona (6th February 2019),Stephan Busch (7th February 2019)

  3. #2
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    163
    Thanks
    55
    Thanked 13 Times in 9 Posts
    Hi, first of all, thanks for this new release.

    Second, the compressor creates .paq8pxdv(version) instead of .paq8pxv(version), so I'm wondering, is the compressor name actually paq8pxv or paq8pxdv, and if it's the first case, can you fix this tiny string in the code? Or do you have the command line to compile this?

    Thanks!

  4. #3
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    138
    Thanked 198 Times in 108 Posts
    I updated jpeg.cfg file in github.

    Will fix version info in next release.

    My plan if i will pull this off is to use config file for compressor. It will look something like this:
    Code:
    stream -1
    detect recur.det
    decode recur.dec
    encode recur.enc
    compress -1
    stream 0
    detect -1
    decode -1
    encode -1
    compress test3d.cfg
    stream 1
    detect jpeg.det
    decode -1
    encode -1
    compress jpeg.cfg
    stream 2
    detect image1.det
    decode -1
    encode -1
    compress test3img.cfg
    stream 3
    detect image4
    decode -1
    encode -1
    compress test3d.cfg
    stream 4
    detect image8.det
    decode -1
    encode -1
    compress test3i8.cfg
    stream 5
    detect image24.det
    decode image24.dec
    encode image24.enc
    compress test3i24.cfg
    stream 6
    detect audio.det
    decode -1
    encode -1
    compress test3d.cfg
    stream 7
    detect exe.det
    decode exe.dec
    encode exe.enc
    compress test3d.cfg
    stream 8
    detect text.det
    decode text.dec
    encode text.enc
    compress test3d.cfg


    For example:
    stream 1
    detect jpeg.det
    decode -1
    encode -1
    compress jpeg.cfg

    Use stream 1, detect data with jpeg.det, decode none, encode none, compress with jpeg.cfg
    Output file will contain encode and compress model. Used when decompressing.

    Its an idea, no code yet.
    KZo


  5. #4
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    138
    Thanked 198 Times in 108 Posts
    paq8pxv_v4



    • jpeg cfg updated from paq8fthis4
    • vm jit optimize array access and other code
    • fix vm and jit compression differences
    • vm is now signed (int,short, char), only << and >> are unsigned
    • add jpeg detection as external code for vm, for testing
    • change all config files


    jpeg.det has jpeg detection code. Filename hard coded now into executable.
    Attached Files Attached Files
    KZo


  6. The Following 3 Users Say Thank You to kaitz For This Useful Post:

    Darek (16th February 2019),moisesmcardona (15th February 2019),xinix (17th February 2019)

  7. #5
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    138
    Thanked 198 Times in 108 Posts
    paq8pxv_v5

    • use config file for detection, decoding, encoding and compression
    • remove unused code
    • change jpeg detection
    • add exe detection



    Encode and decode transforms not present right now.
    Code:
    stream id=0 model=test3d.cfg
    stream id=1 model=jpeg.cfg
    stream id=2 model=image1.cfg
    stream id=3 model=test3img.cfg
    stream id=4 model=test3i8.cfg
    stream id=5 model=test3i24.cfg
    type id=0 no detect model (-1)
    type id=0 no decode model
    type id=0 no encode model
    type id=0 stream id=0
    type id=1 model=jpeg.det
    type id=1 no decode model
    type id=1 no encode model
    type id=1 stream id=1
    type id=3 model=exe.det
    type id=3 no decode model
    type id=3 no encode model
    type id=3 stream id=0
    Total streams 6, total types 3
    Creating archive test.zip.paq8pxv5 with 1 file(s)...
    
    
    File list (17 bytes)
    Compressed from 17 to 21 bytes.
    
    
    1/1  Filename: test.zip (115864 bytes)
    Block segmentation:
     0           | default   |       897 [0 - 896]
     1           | exe.det   |      8986 [897 - 9882]
     2           | jpeg.det  |     52821 [9883 - 62703]
     3           | default   |        38 [62704 - 62741]
     4           | jpeg.det  |     52821 [62742 - 115562]
     5           | default   |       301 [115563 - 115863]
    
    
     Segment data size: 79 bytes
    
    
     TN |Type name |Count      |Total size
    -----------------------------------------
      0 |default   |         3 |      1236
      1 |jpeg.det  |         2 |    105642
      2 |exe.det   |         1 |      8986
    -----------------------------------------
    Total level  0 |         6 |    115864
    
    
    test3d.cfg   stream(0).  Total 10222
    Compressing test3d.cfg   stream(0).  Total 10222
    Stream(0) compressed from 10222 to 3306 bytes
        Model compressed from 13813 to 10632 bytes
    jpeg.cfg   stream(1).  Total 105642
    JPEG model v4
    Compressing jpeg.cfg   stream(1).  Total 105642
    Stream(1) compressed from 105642 to 55684 bytes
        Model compressed from 21587 to 16629 bytes
     Segment data compressed from 79 to 56 bytes
     Total 115864 bytes compressed to 86376 bytes.
    Time 3.93 sec, used 1229 MB (1289507343 bytes) of memory
    Example detection file:
    Code:
    // For XXXX detection
    int buf0,buf1,mystart;
    int type,state,jstart,jend;
    enum {DEFAULT=1,YOURTYPE}; //internal enum
    enum {NONE=0,START,INFO,END,RESET=0xfffffffe,REQUEST=0xffffffff}; //external enum
    // function will report its state 
    // or if i=-1 then state results otherwise i is pos
    // c4 is last 4 bytes
    void reset(){
        state=NONE,type=DEFAULT,jstart=jend=buf0=buf1=mystart=0;
    }
    int detect(int c4,int i) {
        //if state parameters recuested
        if (i==REQUEST){
            if (state==NONE)  return 0xffffffff;
            if (state==START) return jstart;
            if (state==END)   return jend;
            if (state==INFO)  return 0xffffffff;
        }
        if (i==RESET){
            reset();
            return 0xffffffff;
        }
        buf1=(buf1<<8)|(buf0>>24);
        buf0=c4;
        //detect header
        if (buf1==0xFFFFFFFF && mystart==0){
            mystart=i;
        }
    
    
        if (type==DEFAULT && mystart){
            type=YOURTYPE;
            state=START; 
            jstart=mystart-4;
            return state;
        }
        if (i-mystart>0x100){
            if (type==YOURTYPE){
                state=END;
                type=DEFAULT;
                jend=i;
                return state;
          }
          state=NONE;
          type=DEFAULT;
          mystart=0;
        }
        return NONE;
    }
    
    
    int main() {
        reset();
    }
    Attached Files Attached Files
    KZo


  8. The Following 3 Users Say Thank You to kaitz For This Useful Post:

    Darek (17th February 2019),Mike (17th February 2019),moisesmcardona (17th February 2019)

  9. #6
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    138
    Thanked 198 Times in 108 Posts
    paq8pxv_v6
    encode and decode now working. Only exe decode/encode for now.
    Recursion still missing.



    It looks like what I imagined at first.
    Attached Files Attached Files
    KZo


  10. The Following 2 Users Say Thank You to kaitz For This Useful Post:

    Darek (25th February 2019),moisesmcardona (5th March 2019)

  11. #7
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    138
    Thanked 198 Times in 108 Posts
    paq8pxv_v8

    • enable recursion
    • detect base64, decode and encode
    • ---- v7
    • enable multithreading again
    • detect files for bmp 1,8,24 bit images, text
    • vm: free memory correctly, also in interpreted code allocated memory



    base64 detects only continues stream, no line breaks. like in b64sample or .mht files



    It seems that all this has been accomplished, what I was thinking at first.
    Or... remove matchModel and add as part of cfg, add more detection and transform files..., add some *Maps not present now (IndirectMap). Add decode and conf file to archive...
    Attached Files Attached Files
    KZo


  12. The Following 2 Users Say Thank You to kaitz For This Useful Post:

    Mike (5th March 2019),moisesmcardona (5th March 2019)

  13. #8
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    138
    Thanked 198 Times in 108 Posts
    paq8pxv_v12
    Code:
    
    more changes to SIMD code
    fix compile errors on none SIMD code
    
    
    PIC -s7 513216 bytes compressed to 33336 bytes. used 907 MB (951143616 bytes) of memory
    
    
    paq8pxv_v12_AVX2.exe -s7 Time 25.63 sec
    paq8pxv_v12_SSE42.exe -s7 Time 27.91 sec
    paq8pxv_v12_SSE4.exe -s7 Time 27.70 sec
    paq8pxv_v12_SSSE3.exe -s7 Time 27.63 sec
    paq8pxv_v12_SSE2.exe -s7 Time 28.70 sec
    paq8pxv_v12_MMX.exe -s7 Time 28.61 sec
    paq8pxv_v12_None.exe -s7 Time 184.61 sec
    
    
    This release is compatible with v11
    paq8pxv_v11
    Code:
    cleanup
    fix jpeg cfg
    change test3d cfg, add exe cm
    paq8pxv_v10
    Code:
    add small model to archive, now all decompress info is in archive
    change cfg models
    remove matchModel, add to test3d.cfg simple model
    paq8pxv_v9
    Code:
    add conf and decode to archive
    change test3d.cfg
    Attached Files Attached Files
    KZo


  14. The Following 4 Users Say Thank You to kaitz For This Useful Post:

    Darek (4th June 2019),Mike (4th June 2019),moisesmcardona (7th June 2019),Stephan Busch (5th June 2019)

  15. #9
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    865
    Thanks
    450
    Thanked 170 Times in 81 Posts
    paq8pxv_v12 aborts compression of app.tar at 55%:

    Segment data size: 17941 bytes

    TN |Type name |Count |Total size
    -----------------------------------------
    0 |default | 694 | 206215076
    1 |jpeg.det | 56 | 374204
    2 |exe.det | 335 | 100808758
    3 |bmp8.det | 113 | 442616
    4 |bmp1.det | 72 | 18432
    5 |bmp24.det | 78 | 892776
    7 |text.det | 32 | 27434506
    -----------------------------------------
    Total level 0 | 1380 | 336186368

    Decode compressed to : 1366
    test3d.cfg stream(0). Total 334458340
    jpeg.cfg stream(1). Total 374204
    test3img.cfg stream(3). Total 18432
    test3i8.cfg stream(4). Total 442616
    test3i24.cfg stream(5). Total 892776
    Compressing test3d.cfg stream(0). Total 334458340
    Stream(0) compressed from 334458340 to 61897340 bytes
    Model compressed from 16628 to 6142 bytes
    Compressing jpeg.cfg stream(1). Total 374204
    JPEG model v6
    Stream(1) compressed from 374204 to 258128 bytes
    Model compressed from 20420 to 8595 bytes
    Compressing test3img.cfg stream(3). Total 18432
    3 55.55%

  16. The Following User Says Thank You to Stephan Busch For This Useful Post:

    Darek (8th June 2019)

Similar Threads

  1. Machine Learning to identify weight loss parameters
    By Sportman in forum The Off-Topic Lounge
    Replies: 0
    Last Post: 13th August 2016, 13:10
  2. Split-stream compression (or what?) for x86 machine code
    By Paul W. in forum Data Compression
    Replies: 9
    Last Post: 26th April 2014, 21:35
  3. Virtual Hard Disk Compress/Dedupe
    By JayM in forum Data Compression
    Replies: 6
    Last Post: 22nd July 2013, 00:00
  4. Do you have a 64-bit machine at home?
    By encode in forum The Off-Topic Lounge
    Replies: 22
    Last Post: 4th December 2009, 14:09
  5. Virtual test-machines
    By Vacon in forum Data Compression
    Replies: 7
    Last Post: 14th April 2009, 23:19

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •