Page 1 of 2 12 LastLast
Results 1 to 30 of 44

Thread: simv2.9m

  1. #1
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts

    simv2.9m

    -improve jpeg, image24bpp, image8bpp file compression
    the option in commandline is like paq8pxd, in windows i use -s6 option.
    Attached Files Attached Files

  2. #2
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Which memory setting use -s options? -s9 as I found use about 23-24GB, higher methods crashes.

    Some quick insights. It's based on paq8pdx v18 source code? Scores are similar.

    Version isn't fully stable. There are crash on all image models. Wave model compress all files to 150-200 bytes.... weird.
    JPEG model works ok. WRT and text model also but scores are about 2% worse than paq8pxd.

    Darek

  3. #3
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts
    What image ? 24bpp ? 8bpp ? May you post that image file.. More level need more memory

  4. #4
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    >More level need more memory
    Yes, do you have information which amount of memory is needed to methods -s10 to -s15?

    Accordingo to image - for example 1.BMP


    Attached Files Attached Files
    • File Type: 7z 1.7z (460.8 KB, 30 views)

  5. #5
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts
    In my machine this file can be decompressed successfully..

  6. The Following User Says Thank You to suryakandau@yahoo.co.id For This Useful Post:

    Darek (1st September 2017)

  7. #6
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    872
    Thanks
    457
    Thanked 175 Times in 85 Posts
    By what extent is your version different compared to PAQ8PXD?

  8. #7
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts
    I think the compression ratio is better than paq8pxd,paq8px or emma forjpeg,image24bpp,image 8bpp file

  9. #8
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    I think the compression ratio is better than paq8pxd,paq8px or emma forjpeg,image24bpp,image 8bpp file
    It's depend - my JPEG file - f.jpg scores:

    88'688 bytes - sim 2.9m -s7
    82'021 bytes - paq8px v103 -s8
    82'194 bytes - paq8pxd v32 -s4
    83'030 bytes - emma v24.1
    82'561 bytes - cmix v13e

    Darek

  10. #9
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts
    i have removed some experimental hash function so maybe it is more stable for jpeg, image24bpp n image8bpp..
    shox.jpg 446627 byte
    347045 simv2.9m -s6
    348519 paq8pxd_v32x64 -s6
    348226 paq8px_v103 -s6

    pia13872.ppm 2676237 byte
    250875 paq8pxd_v32x64 -s6
    219492 simv2.9m -s6
    253300 paq8px_v103

    nightshot_iso_1600.pgm 7375889 byte
    3434199 paq8pxd_v32x64 -s6
    3397538 simv2.9m -s6
    3434333 paq8px_v103 -s6
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	shox.jpg 
Views:	66 
Size:	436.2 KB 
ID:	5233  
    Attached Files Attached Files

  11. #10
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    520
    Thanks
    196
    Thanked 744 Times in 301 Posts
    Source code?

  12. #11
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    i have removed some experimental hash function so maybe it is more stable for jpeg, image24bpp n image8bpp..
    Yes, now looks more stable then I'll try to made tests on this version.

    One insight - still my 0.wav file is not compressed - just 179bytes of header are saved... - I've attach it.
    Second - time of images compression is very slow.

    first test - 1.BMP
    234'473 - sim v2.9m -s6
    232'475 - fpaq8 v6 -s6
    229'533 - paq8pxd v32 -s6
    228'261 - paq8px v104 -s6
    227'944 - cmix v13g
    225'879 - emma v24.1

    second test - A.TIF
    813'005 - cmix v13g, time 3'696.39s
    698'973 - emma v24.1, time 316.8s
    433'343 - fpaq8 v6 -s6, time 20.34s
    414'483 - paq8pxd v32 -s6, time 27.16s
    410'639 - paq8px v104 -s6, time 30.60s
    407'721 - sim v2.9m -s6, time 1'420.23s

    indeed - compression ratio for a.tif is the best I've seen. Compression time is 46-52 times longer than paq8px and pxd.

    Darek
    Attached Files Attached Files
    • File Type: 7z 0.7z (1.47 MB, 46 views)
    Last edited by Darek; 3rd September 2017 at 14:40.

  13. #12
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts
    thaxk you darek i will check it

  14. #13
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Here my testset scores for -s6 (more memory options I could test if I finish cmix tests). 0.WAV and L.PAK scores are estimated due to error mentioned above - L.PAK contains lots of wave parts which are inproper compressed as 0.WAV.

    In general - there are quite good image scores for the price even 50x slower compression for some files.
    General compression is about 3% worse in average than paqupxd v32.

    Total time of my testset compression (w/o mentioned audio files) = 9'963.24s. With proper audio codec time should be about 11'300s = 6.3x slower than paq8pxd, mainly due to very slow image compression.

    Darek
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	sim_v2_9.jpg 
Views:	60 
Size:	452.6 KB 
ID:	5237  

  15. #14
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    520
    Thanks
    196
    Thanked 744 Times in 301 Posts
    8bpp images:
    Code:
    nightshot_iso_1600.pgm
    3.350.261, EMMA 0.1.24.1 x86, Images (Slow)
    
    blood8.pgm
    1.571.081, simv2.9 -s6
    1.495.177, EMMA 0.1.24.1 x86, Images (Slow)
    
    catheter8.pgm
    207.411, simv2.9 -s6
    202.918, EMMA 0.1.24.1 x86, Images (Slow)
    
    fetus.pgm
    337.723, simv2.9 -s6
    338.756, EMMA 0.1.24.1 x86, Images (Slow)
    
    shoulder.pgm
    407.427, simv2.9 -s6
    381.741, EMMA 0.1.24.1 x86, Images (Slow)
    
    sigma.pgm
    1.597.469, simv2.9 -s6
    1.540.358, EMMA 0.1.24.1 x86, Images (Slow)
    
    lena.bmp (palette color)
    105.988, simv2.9 -s6
    105.596, EMMA 0.1.24.1 x86, Images (Slow)
    24bpp images:
    Code:
    pia13872.ppm
    194.926, EMMA 0.1.24.1 x86, Images (Slow), no color transform
    
    rafale.bmp
    512.523, 1553s, simv2.9 -s6, used 11119 MB of mem
    518.905, 31s, paq8px_v104 -8, used 1758 MB of mem
    496.141, 40s, EMMA v0.1.24.1 x86, Images (Slow)
    
    lena.bmp (24bpp RGB)
    413.768, 317s, simv2.9 -s6, used 11119 MB of mem
    410.905, 7s, paq8px_v104 -8, used 1758 MB of mem
    405.208, 9s, EMMA v0.1.24.1 x86, Images (Slow)
    
    baboon.bmp
    559.831, 316s, simv2.9 -s6, used 11119 MB of mem
    557.453, 8s, paq8px_v104 -8, used 1758 MB of mem
    550.339, 11s, EMMA v0.1.24.1 x86, Images (Slow)
    
    game.bmp (game screenshot)
    1.879.991, 2335s, simv2.9 -s6, used 11119 MB of mem
    1.898.788, 47s, paq8px_v104 -8, used 1758 MB of mem
    1.850.589, 64s, EMMA v0.1.24.1 x86, Images (Slow)
    JPEG:
    Code:
    mill.jpg
    4.976.580, 392s, simv2.9 -s6, used 14594 MB of mem
    4.979.542, 70s, paq8px_v104 -8, used 1852MB of mem
    
    A10.jpg
    628.788, 46s, simv2.9 -s6, used 14594 MB of mem
    630.208, 8.5s, paq8px_v104 -8, used 1852 MB of mem
    
    DSCN3974.jpg
    814.131, 59s, simv2.9 -s6, used 14594 MB of mem
    816.775, 11s, paq8px_v104 -8, used 1852 MB of mem
    
    DSCN4465.jpg
    513.773, 36s, simv2.9 -s6, used 14594 MB of mem
    514.060, 7s, paq8px_v104 -8, used 1852 MB of mem
    
    DSCN5081.jpg
    359.820, 29s, simv2.9 -s6, used 14594 MB of mem
    360.102, 5.5s, paq8px_v104 -8, used 1852 MB of mem
    
    Photo0040.jpg (4:1:1 subsampling)
    1.324.580, 85s, simv2.9 -s6, used 14594 MB of mem
    1.281.314, 16s, paq8px_v104 -8, used 1852 MB of mem
    It seems to be a clone of latest paq8pxd. The fixes we made to the JPEG model to account for subsampling seem to be broken.
    Judging by the huge slowdown and little compression gain, I'm guessing you just added lots and lots of model contexts and/or mixer contexts to those models,
    and upped the memory (level s7 is using over 23GB, s8 crashes immediately on a 64GB of RAM machine).

    Where can we get the source code?

  16. The Following User Says Thank You to mpais For This Useful Post:

    Stephan Busch (3rd September 2017)

  17. #15
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts
    i have fixed wav compression and this is the result
    0.wav
    1324494 -s6
    Attached Files Attached Files

  18. The Following User Says Thank You to suryakandau@yahoo.co.id For This Useful Post:

    Darek (6th September 2017)

  19. #16
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Did you change something in Image and default codec part?
    My scores for 1.BMP and I.EXE for -s6:

    234'473 - 1.BMP - previous score
    248'002 - 1.BMP - actual version score... worse than previous

    223'643 - I.EXE - previous score
    226'909 - I.EXE - actual version score... worse than previous

  20. #17
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    I've started to full test of sim v2.9m1. First insights:

    - there are very good results for wav files - better than both paq8pxd v34 and paq8px v112,
    - as I wrote above - majority of standard/default blocks files have worse results than previous sim v2.9 compile,
    - some image siles also - A.TIF got 9% worse result than previous sim v2.9 compile,
    - compression time overall looks like 5-6times slower than other paq8px variants but graphics files (f.e. A.TIF) time to compress is 200 times slower! - even this compression ratio is worse...
    - sometimes, occasionaly program hungs on some files but running it again helps and it's looks all files from my testset could be passed.

    Darek

  21. #18
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    And still there are crash on L.PAK file...

  22. #19
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts
    Quote Originally Posted by Darek View Post
    And still there are crash on L.PAK file...
    May you attach that file ?

  23. #20
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Yes. File attached.

    Second issue - for settings below -s6 (-s5, -s4) there are also crash for image files - however for -s6, -s7 and above everything looks OK. Example also attached.

    Third weird behaviour is a score for attached A.TIF file:
    443'913 -s6 score
    407'678 -s7 score... strange - it's too big difference for one step more in memory settings. All other files (like attached C.TIF) got only slightly different scores for -s6 and -s7 - as is for all other paq8 compressors...

    Darek
    Attached Files Attached Files
    • File Type: 7z L.7z (3.72 MB, 33 views)
    • File Type: 7z A.7z (1.34 MB, 37 views)
    • File Type: 7z C.7z (370.4 KB, 36 views)

  24. #21
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts
    Quote Originally Posted by Darek View Post
    Yes. File attached.

    Second issue - for settings below -s6 (-s5, -s4) there are also crash for image files - however for -s6, -s7 and above everything looks OK. Example also attached.

    Third weird behaviour is a score for attached A.TIF file:
    443'913 -s6 score
    407'678 -s7 score... strange - it's too big difference for one step more in memory settings. All other files (like attached C.TIF) got only slightly different scores for -s6 and -s7 - as is for all other paq8 compressors...

    Darek
    1.bmp 219316 byte -s6
    a.tif 388321 byte -s6
    c.tif 276625 byte -s6
    l.pak 2726105 byte -s6
    0.wav 1341163 byte -s6
    Attached Files Attached Files

  25. The Following User Says Thank You to suryakandau@yahoo.co.id For This Useful Post:

    Darek (24th September 2017)

  26. #22
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    1.bmp 219316 byte -s6
    a.tif 388321 byte -s6
    c.tif 276625 byte -s6
    l.pak 2726105 byte -s6
    0.wav 1341163 byte -s6
    simv2.9p -s6
    1.bmp 218698
    a.tif 387462
    c.tif 275980
    Attached Files Attached Files

  27. #23
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    1.bmp 219316 byte -s6
    a.tif 388321 byte -s6
    c.tif 276625 byte -s6
    l.pak 2726105 byte -s6
    0.wav 1341163 byte -s6
    Here are scores for my testbed for sim 2.9(m)b (first version), sim 2.9m1 and sim 2.9o.

    Generally version 2.9o is most stable - I could test all my testbed w/o estimates like for previous versions (red scores). But F.JPG file for -s5 and -s4 options still crash...

    This version have also implemented latest mpais and Jan Ondrus image improvements. However v2.9o have worse audio compression than v2.9m1 ...
    I've put comparison for previous version of sim and to paq8pxd v34.

    Strange is that for 0.WAV and L.PAK I've got different scores than you for -s6:

    0.wav - 1'341'335 - small but always...
    l.pak - 2'737'146 - bigger difference = 11KB...

    OK, now I understand - 0.wav and l.pak scores are from sim 2.9p version.

    Darek
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	sim29o.jpg 
Views:	37 
Size:	522.9 KB 
ID:	5354  
    Last edited by Darek; 25th September 2017 at 22:09.

  28. #24
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Scores for my testbed for sim 2.9p. Better compression for image files and some improvements for 0.WAV file and L.PAK.
    Images compression is now at the level of cmix v13l and for 1.BMP file there is a record for my testset!

    This version is also more stable but still there a crash for F.JPG file for -s4 option.

    Other options look ok, except fact that options s9 and above don't run due to very high memory requrements. However options s6, s7 and s8 besides declared high memory required - up to 41GB - they use much less memory in practice - up to 9-11GB.
    This version is also faster than previous by 15-20% due to faster image compression.

    Could you release source code with executables?
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	sim29p.jpg 
Views:	44 
Size:	499.6 KB 
ID:	5356  

  29. #25
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts
    Quote Originally Posted by Darek View Post
    Scores for my testbed for sim 2.9p. Better compression for image files and some improvements for 0.WAV file and L.PAK.
    Images compression is now at the level of cmix v13l and for 1.BMP file there is a record for my testset!

    This version is also more stable but still there a crash for F.JPG file for -s4 option.

    Other options look ok, except fact that options s9 and above don't run due to very high memory requrements. However options s6, s7 and s8 besides declared high memory required - up to 41GB - they use much less memory in practice - up to 9-11GB.
    This version is also faster than previous by 15-20% due to faster image compression.

    Could you release source code with executables?
    Hhmm maybe next year because I stiil have to make test for some files specially jpeg file..may you attach f.jpg mill.jpg & another jpeg file..thank you

  30. #26
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    Here you are f.jpg.
    Attached Files Attached Files
    • File Type: 7z F.7z (110.8 KB, 30 views)

  31. #27
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts
    Quote Originally Posted by Darek View Post
    Here are scores for my testbed for sim 2.9(m)b (first version), sim 2.9m1 and sim 2.9o.

    Generally version 2.9o is most stable - I could test all my testbed w/o estimates like for previous versions (red scores). But F.JPG file for -s5 and -s4 options still crash...

    This version have also implemented latest mpais and Jan Ondrus image improvements. However v2.9o have worse audio compression than v2.9m1 ...
    I've put comparison for previous version of sim and to paq8pxd v34.

    Strange is that for 0.WAV and L.PAK I've got different scores than you for -s6:

    0.wav - 1'341'335 - small but always...
    l.pak - 2'737'146 - bigger difference = 11KB...

    OK, now I understand - 0.wav and l.pak scores are from sim 2.9p version.

    Darek
    I don't implement latest mpais n Jan ondrus image..I just experiment with hash function..

  32. #28
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts
    sim v2.9q1 -s6
    a.tif 373354 bytes
    1.bmp 216511 bytes
    c.tif 273673 bytes
    f.jpg 81751 bytes
    Attached Files Attached Files

  33. The Following User Says Thank You to suryakandau@yahoo.co.id For This Useful Post:

    Darek (14th December 2017)

  34. #29
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    909
    Thanks
    531
    Thanked 359 Times in 267 Posts
    There are scores for my testset. There are really good scores for image files. For three files I've got absolute records for my testset - A.TIF, B.TGA and F.JPG.
    However for other file scores are mixed - sometimes much better, sometimes much worse....

    This version is also about 66% faster than v2.9p on my laptop. Setting is -s6. I can't test settings above -s6 now.

    Error1 - for both -s5 and -s4 there are crash for three files 1.bmp, a.tif and b.tga.
    Error2 - f.jpg after decompression have bad checksum and image isn't open. Other images after decompression are OK.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	sim_v29q1.jpg 
Views:	35 
Size:	508.1 KB 
ID:	5545  
    Last edited by Darek; 15th December 2017 at 02:05.

  35. #30
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    47
    Thanks
    3
    Thanked 7 Times in 7 Posts
    Quote Originally Posted by Darek View Post
    There are scores for my testset. There are really good scores for image files. For three files I've got absolute records for my testset - A.TIF, B.TGA and F.JPG.
    However for other file scores are mixed - sometimes much better, sometimes much worse....

    This version is also about 66% faster than v2.9p on my laptop. Setting is -s6. I can't test settings above -s6 now.

    Error1 - for both -s5 and -s4 there are crash for three files 1.bmp, a.tif and b.tga.
    Error2 - f.jpg after decompression have bad checksum and image isn't open. Other images after decompression are OK.
    Yes there is crash for s5 & s4, i will fix it...f.jpg decompress successfull in my laptop...

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •