Page 1 of 2 12 LastLast
Results 1 to 30 of 35

Thread: Huffmix: a PNGOUT -r catalyst

  1. #1
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts

    Huffmix: a PNGOUT/KZIP -r/-rn catalyst

    Hello,
    let me introduce you a new tool called Huffmix.

    As you probably know PNG and Gzip compression is based on Deflate.
    A Deflate stream is made of one or more blocks.
    PNGOUT/KZIP lets you choose how many blocks it produces (-n option, or -b to set the block split threshold).
    PNGOUT/KZIP also has a random switch (-r randomized initial tables "good for many trials with same options", this is -rn in KZIP).
    PNGSlim alike scripts used to run hundreds of random trials in a row and only kept the smallest PNG file produced (this takes a lot of time).

    Huffmix compares two files produced in random mode and cherry-picks the smallest blocks to combine them into a new file.

    For instance if we have 3 blocks.
    File A:
    block 1: 110 bits
    block 2: 317 bits
    block 3: 272 bits
    Overall: 699 bits

    File B:
    block 1: 104 bits
    block 2: 312 bits
    block 3: 289 bits
    Overall: 705 bits

    Until now optimization scripts would have kept File A and deleted File B (since A is smaller than B), Huffmix combines File A and File B into File C by picking the smallest blocks, this gives:

    File C:
    block 1: 104 bits (from File B)
    block 2: 312 bits (from File B)
    block 3: 272 bits (from File A)
    Overall: 688 bits

    We end up with a smaller file and managed to take profit of the calculations done to produce the biggest file, it works like a catalyst.
    Of course since there's randomness involved it will not always work as demonstrated, nevertheless my first tests suggests that it actually works pretty well.

    After about 20 cycles of pngout + deflopt + defluff + huffmix these 3 already optimized files are now a few bytes smaller:
    bigmac.png 33906 -> 33902 bytes
    getadrink.png 65317 -> 65293 bytes
    mouse.png 105035 -> 104866 bytes


    Huffmix 0.6b1 is available for testing.

    Usage: huffmix [options] file1.[png|gz] file2.[png|gz] output.[png|gz]
    Options:
    -v Verbose
    -q Quiet
    -k Keep same block sequence as in first file

    output can be the same as file1 or file2 (this is a fast and easy way to screw your file if there's a bug in Huffmix, you have been warned)

    Version 0.6b2
    - new option force
    - about 20% faster

    Version 0.6b1
    - solves some "Could not find matching sub-blocks" problems

    Version 0.5b4
    - internal number of Deflate blocks limit set to 1024
    - a memory allocation bug correction (led to rare segfaults or unexpected EOF)

    Version 0.5b2/b3
    - handles .gz files (gzip files created by kzip2gz)
    - internal number of Deflate blocks limit set to 600
    - a memory allocation bug correction (led to rare segfaults)

    Version 0.4b2
    - new -k and -q options
    - Huffmix has its own webpage : http://frdx.free.fr/huffmix/

    Version 0.4b1
    - no longer copies the input file if it's the same as the output.
    - allows you to combine files with different numbers of blocks (e.g. -n3 and -n5).
    Code:
     File Type C-Offset C-Length U-Offset U-Length
       A    2         0   289296        0   200724
       A    2    289296    11622    31014    46169
       A    2    300918    34023    3c46d    54457
    
     File Type C-Offset C-Length U-Offset U-Length
       B    2         0   110543        0    75987
       B    2    110543   104305    128d3    67393
       B    2    214848    75383    23014    57344
       B    2    290231    11608    31014    46169
       B    2    301839    34029    3c46d    54457
    
     File C-Offset C-Length
       A         0   289296
       B    290231    11608
       A    300918    34023
    
    IDAT new size 41872 (0xa390) bytes, saved 14 bits, output file size 41929 bytes
    Occasionally this will produce files that have different blocks compared to those produced by PNGOUT, use the -k option to preserve the block sequence.


    To ensure that PNGOUT will not modify filter or palette entries it has to be called with these options:
    pngout -ks -kp -f6

    There was a bug in PNGOUT in -f5 mode. Use the latest version of PNGOUT (May 2012 or February 2013) with Huffmix.

    For instance a simple Huffmix roundtrip looks like this:
    Code:
    pngout -force -ks -kp -f6 -r -y my_file.png my_temp_file.png
    defluff <my_temp_file.png >my_defluffed_temp_file.png
    huffmix my_file.png my_temp_file.png my_file.png
    huffmix my_file.png my_defluffed_temp_file.png my_file.png
    
    or
    
    kzip -rn -y my_temp_file.zip my_file
    kzip2gz my_temp_file.zip my_temp_file.gz
    defluff <my_temp_file.gz >my_defluffed_temp_file.gz
    huffmix my_file.gz my_temp_file.gz my_file.gz
    huffmix my_file.gz my_defluffed_temp_file.gz my_file.gz
    Start over again.

    Todo list:
    - FreeBSD version
    - Code clean up
    - Handle type zero blocks
    - Add support for full .zip files
    - Remove limitation on internal number of Deflate blocks (1024 since 0.5b4)
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	bigmac-hm.png 
Views:	216 
Size:	33.1 KB 
ID:	1591   Click image for larger version. 

Name:	getadrink-hm.png 
Views:	218 
Size:	63.8 KB 
ID:	1592   Click image for larger version. 

Name:	mouse-hm.png 
Views:	219 
Size:	102.4 KB 
ID:	1593  
    Attached Files Attached Files
    Last edited by caveman; 6th May 2014 at 12:22. Reason: 0.6b2 prerelease

  2. #2
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    50
    Thanks
    12
    Thanked 0 Times in 0 Posts
    The interesting idea, and is the version for Windows?

  3. #3
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,460
    Thanks
    8
    Thanked 37 Times in 27 Posts
    Todo list:
    - Mac OS X port
    - Windows port
    - Smarter file copying when nothing to do
    - Handle asymmetric blocks
    - Handle type zero blocks

  4. #4
    Member
    Join Date
    Oct 2009
    Location
    usa
    Posts
    36
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Results look good, and it is a very great application for those who want to squeeze the last bytes out of PNG files.

    How easy would it be to port the Linux version to DOS?

  5. #5
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts
    Quote Originally Posted by zyzzle View Post
    How easy would it be to port the Linux version to DOS?
    If by DOS you mean Windows NT/2000/XP/Vista/7 command line... it's available now.

  6. #6
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts
    As I wrote earlier I suspected a bug in PNGOUT when using the -f6 option, Ken Silverman was kind enough to check it out and he has found a bug in... -f5 mode.
    A new version of PNGOUT is available today (dated 07/02/2011).

    And before lorents17 shows up with smaller versions of the images I used, here is what I get after a few hours:
    bigmac.png 33902 -> 33900 bytes (2 bytes smaller)
    getadrink.png 65293 -> 65286 bytes (7 bytes smaller)
    mouse.png 104866 -> 104784 bytes (82 bytes smaller)
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	bigmac-hm3.png 
Views:	235 
Size:	33.1 KB 
ID:	1600   Click image for larger version. 

Name:	getadrink-hm3.png 
Views:	199 
Size:	63.8 KB 
ID:	1601   Click image for larger version. 

Name:	mouse-hm3.png 
Views:	205 
Size:	102.3 KB 
ID:	1602  
    Last edited by caveman; 3rd July 2011 at 02:35.

  7. #7
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts

    Pngdb a companion app to Huffmix

    pngdb has been updated and is now called defdb, it has its own thread here.

    I needed a tool to quickly display some internals of the Deflate stream to compare files produced by Huffmix. I make it available here since it could be helpful to others.

    For each Deflate block pngdb displays:
    - its type: 0 uncompressed, 1 fixed Huffman tree, 2 dynamic Huffman tree
    - its boundary: this is the offset in the uncompressed data where a block ends and a new one starts.
    - its compressed size in bits: self-explanatory
    and a short summary: full size of the stream in bits, number of blocks

    It looks like this:
    Code:
    T Boundary     Size
    2        0     3163
    2      dcc    16476
    2     2955    18865
    2     42e2    18606
    2     5b9a    18296
    2     7427     8514
    2     817b    14479
    98399 bits long, composed of 7 blocks
    It can also be called with two options:
    -s : displays only the overall Deflate stream size in bits
    -d : dumps the Deflate stream in human readable form:
    - [size] XX, a literal value XX in hexa (ranges from 00 to FF that is 0 to 255), took size bits compressed.
    - [size] (length, distance), LZ pair in decimal, length (3 to 25 distance (1 to 3276, took size bits compressed.
    - [size] EoB, End of Block, took size bits compressed.
    Huffman trees of type 2 blocks are not dumped (I may add this feature in the future), nor are type 0 blocks.
    It looks like this:
    Code:
     [3] 00
     [9] (3,1)
     [5] 1E
     [3] 00
    [11] (7,1)
     [5] 21
    [10] (8,9)
     [4] 40
     [4] 80
     [8] (6,9)
    ...
    [12] (9,432)
    [12] (9,450)
    [12] (9,468)
    [12] (9,486)
     [6] EoB
    Attached Files Attached Files
    Last edited by caveman; 10th December 2011 at 06:03. Reason: defdb is available

  8. #8
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts
    Version 0.4b1 released (first post updated).

  9. #9
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts
    Version 0.4b2 released (first post updated).

    The new -k option allows you to keep the same block sequence as in the first file.

    For instance you have a file with 3 blocks and an other one with 5 blocks, without the -k switch Huffmix could have produced an output file made of 3, 4 or 5 blocks, you can now enforce to keep the same block sequence as in the first file with the -k option.

    For instance if we have 3 and 5 blocks.
    File A:
    block 1: 110 bits
    block 2: 317 bits
    block 3: 272 bits
    Overall: 699 bits

    PNGOUT -n5 option has split block 2 and 3 in two.
    File B:
    block 1: 104 bits
    block 2: 201 bits (was part of block 2 in File A)
    block 3: 114 bits (was part of block 2 in File A) blocks 2&3 315 bits
    block 4: 148 bits (was part of block 3 in File A)
    block 5: 141 bits (was part of block 3 in File A) blocks 4&5 289 bits
    Overall: 708 bits

    Huffmix without -k outputs this file:
    File C:
    block 1: 104 bits (from File B)
    block 2: 201 bits (from File B since blocks 2&3 are smaller than block 2 from File A)
    block 3: 114 bits (from File B same as above)
    block 4: 272 bits (from File A since blocks 4&5 are bigger than block 3 from File A)
    Overall: 691 bits

    Huffmix with -k outputs this file:
    File D:
    block 1: 104 bits (from File B)
    block 2: 317 bits (from File A to preserve block sequence)
    block 3: 272 bits (from File A to preserve block sequence)
    Overall: 693 bits

    Jonathon Fowler has build the new version of PNGOUT for Linux, Mac OS X and FreeBSD.

    Using this version produced different block boundaries on some files leading to smaller files:
    bigmac.png 33900 -> 33864 bytes (36 bytes smaller)
    getadrink.png 65286 -> 65225 bytes (61 bytes smaller)
    mouse.png 104784 -> 104740 bytes (40 bytes smaller)
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	bigmac-new.png 
Views:	212 
Size:	33.1 KB 
ID:	1624   Click image for larger version. 

Name:	getadrink-new.png 
Views:	232 
Size:	63.7 KB 
ID:	1625   Click image for larger version. 

Name:	mouse-new.png 
Views:	230 
Size:	102.3 KB 
ID:	1626  
    Last edited by caveman; 27th July 2011 at 11:56. Reason: Ever-shrinking mouse latest news

  10. #10
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts
    Added Linux x86 64-bits versions of Huffmix and Pngdb.

  11. #11
    Member
    Join Date
    Oct 2012
    Location
    Dracula's country
    Posts
    21
    Thanks
    16
    Thanked 1 Time in 1 Post

  12. #12
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts
    Quote Originally Posted by Jaff View Post
    You can download it from the first message over-here.
    A new version supporting Gzip files is on its way.

  13. #13
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts
    Version 0.5b2 released (first post updated).

    Version 0.5b2
    - handles .gz files (gzip files created by kzip2gz)
    - internal number of Deflate blocks limit set to 600
    - a memory allocation bug correction (led to rare segfaults)

  14. #14
    Member
    Join Date
    Oct 2012
    Location
    Dracula's country
    Posts
    21
    Thanks
    16
    Thanked 1 Time in 1 Post
    Thank you for the update!

  15. #15
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts
    Windows version 0.5b2 has been replaced by 0.5b3 (to fix some unpexpected crash)

  16. #16
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    50
    Thanks
    12
    Thanked 0 Times in 0 Posts
    Good afternoon!
    Wanted to learn, whether zip support will be realized?

  17. #17
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts
    Sorry, full zip archive support has very low priority do not expect it this year, but some minor updates are in the works.

    Version 0.5b4 was silently released (it corrected a bug submitted by Aaron Reitz).
    Last edited by caveman; 14th May 2013 at 14:21.

  18. #18
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts
    Version 0.6b1 released (first post updated).
    - solves some "Could not find matching sub-blocks" problems
    Last edited by caveman; 14th May 2013 at 14:21.

  19. #19
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    448
    Thanks
    9
    Thanked 6 Times in 6 Posts
    would ti be possible to add multiple file support, so vi can find the best block from more files in one run would make it easier to batch.

  20. #20
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts
    Quote Originally Posted by SvenBent View Post
    would ti be possible to add multiple file support, so vi can find the best block from more files in one run would make it easier to batch.
    Had another similar request... I'm a bit bit busy right now but I'll try to add this feature.

  21. #21
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    448
    Thanks
    9
    Thanked 6 Times in 6 Posts
    I'm finally updating my png optimizing batch and implanting huffmix and i really love this utility.

    please tell me if im wrong in this.
    it would be possible to run multi instance of pngout /r and huffmix trials in parralel and then just huff mix the resulting files in the end
    e.g i run 4 batch files tha does 200 iterations of png /r and huffmix individually. and then the 4 resulting files i huffmix together. i would end up with the optimal files of block from all 800 pngout /r trials.

    or wold it be more efficient to run just one run of 800 trials ? i believe the compression ratio should be identical (except random is random) but am i wrong.?

    if it works huffmix makes it pretty easy to multi thread pngout /r trials this way
    Last edited by SvenBent; 22nd April 2014 at 07:28.

  22. #22
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts
    It should work, there's only one problem that could occur:
    The randomness is apparently based on time, if you fire two (or more) pngout instances exactly at the same time they may use the same random seed and produce exactly the same file. Now this has probably been mitigated by version 20120530 (and newer) of pngout, since on the Unix ports page it is stated : "Better randomness when the -r switch is used.", eventually ask Ken about it.

    Beware that palette ordering may change during a standard pngout -r run and since this produces an entirely new image huffmix will not work, you have to pick a palette first (truepng or pngout itself) and then lock it using -kp when doing the random runs.

    Don't forget to run deflopt and defluff (I run deflopt/defluff/deflopt again) right after pngout before huffmix.

    Quote Originally Posted by SvenBent View Post
    I'm finally updating my png optimizing batch and implanting huffmix and i really love this utility.

    please tell me if im wrong in this.
    it would be possible to run multi instance of pngout /r and huffmix trials in parralel and then just huff mix the resulting files in the end
    e.g i run 4 batch files tha does 200 iterations of png /r and huffmix individually. and then the 4 resulting files i huffmix together. i would end up with the optimal files of block from all 800 pngout /r trials.

    or wold it be more efficient to run just one run of 800 trials ? i believe the compression ratio should be identical (except random is random) but am i wrong.?

    if it works huffmix makes it pretty easy to multi thread pngout /r trials this way

  23. The Following User Says Thank You to caveman For This Useful Post:

    SvenBent (26th April 2014)

  24. #23
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    448
    Thanks
    9
    Thanked 6 Times in 6 Posts
    Wait deflopt before huffmix ? i used it after my pngout /r and huffmix trials

    Im actually integrating huff mix more and more into every stage possible. Even on trials on different block sizes cause its fast and just might give a few bytes more.
    it is fun seeing how a bigger files outpout from pngout still results in a smaller file in the end after huffmixing it

    Code:
     In:  908975 bytes               magicolor01.grey.png.thread1.png /c0 /f5 /d8
    Out:  909590 bytes               magicolor01.grey.png.tmp1.png /c0 /f6 /d8
    Chg:    +615 bytes (100% of original)
    IDAT new size 908787 (0xdddf3) bytes, saved 1047 bits, output file size 908844 b
    ytes
    Last edited by SvenBent; 26th April 2014 at 03:49.

  25. #24
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    448
    Thanks
    9
    Thanked 6 Times in 6 Posts
    i just did some trials of using deflopt in the png out /r mixing or just waiting and do deflopt at the end.
    Im using the kodak testsuit but missing pictures 8 (I messed it up)
    however this is just one interation of pngout /r + pngout of the original. i copied the /r output file and made an extra deflopted version of it. so all in all i had 3 starting files

    Original
    /r
    /r deflopted

    Then i mixed "original" + "/r" into mixedNO
    Then i mixed "original" + "/r deflopted" into mixdefl

    i wrote down the sizes

    Then i deflopted all the mixedno ("original" + "/r") and wrote their size as "deflopt at end"
    I deflopted all the mixedelf ("original" + "/r deflopt") again and wrote down their size as "Deflopted mixed +end"

    funny part is here the one that is only deflopted at the end is actually smaller in average than the ones deflopted both inside the mixing and at the end.
    Lastly i just mixed the "deflopt end" + "Deflopt mixed +end"


    the sizes :



    Code:
    MIXED NO DEFLOPT         14,252,970 bytes
    
    MIXED WITH DEFLOPT       14,252,719 bytes
    DEFLOPT AT END           14,252,443 bytes
    DEFLOPT MIXED+END        14,252,456 bytes
    
    Mixed of END/mixed+end   14,252,441 bytes
    
    
    MIXED NO DEFLOPT
    04/25/2014  20:12           694,608 kodim01.png.mixedno.png
    04/25/2014  20:12           607,858 kodim02.png.mixedno.png
    04/25/2014  20:12           482,799 kodim03.png.mixedno.png
    04/25/2014  20:12           621,624 kodim04.png.mixedno.png
    04/25/2014  20:12           758,955 kodim05.png.mixedno.png
    04/25/2014  20:12           604,207 kodim06.png.mixedno.png
    04/25/2014  20:12           556,755 kodim07.png.mixedno.png
    04/25/2014  20:12           561,963 kodim09.png.mixedno.png
    04/25/2014  20:12           569,433 kodim10.png.mixedno.png
    04/25/2014  20:12           600,153 kodim11.png.mixedno.png
    04/25/2014  20:12           546,449 kodim12.png.mixedno.png
    04/25/2014  20:12           803,982 kodim13.png.mixedno.png
    04/25/2014  20:12           694,534 kodim14.png.mixedno.png
    04/25/2014  20:12           595,772 kodim15.png.mixedno.png
    04/25/2014  20:12           532,702 kodim16.png.mixedno.png
    04/25/2014  20:12           597,972 kodim17.png.mixedno.png
    04/25/2014  20:12           754,566 kodim18.png.mixedno.png
    04/25/2014  20:12           637,207 kodim19.png.mixedno.png
    04/25/2014  20:12           475,903 kodim20.png.mixedno.png
    04/25/2014  20:12           636,224 kodim21.png.mixedno.png
    04/25/2014  20:12           689,341 kodim22.png.mixedno.png
    04/25/2014  20:12           552,015 kodim23.png.mixedno.png
    04/25/2014  20:12           677,948 kodim24.png.mixedno.png
    MIXED NO DEFLOPT         14,252,970 bytes
    
    
    
    MIXED WITH DEFLOPT
    04/25/2014  20:14           694,591 kodim01.png.mixdelf.png
    04/25/2014  20:14           607,845 kodim02.png.mixdelf.png
    04/25/2014  20:14           482,799 kodim03.png.mixdelf.png
    04/25/2014  20:14           621,615 kodim04.png.mixdelf.png
    04/25/2014  20:14           758,952 kodim05.png.mixdelf.png
    04/25/2014  20:14           604,199 kodim06.png.mixdelf.png
    04/25/2014  20:14           556,743 kodim07.png.mixdelf.png
    04/25/2014  20:14           561,936 kodim09.png.mixdelf.png
    04/25/2014  20:14           569,422 kodim10.png.mixdelf.png
    04/25/2014  20:14           600,148 kodim11.png.mixdelf.png
    04/25/2014  20:14           546,437 kodim12.png.mixdelf.png
    04/25/2014  20:14           803,968 kodim13.png.mixdelf.png
    04/25/2014  20:14           694,526 kodim14.png.mixdelf.png
    04/25/2014  20:14           595,752 kodim15.png.mixdelf.png
    04/25/2014  20:14           532,688 kodim16.png.mixdelf.png
    04/25/2014  20:14           597,955 kodim17.png.mixdelf.png
    04/25/2014  20:14           754,566 kodim18.png.mixdelf.png
    04/25/2014  20:14           637,180 kodim19.png.mixdelf.png
    04/25/2014  20:14           475,891 kodim20.png.mixdelf.png
    04/25/2014  20:14           636,212 kodim21.png.mixdelf.png
    04/25/2014  20:14           689,336 kodim22.png.mixdelf.png
    04/25/2014  20:14           552,011 kodim23.png.mixdelf.png
    04/25/2014  20:14           677,947 kodim24.png.mixdelf.png
    MIXED WITH DEFLOPT       14,252,719 bytes
    
    
    
    DEFLOPT AT END
    04/25/2014  20:19           694,583 kodim01.png.mixedno.png
    04/25/2014  20:19           607,837 kodim02.png.mixedno.png
    04/25/2014  20:19           482,787 kodim03.png.mixedno.png
    04/25/2014  20:19           621,595 kodim04.png.mixedno.png
    04/25/2014  20:19           758,941 kodim05.png.mixedno.png
    04/25/2014  20:19           604,167 kodim06.png.mixedno.png
    04/25/2014  20:19           556,736 kodim07.png.mixedno.png
    04/25/2014  20:19           561,928 kodim09.png.mixedno.png
    04/25/2014  20:19           569,401 kodim10.png.mixedno.png
    04/25/2014  20:19           600,126 kodim11.png.mixedno.png
    04/25/2014  20:19           546,435 kodim12.png.mixedno.png
    04/25/2014  20:19           803,960 kodim13.png.mixedno.png
    04/25/2014  20:19           694,514 kodim14.png.mixedno.png
    04/25/2014  20:19           595,729 kodim15.png.mixedno.png
    04/25/2014  20:19           532,686 kodim16.png.mixedno.png
    04/25/2014  20:19           597,949 kodim17.png.mixedno.png
    04/25/2014  20:19           754,537 kodim18.png.mixedno.png
    04/25/2014  20:19           637,178 kodim19.png.mixedno.png
    04/25/2014  20:19           475,886 kodim20.png.mixedno.png
    04/25/2014  20:19           636,198 kodim21.png.mixedno.png
    04/25/2014  20:19           689,323 kodim22.png.mixedno.png
    04/25/2014  20:19           552,007 kodim23.png.mixedno.png
    04/25/2014  20:19           677,940 kodim24.png.mixedno.png
    DEFLOPT AT END           14,252,443 bytes
    
    
    
    DEFLOPT MIXED+END
    04/25/2014  20:24           694,583 kodim01.png.mixdelf.png
    04/25/2014  20:24           607,837 kodim02.png.mixdelf.png
    04/25/2014  20:24           482,787 kodim03.png.mixdelf.png
    04/25/2014  20:24           621,595 kodim04.png.mixdelf.png
    04/25/2014  20:24           758,942 kodim05.png.mixdelf.png
    04/25/2014  20:24           604,170 kodim06.png.mixdelf.png
    04/25/2014  20:24           556,739 kodim07.png.mixdelf.png
    04/25/2014  20:24           561,928 kodim09.png.mixdelf.png
    04/25/2014  20:24           569,401 kodim10.png.mixdelf.png
    04/25/2014  20:24           600,126 kodim11.png.mixdelf.png
    04/25/2014  20:24           546,436 kodim12.png.mixdelf.png
    04/25/2014  20:24           803,958 kodim13.png.mixdelf.png
    04/25/2014  20:24           694,514 kodim14.png.mixdelf.png
    04/25/2014  20:24           595,735 kodim15.png.mixdelf.png
    04/25/2014  20:24           532,686 kodim16.png.mixdelf.png
    04/25/2014  20:24           597,949 kodim17.png.mixdelf.png
    04/25/2014  20:24           754,537 kodim18.png.mixdelf.png
    04/25/2014  20:24           637,178 kodim19.png.mixdelf.png
    04/25/2014  20:24           475,887 kodim20.png.mixdelf.png
    04/25/2014  20:24           636,198 kodim21.png.mixdelf.png
    04/25/2014  20:24           689,323 kodim22.png.mixdelf.png
    04/25/2014  20:24           552,007 kodim23.png.mixdelf.png
    04/25/2014  20:24           677,940 kodim24.png.mixdelf.png
    DEFLOPT MIXED+END        14,252,456 bytes
    
    
    
    Mixed of END/mixed+end
    04/25/2014  20:29           694,583 kodim01.png.end.png
    04/25/2014  20:29           607,837 kodim02.png.end.png
    04/25/2014  20:29           482,787 kodim03.png.end.png
    04/25/2014  20:29           621,595 kodim04.png.end.png
    04/25/2014  20:29           758,941 kodim05.png.end.png
    04/25/2014  20:29           604,167 kodim06.png.end.png
    04/25/2014  20:29           556,736 kodim07.png.end.png
    04/25/2014  20:29           561,928 kodim09.png.end.png
    04/25/2014  20:29           569,401 kodim10.png.end.png
    04/25/2014  20:29           600,126 kodim11.png.end.png
    04/25/2014  20:29           546,435 kodim12.png.end.png
    04/25/2014  20:29           803,958 kodim13.png.end.png
    04/25/2014  20:29           694,514 kodim14.png.end.png
    04/25/2014  20:29           595,729 kodim15.png.end.png
    04/25/2014  20:29           532,686 kodim16.png.end.png
    04/25/2014  20:29           597,949 kodim17.png.end.png
    04/25/2014  20:29           754,537 kodim18.png.end.png
    04/25/2014  20:29           637,178 kodim19.png.end.png
    04/25/2014  20:29           475,886 kodim20.png.end.png
    04/25/2014  20:29           636,198 kodim21.png.end.png
    04/25/2014  20:29           689,323 kodim22.png.end.png
    04/25/2014  20:29           552,007 kodim23.png.end.png
    04/25/2014  20:29           677,940 kodim24.png.end.png
    Mixed of END/mixed+end   14,252,441 bytes
    This results of this test is that it seems to be better to not use deflopt inside the mixing compared to just do it at the end. Unless you again remix everything.

    Sadly im off to work. ima make a batch run over more files with several runs now that i found my method to compare it
    Last edited by SvenBent; 26th April 2014 at 05:52.

  26. #25
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    172
    Thanks
    6
    Thanked 36 Times in 22 Posts
    Yes you should use deflopt before huffmix and you should even use option -b (saves the file even if could not save an entire byte but just a few bits).

    Usually I do not rely on threshold sizes but number of blocks -n, since different thresholds could lead to the same number of blocks, thresholds may be interesting for large files.
    When huffmixing files with different numbers of blocks always huffmix those holding more blocks first, for instance if you produced files holding between 2 and 8 blocks using the random option, huffmix the files holding 8 and 7 blocks first and then the one with 6 blocks and so on.

    Quote Originally Posted by SvenBent View Post
    Wait deflopt before huffmix ? i used it after my pngout /r and huffmix trials

    Im actually integrating huff mix more and more into every stage possible. Even on trials on different block sizes cause its fast and just might give a few bytes more.

  27. #26
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    448
    Thanks
    9
    Thanked 6 Times in 6 Posts
    yeah i just came back to report i didint use /b switch whcih would probably be better.

    ima set up a batch run just to do some testing that cna run while im at work. ty for feedback

  28. #27
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    448
    Thanks
    9
    Thanked 6 Times in 6 Posts
    ok here are some testing of trucolor images (next up is gray tone... just because) of whatever to use deflopt at the end or in every iteration og pngout /r

    Image Source: RGB8bit form http://www.imagecompression.info/test_images/
    Method:
    I first made 140 iterations of each file with pngout /r /f6 /force
    sadly i discovered that my original was not mix compatible with my pngout iterations so i removed the original and recopied files form iteration 1 and used as "original'"

    I then mixed original with iteration N into original,for the mixing with no deflopt results.
    i did this in sets of 35 iterations and noted down results
    The resulting files got save in a new MixedNO folder

    I recopied files from iteration 1 into the spot for original files thereby having the same original starting files
    I then did a new batch that deflopted iteration N. Then mixed Original with Iteration N into original files.
    I again did this test in sets of 35 and noted down the results
    The resulting files was copied into folder MixedWith.

    In mixedNO and Mixed with folder i created a subfolder called DefloptedEND.
    I copied the files from each folder into there respective DefloptedEND subfolder
    ran deflopt on the files in both DefloptedEND folder

    This proces was to insure that there was no different between the /r iterations from each test stride. as different between test strides could affect the and scew the result
    The defloptedEND files of mixedWITH did no decrease in size and is therefore no in the info below
    The reason i noted down after each 35 iterations was because i wanted to compared them in subsets but i forgot to deflopt the MixedNO files so i couldn't compared it in the end anyway


    Code:
    Deflopted 140Trials     227,965,129 bytes
    Deflopted at the end    227,968,065 bytes
    
    
    No Deflopt 35trials
    04/29/2014  01:46         1,493,757 artificial.png
    04/29/2014  01:50        62,439,821 big_building.png
    04/29/2014  01:53        47,827,971 big_tree.png
    04/29/2014  01:55        19,694,512 bridge.png
    04/29/2014  01:56         9,311,583 cathedral.png
    04/29/2014  01:57        21,153,221 deer.png
    04/29/2014  01:58         5,398,142 fireworks.png
    04/29/2014  01:58         2,989,063 flower_foveon.png
    04/29/2014  01:58         6,823,395 hdr.png
    04/29/2014  01:59        11,270,035 leaves_iso_1600.png
    04/29/2014  02:00        10,139,929 leaves_iso_200.png
    04/29/2014  02:00         7,418,650 nightshot_iso_100.png
    04/29/2014  02:01        12,385,190 nightshot_iso_1600.png
    04/29/2014  02:02         9,840,290 spider_web.png
    No Deflopt 35trials     228,185,559 bytes
    
    
    No Deflopt 70trials
    04/29/2014  02:43         1,493,640 artificial.png
    04/29/2014  02:47        62,409,724 big_building.png
    04/29/2014  02:50        47,827,407 big_tree.png
    04/29/2014  02:51        19,680,829 bridge.png
    04/29/2014  02:52         9,311,384 cathedral.png
    04/29/2014  02:53        21,153,212 deer.png
    04/29/2014  02:54         5,396,662 fireworks.png
    04/29/2014  02:54         2,988,037 flower_foveon.png
    04/29/2014  02:54         6,819,666 hdr.png
    04/29/2014  02:55        11,257,206 leaves_iso_1600.png
    04/29/2014  02:55        10,123,450 leaves_iso_200.png
    04/29/2014  02:56         7,414,917 nightshot_iso_100.png
    04/29/2014  02:57        12,385,148 nightshot_iso_1600.png
    04/29/2014  02:57         9,830,673 spider_web.png
    No Deflopt 70trials     228,091,955 bytes
    
    
    No deflopt 105trials
    04/29/2014  03:49         1,493,519 artificial.png
    04/29/2014  03:53        62,405,606 big_building.png
    04/29/2014  03:55        47,827,360 big_tree.png
    04/29/2014  03:57        19,680,820 bridge.png
    04/29/2014  03:58         9,311,138 cathedral.png
    04/29/2014  03:59        21,153,204 deer.png
    04/29/2014  04:00         5,394,256 fireworks.png
    04/29/2014  04:00         2,986,561 flower_foveon.png
    04/29/2014  04:00         6,818,073 hdr.png
    04/29/2014  04:01        11,257,157 leaves_iso_1600.png
    04/29/2014  04:02        10,122,894 leaves_iso_200.png
    04/29/2014  04:02         7,412,697 nightshot_iso_100.png
    04/29/2014  04:03        12,385,099 nightshot_iso_1600.png
    04/29/2014  04:04         9,824,101 spider_web.png
    No deflopt 105trials    228,072,485 bytes
    
    
    No deflopt 140trials
    04/29/2014  04:54         1,493,429 artificial.png
    04/29/2014  04:57        62,324,746 big_building.png
    04/29/2014  05:01        47,827,325 big_tree.png
    04/29/2014  05:03        19,680,816 bridge.png
    04/29/2014  05:04         9,311,060 cathedral.png
    04/29/2014  05:04        21,153,203 deer.png
    04/29/2014  05:05         5,393,544 fireworks.png
    04/29/2014  05:06         2,985,768 flower_foveon.png
    04/29/2014  05:06         6,813,635 hdr.png
    04/29/2014  05:07        11,257,136 leaves_iso_1600.png
    04/29/2014  05:07        10,111,691 leaves_iso_200.png
    04/29/2014  05:08         7,408,903 nightshot_iso_100.png
    04/29/2014  05:09        12,385,083 nightshot_iso_1600.png
    04/29/2014  05:09         9,822,529 spider_web.png
    No deflopt 140trials    227,968,868 bytes
    
    
    Deflopted at the end
    04/29/2014  16:00         1,493,387 artificial.png
    04/29/2014  16:00        62,324,566 big_building.png
    04/29/2014  16:00        47,827,324 big_tree.png
    04/29/2014  16:00        19,680,795 bridge.png
    04/29/2014  16:00         9,311,052 cathedral.png
    04/29/2014  16:00        21,153,202 deer.png
    04/29/2014  16:00         5,393,384 fireworks.png
    04/29/2014  16:00         2,985,690 flower_foveon.png
    04/29/2014  16:00         6,813,582 hdr.png
    04/29/2014  05:07        11,257,136 leaves_iso_1600.png
    04/29/2014  16:00        10,111,687 leaves_iso_200.png
    04/29/2014  16:00         7,408,841 nightshot_iso_100.png
    04/29/2014  16:00        12,385,027 nightshot_iso_1600.png
    04/29/2014  16:00         9,822,392 spider_web.png
    Deflopted at the end    227,968,065 bytes
    
    
    ---  ---  ---  ---  ---
    
    
    Deflopted 35Trials
    04/29/2014  05:54         1,493,713 artificial.png
    04/29/2014  05:59        62,439,610 big_building.png
    04/29/2014  06:03        47,827,485 big_tree.png
    04/29/2014  06:05        19,694,485 bridge.png
    04/29/2014  06:06         9,311,430 cathedral.png
    04/29/2014  06:08        21,153,220 deer.png
    04/29/2014  06:09         5,397,968 fireworks.png
    04/29/2014  06:09         2,989,029 flower_foveon.png
    04/29/2014  06:09         6,823,290 hdr.png
    04/29/2014  06:11        11,270,035 leaves_iso_1600.png
    04/29/2014  06:11        10,139,796 leaves_iso_200.png
    04/29/2014  06:12         7,418,617 nightshot_iso_100.png
    04/29/2014  06:13        12,385,118 nightshot_iso_1600.png
    04/29/2014  06:14         9,840,180 spider_web.png
    Deflopted 35Trials      228,183,976 bytes
    
    
    Deflopted 70Trials
    04/29/2014  12:37         1,493,595 artificial.png
    04/29/2014  12:42        62,409,505 big_building.png
    04/29/2014  12:46        47,827,288 big_tree.png
    04/29/2014  12:48        19,680,807 bridge.png
    04/29/2014  12:49         9,311,243 cathedral.png
    04/29/2014  12:50        21,153,211 deer.png
    04/29/2014  12:52         5,396,508 fireworks.png
    04/29/2014  12:52         2,988,011 flower_foveon.png
    04/29/2014  12:53         6,819,188 hdr.png
    04/29/2014  12:54        11,257,206 leaves_iso_1600.png
    04/29/2014  12:55        10,123,446 leaves_iso_200.png
    04/29/2014  12:56         7,414,882 nightshot_iso_100.png
    04/29/2014  12:57        12,385,082 nightshot_iso_1600.png
    04/29/2014  12:58         9,830,574 spider_web.png
    Deflopted 35Trials      228,090,546 bytes
    
    
    Deflopted 105Trials
    04/29/2014  13:01         1,493,473 artificial.png
    04/29/2014  13:06        62,402,734 big_building.png
    04/29/2014  13:10        47,827,242 big_tree.png
    04/29/2014  13:12        19,680,798 bridge.png
    04/29/2014  13:13         9,311,105 cathedral.png
    04/29/2014  13:15        21,153,203 deer.png
    04/29/2014  13:16         5,394,124 fireworks.png
    04/29/2014  13:16         2,986,532 flower_foveon.png
    04/29/2014  13:17         6,817,959 hdr.png
    04/29/2014  13:18        11,257,157 leaves_iso_1600.png
    04/29/2014  13:18        10,122,890 leaves_iso_200.png
    04/29/2014  13:19         7,412,657 nightshot_iso_100.png
    04/29/2014  13:20        12,385,033 nightshot_iso_1600.png
    04/29/2014  13:21         9,823,992 spider_web.png
    Deflopted 105Trials     228,068,899 bytes    
    
    
    Deflopted 140Trials 
    04/29/2014  15:35         1,493,386 artificial.png
    04/29/2014  15:40        62,321,765 big_building.png
    04/29/2014  15:44        47,827,207 big_tree.png
    04/29/2014  15:46        19,680,795 bridge.png
    04/29/2014  15:47         9,311,047 cathedral.png
    04/29/2014  15:49        21,153,202 deer.png
    04/29/2014  15:50         5,393,384 fireworks.png
    04/29/2014  15:50         2,985,690 flower_foveon.png
    04/29/2014  15:51         6,813,580 hdr.png
    04/29/2014  15:51        11,257,136 leaves_iso_1600.png
    04/29/2014  15:53        10,111,687 leaves_iso_200.png
    04/29/2014  15:54         7,408,841 nightshot_iso_100.png
    04/29/2014  15:55        12,385,018 nightshot_iso_1600.png
    04/29/2014  15:55         9,822,391 spider_web.png
    Deflopted 140Trials     227,965,129 bytes
    Im about to relook into the files and try find the individuelle smallest files from each iteration to compared how much Huffmix increased the efficiency of doing pngout/r trials.
    I just need to find an easy way to copy the smallest files from withing 140 sub folders
    Last edited by SvenBent; 30th April 2014 at 01:44.

  29. #28
    Member chornobyl's Avatar
    Join Date
    May 2008
    Location
    ua/kiev
    Posts
    153
    Thanks
    0
    Thanked 0 Times in 0 Posts
    If files have same names, totalcommander can do the task.

  30. #29
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    448
    Thanks
    9
    Thanked 6 Times in 6 Posts
    TY but i figure out i didn need the file just the filesize so a copule of simple dirs and idid it manually. im just to lazy atm to get back to my main rig and write the results

  31. #30
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    448
    Thanks
    9
    Thanked 6 Times in 6 Posts
    Code:
    NO MIXING
    04/29/2014  15:35         1,495,944 artificial.140.png
    04/29/2014  15:39        62,691,953 big_building.129.png
    04/29/2014  15:42        47,830,297 big_tree.118.png
    04/29/2014  12:46        19,711,142 bridge.39.png
    04/29/2014  15:47         9,330,023 cathedral.138.png
    04/29/2014  06:07        21,153,359 deer.11.png
    04/29/2014  06:08         5,414,884 fireworks.30.png
    04/29/2014  15:50         3,000,477 flower_foveon.107.png
    04/29/2014  13:17         6,839,930 hdr.105.png
    04/29/2014  13:17        11,272,024 leaves_iso_1600.89.png
    04/29/2014  15:53        10,159,599 leaves_iso_200.130.png
    04/29/2014  13:19         7,447,499 nightshot_iso_100.99.png
    04/29/2014  15:54        12,385,325 nightshot_iso_1600.108.png
    04/29/2014  13:21         9,897,267 spider_web.96.png
                             228,629,723 bytes
    Deflopted 140Trials 227,965,129 bytes
    Deflopted at the end 227,968,065 bytes
    NO Huffmix 228,629,723 bytes

    adding in huffmix in 140 trials of pngout /r gave an adition reduction of around 645kb. with very little time increase.

    im doing a new test with 8 bit gray tones.
    however this time ima look at time efficiency as well
    Last edited by SvenBent; 3rd May 2014 at 02:42.

Page 1 of 2 12 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •