Results 1 to 6 of 6

Thread: PngBest.bat Multithreade Fast/extrme png optizmizing

  1. #1
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts

    PngBest.bat Multithreade Fast/extrme png optizmizing

    I'm just putting up my pngbest.bat file for people to use

    it sway away from other extreme batches in a couple of ways

    1. Tuned for speed and extreme compression.
    2. Customizable/Scalable compression strength.
    3. Multithreaded
    4. Simple heuristic to avoid unneeded work
    5. Modes for just extra huffman mixing or enable Palette expansion trials



    1. It has been tuned for speed and performance in the following image sets

    True Color Kodak Images
    The New Test Images - Image Compression Benchmark RGB 8 bit
    The New Test Images - Image Compression Benchmark Grey 8 bit

    Any tools or method that did not improve compression was removed (OptiPNG, AdvPNG, lower compression strength of PNGout).


    2. its Scalable.
    you can select the Strength of compression applied with a simple syntax

    PngBest.bat Somefile.png <Strenght level 1-?>

    The strength level determines thee amount of zopfli trials to use in PNGwolf and the amount of PNGout /r trials to run with block mixing.
    As defaults Zopfli compression is set to 15 and PNGout /r trials is set to 0 iterations.
    Please note due to how multithreaded is handled atm. the actual PNGout/r trials are rounded down to nearest multiple of cpu cores.


    3. Multithreaded
    This batch file will automatically detect the numbers of cores and spread out the PNGout /r trials equally among you cpu cores
    The first couple of stages are not multithreaded yet. I will probably rework the first couple of stages to implant multithreading and improve compression as well


    4. Simple heuristic to avoid unneeded work
    The bath files will automatically skip the entire PNGout /r trials if it detects its going to be unhelpful
    It's a very basic detection that just simply check if the PNGout filter and block threshold size gave any size improvements.
    if they didn't the batch will assume the PNGwolf stage made it to small and skip the PNGout /r trial all together.
    This got implanted because it was not possible to HuffMix the output files from PNGwolf file with the one from PNGout, so the batch wasted to much i time just making useless PNGout /r trials
    This also make it skip PNGout /r trials on files that was previously optimized by PngBest.bat

    This behavior might be reworked again later or at least get a disable switch.


    5. Modes for just extra huffman mixing or enable Palette expansion trials
    A third parameter chooses some more specific work modes

    PngBest.bat SomeFile.png <Strenght 1-?> <Mode>

    Modes supported:
    Code:
    pal    = Expanding palette mode. This will try to expand the palette size to 8bits/entry to possible improve compression
    brutal = Does the PNGout Filter and block threshold by brute force instead of smart method (temporarily disabled until this is multithreaded)
    mixing = Skip all previous stages and goes directly to PNGout /r trials with HuffMix. Good for working on a previously optimized file (disables the skip heuristic)
    Please note all modes has to written in lowercase exactly as above



    TODO list:
    - Multithread Pngout filter/block threshold stage (waiting for caveman on this one if he makes the -f flag work in huffmix)
    - Adjust PNGout /r trials count so it gets rounded up instead of down when dividing it among cpu cores
    - Test if Implanting Defluff into the PNGout /r trials improves compression/time efficiency ( Completed. It does help)
    - Change from Block threshold to number of blocks
    - Implant further Palette optimizing methods


    Files needed
    PNGwolf (Zopfli version)
    PNGout
    Defluff
    Deflopt
    Huffmix

    Code:
    @ECHO OFF
    set strenght=%2
    if "%2"=="" set strenght=15
    set RandIt=0
    set /a RandIt=(%2/%NUMBER_OF_PROCESSORS%)
    set /a TotalIt=%RandIt%"*%NUMBER_OF_PROCESSORS%
    
    ECHO --- SvenBent's PNGbest is optimizing your PNG file
    ECHO CPU threads: %NUMBER_OF_PROCESSORS%
    ECHO Zopfli compression strenght: %strenght%
    ECHO Random initial tables : %TotalIt%
    if "%3"=="brutal" ECHO Extended brute force: Enabled (not supported)
    if "%3"=="mixing" ECHO Mixing only mode: Enabled
    if "%3"=="pal" ECHO Expanding palette size: Enabled (use for less than 8bits pallette images)
    ECHO File: %1
    ECHO Size: %~z1
    
    if "%3"=="mixing" Goto Mixing
    
    
    
    ECHO --- PNGwolf-Zopfli ---
    pngwolf --in=%1 --out=%1 --zopfli-iterations=%strenght%
    
    set wolfsize=%~z1
    
    ECHO --- Finding optimal filter ---
    for %%f in (6,0,1,2,3,4,5) do pngout %1 /f%%f /d0
    if "%3"=="pal" for %%f in (6,0,1,2,3,4,5) do pngout %1 /f%%f /d8
    
    
    
    
    ECHO --- Block split threshold ---
    for %%b in (64,96,128,192,384,512,768,1024,1536,2048,3072,4096,8192,16384,0) do pngout %1 /f6 /b%%b
    
    
    
    if "%~z1"=="%wolfsize%" goto norandom
    if "%2"=="" goto norandom
    
    
    
    :Mixing
    ECHO --- Random initial tables and mixing blocks ---
    
    echo   @ECHO OFF >%1.pngmix.bat
    echo   for /l %%%%r in (1,1,%RandIt%) do ( >>%1.pngmix.bat
    echo   ECHO --- Trial: %%%%r of %RandIt% --- >>%1.pngmix.bat
    echo   pngout  %1.thread.%%1.png %1.tmp.%%1.png /r /force /f6 /kp >>%1.pngmix.bat
    echo   Deflopt /b /s %1.tmp.%%1.png >>%1.pngmix.bat
    echo   huffmix %1.thread.%%1.png %1.tmp.%%1.png %1.thread.%%1.png  >>%1.pngmix.bat
    echo   del %1.tmp.%%1.png  >>%1.pngmix.bat
    echo )  >>%1.pngmix.bat
    echo del %1.%%1.tag >>%1.pngmix.bat
    echo exit >>%1.pngmix.bat
    
    for /l %%t in (1,1,%NUMBER_OF_PROCESSORS%) do (
    Copy %1 %1.thread.%%t.png
    echo tag > %1.%%t.tag
    start "Thread%%t" %1.PNGmix.bat %%t
    )
    
    ECHO Waiting for threads to close
    :keepwait
    timeout 1 >nul
    if exist %1.?.tag goto keepwait
    if exist %1.??.tag goto keepwait
    if exist %1.???.tag goto keepwait
    if exist %1.????.tag goto keepwait
    
    ECHO --- Final mixing stage ---
    for /l %%t in (1,1,%NUMBER_OF_PROCESSORS%) do (
    huffmix %1 %1.thread.%%t.png %1
    del %1.thread.%%t.png
    )
    del %1.pngmix.bat
    
    :norandom
    
    
    ECHO --- Optimizing deflate structure ---
    deflopt /b %1
    defluff <%1 >%1.fluff.png
    deflopt /b %1.fluff.png
    huffmix %1 %1.fluff.png %1
    deflopt /b %1
    del %1.fluff.png

    P.S. if you are running on an older windows you might need to replaced the line
    timeout 1 >nul
    with
    Ping 127.0.0.1 >nul



    -- Update --

    New Version

    - Minor improvement of PNGout handling
    Now it compares the PNGwolf output size vs the PNGout output size AFTER all the out filters/block threshold are done.
    Before it would compare it after each PNGout iteration and could therefore lead to missing filter/threshold combos that would results in smaller files

    - Minor improvement to auto skip of mixing stage
    If the PNGout files is equal to the PNGwolf output size it will now continue with the mixing stage. Before it would skip the mixing stage
    It still skips this stage if the PNGwolf or the original size is smaller than the PNGout file

    - More Multithreading
    The PNGout trials with different block threshold size are now multithreaded. Albeit only 2 threads since I havent found a method to spread out the work dynamically for multiple threads
    still some work balancing needs to be done as one thread completes way faster than the other.


    Code:
    @ECHO OFF
    set strenght=%2
    if "%2"=="" set strenght=15
    set RandIt=0
    set /a RandIt=(%2/%NUMBER_OF_PROCESSORS%)
    set /a TotalIt=%RandIt%"*%NUMBER_OF_PROCESSORS%
    
    ECHO --- SvenBent's PNGbest is optimizing your PNG file
    ECHO CPU threads: %NUMBER_OF_PROCESSORS%
    ECHO Zopfli compression strenght: %strenght%
    ECHO Random initial tables : %TotalIt%
    if "%3"=="brutal" ECHO Extended brute force: Enabled (not supported)
    if "%3"=="mixing" ECHO Mixing only mode: Enabled
    if "%3"=="pal" ECHO Expanding palette size: Enabled (use for less than 8bits pallette images)
    ECHO File: %1
    ECHO Size: %~z1
    
    if "%3"=="mixing" Goto Mixing
    
    
    
    ECHO --- PNGwolf-Zopfli ---
    pngwolf --in=%1 --out=%1 --zopfli-iterations=%strenght%
    
    set wolfsize=%~z1
    
    
    
    @echo off
    
    
    
    ECHO --- Finding optimal filter for PNGout ---
    Pngout %1 %1.out.png /f6 /force 
    for %%f in (6,0,1,2,3,4,5) do pngout %1.out.png /f%%f /d0
    if "%3"=="pal" for %%f in (6,0,1,2,3,4,5) do pngout %1.out.png /f%%f /d8
    
    
    
    ECHO --- Block split threshold ---
    
    Copy %1.out.png %1.split1.png
    echo > %1.Split1.tag
    echo @echo off > %1.Split1.bat
    echo for %%%%b in (64,96,128,192,384,512,768) do pngout %1.split1.png /f6 /b%%%%b >> %1.Split1.bat
    echo del %1.Split1.tag >> %1.Split1.bat
    echo exit >> %1.Split1.bat
    
    Copy %1.out.png %1.split2.png
    echo > %1.Split2.tag
    echo @echo off > %1.Split2.bat
    echo for %%%%b in (1024,1536,2048,3072,4096,8192,16384,0) do pngout %1.split2.png /f6 /b%%%%b  >> %1.Split2.bat
    echo del %1.Split2.tag >> %1.Split2.bat
    echo exit >> %1.Split2.bat
    
    start %1.Split1.bat
    start %1.Split2.bat
    
    echo Waiting for threads to close
    :splitwait
    Timeout 1 >nul
    if exist %1.Split?.tag goto splitwait
    del %1.Split?.bat
    
    huffmix %1.out.png %1.split1.png %1.out.png
    huffmix %1.out.png %1.split2.png %1.out.png
    del %1.split1.png
    del %1.split2.png
    
    
    Echo -- Comparing PNGwolf vs PNGout  --
    echo @echo off > %1.chksmll.bat
    echo if %%~z1 LSS %%~z2 Del %%2 >>%1.chksmll.bat
    echo if %%~z1 GEQ %%~z2 Del %%1 >>%1.chksmll.bat
    echo if exist %%2 ren %%2 %%1 >>%1.chksmll.bat
    echo exit >>%1.chksmll.bat
    
    start /wait %1.chksmll.bat %1 %1.out.png
    del %1.chksmll.bat
    
    
    
    if %~z1 GTR %wolfsize% goto norandom
    if "%2"=="" goto norandom
    
    
    
    :Mixing
    ECHO --- Random initial tables and mixing blocks ---
    
    echo   @ECHO OFF >%1.pngmix.bat
    echo   for /l %%%%r in (1,1,%RandIt%) do ( >>%1.pngmix.bat
    echo   ECHO --- Trial: %%%%r of %RandIt% --- >>%1.pngmix.bat
    echo   pngout  %1.thread.%%1.png %1.tmp.%%1.png /r /force /f6 /kp >>%1.pngmix.bat
    echo   Deflopt /b /s %1.tmp.%%1.png >>%1.pngmix.bat
    echo   huffmix %1.thread.%%1.png %1.tmp.%%1.png %1.thread.%%1.png  >>%1.pngmix.bat
    echo   del %1.tmp.%%1.png  >>%1.pngmix.bat
    echo )  >>%1.pngmix.bat
    echo del %1.%%1.tag >>%1.pngmix.bat
    echo exit >>%1.pngmix.bat
    
    for /l %%t in (1,1,%NUMBER_OF_PROCESSORS%) do (
    Copy %1 %1.thread.%%t.png
    echo tag > %1.%%t.tag
    start "Thread%%t" %1.PNGmix.bat %%t
    )
    
    ECHO Waiting for threads to close
    :keepwait
    timeout 1 >nul
    if exist %1.?.tag goto keepwait
    if exist %1.??.tag goto keepwait
    if exist %1.???.tag goto keepwait
    if exist %1.????.tag goto keepwait
    
    ECHO --- Final mixing stage ---
    for /l %%t in (1,1,%NUMBER_OF_PROCESSORS%) do (
    huffmix %1 %1.thread.%%t.png %1
    del %1.thread.%%t.png
    )
    del %1.pngmix.bat
    
    
    :norandom
    
    
    ECHO --- Optimizing deflate structure ---
    deflopt /b %1
    defluff <%1 >%1.fluff.png
    deflopt /b %1.fluff.png
    huffmix %1 %1.fluff.png %1
    deflopt /b %1
    del %1.fluff.png
    Last edited by SvenBent; 14th November 2014 at 13:29.

  2. The Following 2 Users Say Thank You to SvenBent For This Useful Post:

    lorents17 (29th August 2015),Sebastian W (16th June 2014)

  3. #2
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by SvenBent View Post
    P.S. if you are running on an older windows you might need to replaced the line
    timeout 1 >nul
    with
    Ping 127.0.0.1 >nul
    timeout 1 command takes averagely 1.109 sec to complete, while ping by default sends four packets and takes averagely 3.015 sec. So its better to use ping -n 2 127.0.0.1 which is averagely takes 1.093 sec.

  4. #3
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    nice to know but for this it doesn matter much for this bat

    the timeout and ping command is not to make a delay its rather to emulate Sleep() so it does not eat up an entire cpu just checking to see if the threads are done.
    1 second or 4 second would worst case delay the completion with 3 secs which is pretty much unnoticeable for this batch.

    but then again every second saved is a boost

  5. #4
    Member
    Join Date
    Jul 2013
    Location
    Germany
    Posts
    24
    Thanks
    4
    Thanked 4 Times in 4 Posts
    Would you like to pass the parallel-part to an external tool, made for this kind of work? It might be faster.

    After trying to write my own multithreading routine in AutoIt and as batch-files, I finally found PPX2, which is an open source windows clone of the Linux standard tool xargs. Now parallelization is very easy. I mentioned it in this thread.

    You just pass a list of lines with the pipe-character and the tool runs a command line in parallel and replaces the placeholders with the lines you pass, starting from the top. It does not have to be a file-list, like in my example. It can be a list of parameters too.

    It is a good idea to sort the items in the list by the time necessary to process them. Long time first, short time last. So it is more likely, that the work is divided evenly between the processors.

    I am using this tool for at least a year and it is very stable. I tried it with a large number of small tasks, a small number of large tasks, running for seconds to days. No crash, no skipping, everything fine. (Win7, x64) Typical for a specialized Linux-tool. Well ported. Some features are missing but the features available run very well.

  6. The Following User Says Thank You to Sebastian W For This Useful Post:

    SvenBent (13th June 2014)

  7. #5
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    THank you sebastion i will look into it. i promised myself to also learn powershell scripting this year. and maybe even a real programming lanaguage.

    but so far i just had forcing batch files to do something they where not really made for



    Also i forgot to mentioned the Compression strength and mode are optional. and doesn't have to be specified. (but they have to be in the right order/position)
    It even work by just dropping a file on the batch file


    Im thinking about reworking it so that sub batches files is created in %tmp% instead

  8. #6
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    Updated faster version with microscopically improved compression

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •