Results 1 to 5 of 5

Thread: StartMT (an idea)

  1. #1
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts

    StartMT (an idea)

    Im wondering if anyoee would be intersted in making this idea come true

    Code:
    StartMT <Options> [Filelist] [Command]
    Start commands for each files in multiple threads.
    
    <Options> Optional
    -t#     Set numbers of threads to use to # (default: Autodetects)
    -n0     Dont use CPU0 (Enables -al)
    -NoHT   Dont count Virtual cores on HT enabled processors (enables -al)
    -al     Locks each thread to a cores (default: threads are not affinity locked
    -s      Treverse Subdiretories
    
    [Filelist] Specifies the the files to execute command one
    
    {Command} Specifies what command to run on files
              use %file% in command line for file selection.
              use %thread% in command line to insert thread number
    
    Example:
    StartMT -n0 *.png Pngout %file% /kp
       runs pngout with parameter /kp on all *.png files. in a sepraet thread    for all cpu except core0
    With this tool it would be easy to make simple file Command compression multit hreaded as StartMT.exe will autmatic execute the command in parralel and feed an individual filename from the filelist to each command
    using above example command in the kodak suit it would execute these command in sequence

    kodim01.png

    pngout kodim01.png /kp
    pngout kodim02.png /kp
    pngout kodim03.png /kp
    pngout kodim04.png /kp
    pngout kodim05.png /kp
    pngout kodim06.png /kp
    pngout kodim07.png /kp
    pngout kodim08.png /kp
    pngout kodim09.png /kp
    pngout kodim10.png /kp
    pngout kodim11.png /kp
    pngout kodim12.png /kp
    pngout kodim13.png /kp

    etc etc
    but only enough at a time to fill out the current amount of cpu execution units exempt for core0 (set affinity for the individual thread to cpu 1 2 3 on a 4 core system)

    The disabled HT option would set affinity on threads to each first cpu in a pairs e.g. 0 2 4 6 on a Quadcore cpu with ht (8 execution units, virtual cores0-7)

    -al is the options to use affinity on each thread without it it woudl ust star the threads without affininty. Which is why is set by -n0 and -NoHT options
    -s options will run the command and file in each sub directory


    Offcouse thes options would be icing on the cake and the core functionality of easy multithreading would be nice.

    anyway just an idea. i f i had the ability/skill i would have made it myself
    Last edited by SvenBent; 13th May 2014 at 01:56.

  2. #2
    Member
    Join Date
    Feb 2013
    Location
    San Diego
    Posts
    1,057
    Thanks
    54
    Thanked 71 Times in 55 Posts
    This exists on *nix systems. It's called xargs. Doing a web search on xargs led to some other, similar programs that I didn't know about.

  3. #3
    Member
    Join Date
    Jul 2013
    Location
    Germany
    Posts
    24
    Thanks
    4
    Thanked 4 Times in 4 Posts
    For Windows I use PPX2.exe. The source code is available.
    http://www.pirosa.co.uk/demo/wxargs/wxargs.html

    You can use it like this:

    dir *.png /B /O-S | ppx2.exe -P 8 -L 1 pngcompressor.bat "{}"

    It does not handle filename-wildcards by itself. You can pipe any list of files or anything else, like in my example above. It is not aware of threads and cores and hyperthreding, like your suggestion is.
    Last edited by Sebastian W; 13th May 2014 at 15:29.

  4. #4
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    its the multithreading that I'm the most concerned about. Atm i have to rely and very ugly tricks in batch files like this

    Code:
    copy %1 %1.thread1.png
    echo tag > %1.1.tag
    echo for /l %%%%r in (1,1,%2) do ( >%1.pngmix1.bat
    echo   pngout  %1.thread1.png %1.tmp1.png /r /force /f6 /kp >>%1.pngmix1.bat
    echo   huffmix %1.thread1.png %1.tmp1.png %1.thread1.png  >>%1.pngmix1.bat
    echo   del %1.tmp1.png  >>%1.pngmix1.bat
    echo )  >>%1.pngmix1.bat
    echo del %1.1.tag >>%1.pngmix1.bat
    echo exit >>%1.pngmix1.bat
    
    copy %1 %1.thread2.png
    echo tag > %1.2.tag
    echo for /l %%%%r in (1,1,%2) do ( >%1.pngmix2.bat
    echo   pngout  %1.thread2.png %1.tmp2.png /r /force /f6 /kp >>%1.pngmix2.bat
    echo   huffmix %1.thread2.png %1.tmp2.png %1.thread2.png  >>%1.pngmix2.bat
    echo   del %1.tmp2.png  >>%1.pngmix2.bat
    echo )  >>%1.pngmix2.bat
    echo del %1.2.tag >>%1.pngmix2.bat
    echo exit >>%1.pngmix2.bat
    
    copy %1 %1.thread3.png
    echo tag > %1.3.tag
    echo for /l %%%%r in (1,1,%2) do ( >%1.pngmix3.bat
    echo   pngout  %1.thread3.png %1.tmp3.png /r /force /f6 /kp >>%1.pngmix3.bat
    echo   huffmix %1.thread3.png %1.tmp3.png %1.thread3.png  >>%1.pngmix3.bat
    echo   del %1.tmp3.png  >>%1.pngmix3.bat
    echo )  >>%1.pngmix3.bat
    echo del %1.3.tag >>%1.pngmix3.bat
    echo exit >>%1.pngmix3.bat
    
    copy %1 %1.thread4.png
    echo tag > %1.4.tag
    echo for /l %%%%r in (1,1,%2) do ( >%1.pngmix4.bat
    echo   pngout  %1.thread4.png %1.tmp4.png /r /force /f6 /kp >>%1.pngmix4.bat
    echo   huffmix %1.thread4.png %1.tmp4.png %1.thread4.png  >>%1.pngmix4.bat
    echo   del %1.tmp4.png  >>%1.pngmix4.bat
    echo )  >>%1.pngmix4.bat
    echo del %1.4.tag >>%1.pngmix4.bat
    echo exit >>%1.pngmix4.bat
    
    Start %1.pngmix1.bat %1 %2
    Start %1.pngmix2.bat %1 %2
    Start %1.pngmix3.bat %1 %2
    Start %1.pngmix4.bat %1 %2
    
    :keepwait
    Ping 127.0.0.1
    if exist %1.?.tag goto keepwait
    
    huffmix %1 %1.thread1.png %1
    huffmix %1 %1.thread2.png %1
    huffmix %1 %1.thread3.png %1
    huffmix %1 %1.thread4.png %1
    
    del %1.thread?.png
    del %1.pngmix?.bat
    I have to spawn sub batch files and make small tag files to keep score if a thread is done or not.
    ugly ping test as delays the loops to check if threads are done
    etc etc etc

    This is off cause different kind of multithreading as my batch multithreads on a single/same file.

  5. #5
    Member
    Join Date
    Mar 2014
    Location
    Helsinki, Finland
    Posts
    10
    Thanks
    0
    Thanked 5 Times in 3 Posts
    Quote Originally Posted by SvenBent View Post
    but only enough at a time to fill out the current amount of cpu execution units exempt for core0 (set affinity for the individual thread to cpu 1 2 3 on a 4 core system)

    The disabled HT option would set affinity on threads to each first cpu in a pairs e.g. 0 2 4 6 on a Quadcore cpu with ht (8 execution units, virtual cores0-7)

    -al is the options to use affinity on each thread without it it woudl ust star the threads without affininty. Which is why is set by -n0 and -NoHT options
    -s options will run the command and file in each sub directory
    I'm not sure if there's any significant advantage in manual thread handling. AFAIK, the task of assigning threads to physical/logical cores is pretty well handled by modern kernels. After all, you could have a PC with multiple HT cores in 2002.

    Quote Originally Posted by SvenBent View Post
    Offcouse thes options would be icing on the cake and the core functionality of easy multithreading would be nice.

    anyway just an idea. i f i had the ability/skill i would have made it myself
    If a computational problem is embarassingly parallel, there's usually an easy way of making it run on all cores.

    Unix has always been the more natural platform for doing parallel processing.

    For example, if you have bash script with a serial loop:

    Code:
    for ((i=1 ; i<=$operations ; i++))
    do
      do_something "$arg1" "$arg2"
      if [[ $? -gt 0 ]] ; then exit 1 ; fi
    done
    Then by using 'xargs' (or 'parallel'..) you can convert it into a parallel loop like this:

    Code:
    threads=$(grep -c ^processor /proc/cpuinfo)
    procargs=""
    for ((i=1 ; i<=$operations ; i++))
    do
      procargs="$procargs \"$arg1\" \"$arg2\""
      let buffer=$i%$threads
      if [[ $buffer -eq 0 ]] || [[ $i -eq $operations ]] ; then
        echo $procargs | xargs --max-procs=$threads --max-args=2 do_something
        if [[ $? -gt 0 ]] ; then exit 1 ; fi
        procargs=""
      fi
    done
    Or you could write a suitable Makefile and use 'make -j'. In Unix/Linux there are many possibilities.

    On the other hand, Windows has been developed starting from a single-user, single-core environment. Only recently has there been features introduced in Windows Powershell for running things in parallel. But then there's MinGW/MSYS..

Similar Threads

  1. Idea: Smoothed probability
    By Piotr Tarsa in forum Data Compression
    Replies: 2
    Last Post: 19th February 2013, 21:30
  2. Idea: Combine Compression & Encryption
    By dirks in forum Data Compression
    Replies: 16
    Last Post: 22nd February 2010, 11:49
  3. Compression idea: Base conversion
    By Nightgunner5 in forum Data Compression
    Replies: 8
    Last Post: 30th October 2009, 07:58
  4. Idea - Don't flame me pls...
    By rubendodge in forum Data Compression
    Replies: 7
    Last Post: 24th April 2009, 19:44
  5. New Idea for Hybrid 7-Zip Archiver
    By DeathTheSheep in forum Data Compression
    Replies: 22
    Last Post: 30th December 2008, 22:57

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •