Results 1 to 5 of 5

Thread: parallel compression with batches

  1. #1
    Member evg's Avatar
    Join Date
    May 2009
    Location
    Austria
    Posts
    23
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Post parallel compression with batches

    Hello all,

    after a bit of experimenting with pipes i found a fairly quick method of compressing huge files at the cost of lots of memory. The attached batches simply run both commands in parallel, usefull if you got memory to burn and at least a dual core machine. It also helps to overcome the 2GB process memory limit in XP32. At the bottom there are some measurings.

    Attached you will find 4 batches for use with rep 1.2a and 7z 9.07/ccm 1.30c.
    The batches are really crude, no error checking whatsoever, maybe somebody who is better at batches can elaborate.

    Usage is simple: batch file_to_compress.

    Unfortunatly there are not many compressors/filters which are able to use pipes for processing.

    Maybe there is a wrapper to let normal compressors use stdin/out? I seem to dimly remember there was something similar for DOS/djgpp in the olden days.

    Hope you find it usefull and best regards
    evg

    Source
    Ubuntu karmic-desktop-i386.iso.pcf40, size 1,79GB (.iso 657MB)
    precomp40 -slow -l3, took some hours

    fa 0.52 Sept 8 2009
    comp 419984KB, 2097 seconds, used memory ~1.4GB
    fa decomp 177 seconds
    -mrep:a99:b1gb:h24+exe+delta+lzma:128mb:max:a1:273: mc256

    rep7z rep 1.2a+7za 9.07b
    comp 419760KB, 1176 seconds, used memory ~2.5GB (Taskmanager)
    unrep7z decomp 133 seconds
    similar settings to fa, see batch

    All files md5 checked out ok
    Timings on Athlon X2 5600/4GB and XP32-SP3
    Attached Files Attached Files

  2. #2
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    LOL just add -lc- to fa cmdline

  3. #3
    Member evg's Avatar
    Join Date
    May 2009
    Location
    Austria
    Posts
    23
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Unhappy

    Quote Originally Posted by Bulat Ziganshin View Post
    LOL just add -lc- to fa cmdline
    Hello Bulat,

    I tried out your suggestion but arc tries to allocate ~2.4GB memory and quietly fails without error message. This is quiet unsurprising because on a stock 32bit XP process memory is limited to 2GB minus several mb. I dont think that arc spawns a second process to overcome that limit. Maybe its possible with /3GB or /UserVA boot.ini switches but on my system this does not work out due to crappy drivers.

    Also your suggested switch -lc- is neither documented in the program help or the online 0.40 translation. Maybe it would be time to update that too. I'd volunteer to help with the translation but my russian is nonexistent.

    Dont get me wrong, I dont want to bash freearc, I've been using it intensively in the last time and find a very good archiver.

    Best regards
    evg

    update:
    rechecked my batch with 1536mb dictionary for rep, works quite fine too but reaches the limits of my system.
    rep 1536m+7z 128m allocation in taskmanager: rep 1636724K (1598,36G) + 7z 1427512K (1394,05G) = total 3064236K (2992,42G)
    Last edited by evg; 16th September 2009 at 18:24. Reason: update

  4. #4
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    you are right, it's better to use 64-bit windows if you ahve 4gb of ram

  5. #5
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    For me personally i think these batches have a moot point and only have a benefit for people in af very specific HW layout


    a: 7-zip LZMA2 can work with several threads anyhow.

    b: this only utilize one extra core above 7-zip

    b: they are for people that have to much memory and yet not enough.
    They must have enough memory for rep + 7-zip yet not enough to just use the same dictionay with 7.zip



    basically hey have a sweet spot with triple cores machine with a couple of gigs ram to spare (~4-6gb)


    people with more ram just use 7-zip dictionary and not rep
    people with more cores can just use more threads with 7-zip

    my own batch for brute force comrpession uses 5-6 cores
    the decoding batch uses around one core for each file to decompress



    but basically Rep is getting useless for me, because for less then 1GB dictionaries i just use 7-zip
    and Rep max dictionare is still "just" 2GB
    so its only a few files rep show anybenefits to me
    Last edited by SvenBent; 17th September 2009 at 18:20.

Similar Threads

  1. Parallel BZIP2 (PBZIP2)
    By joerg in forum Data Compression
    Replies: 21
    Last Post: 18th January 2012, 23:33
  2. LZTURBO 0.91 Parallel Compressor (Win32/Linux)
    By donotdisturb in forum Forum Archive
    Replies: 26
    Last Post: 19th April 2008, 20:15
  3. LZTURBO 0.9 parallel compressor
    By donotdisturb in forum Forum Archive
    Replies: 18
    Last Post: 6th March 2008, 01:23
  4. LZTURBO 0.1 parallel compressor
    By donotdisturb in forum Forum Archive
    Replies: 5
    Last Post: 7th October 2007, 22:44
  5. Parallel implmentation of gzip: pigz
    By nimdamsk in forum Forum Archive
    Replies: 2
    Last Post: 13th March 2007, 20:44

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •