Results 1 to 13 of 13

Thread: Does rep support >2GB Dictionary size ?

  1. #1
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts

    Does rep support >2GB Dictionary size ?

    I'm compressing a file which REALLY like big dictionary size. how ever I'm not sure if Rep does actually support 2GB dictionary and above.

    the memory utilization in task manager looks fine until i select 2gb or 2048mb. (2047mb is fine) then it drops back to only 2.1xx.xxx K and stays there no matter how much i increase dictionary size

    Dict = VM size
    512m = 657.688 K
    1024m = 1.314.332 K
    1536m = 1.839.644 k
    1999m = 2.577.336 K
    2047m = 2.626.584 K

    2048m = 2.126.288 K
    2g = 2.126.288 K
    3072m = 2.126.288 K
    3g = 2.126.288 K
    3072m = 2.126.288 K

    thats does'n look right.

    and i need some LZ based compression which can have more than 2GB dictionary size. but use less then 8GB memory for compression

  2. #2
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by SvenBent
    I'm compressing a file which REALLY like big dictionary size. how ever I'm not sure if Rep does actually support 2GB dictionary and above.
    I'm not sure about possibility of REP to use 2048m or more but as far as I know you can put two or maybe more REPs in your chain
    Code:
    -mrep:2047m+rep:2047m+...
    You can also try to include -lc- to your command line.

    Quote Originally Posted by SvenBent
    and i need some LZ based compression which can have more than 2GB dictionary size. but use less then 8GB memory for compression
    Try arc -mlzma:dXXXXm:mfht4
    Please let me know about your tests cause I don't have such amount of RAM so can't test for myself.

  3. #3
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    i dont use arc. i just use the standalone rep versioen 1.2

    i triede using to reps in a row
    second pass did only reduced the file with 0.2%

    this is a 8.2GB iso file which by using rep -b2047mb comes down to 3.32GB

  4. #4
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by SvenBent View Post
    i dont use arc. i just use the standalone rep versioen 1.2
    So why not to try it ? You've asked for LZ based compression and correct me if I wrong, LZMA is LZ based. Furhermore REP included in FreeARC so you can have all-in-one solution. Also. Why you think your version of REP is 1.2 ? I don't see any version neither in console output, neither in sources.

    I don't know which command line you're using but you can try:
    Code:
    -b2047 -l512 -h28 -a99

  5. #5
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    So why not to try it ?
    Because if rep is the big dictionary preprocesser/compressor in FA and that is not big enough. its probably not going to help to use Arc.
    but i will try it


    Why you think your version of REP is 1.2 ?
    Because this is what it says when you run REP without arguments

    Code:
    REP preprocessor v1.2 alpha (c) Bulat.Ziganshin@gmail.com  2008-11-18
    
     Usage: rep [options and files in any order]
       -z      -- force compression
       -d      -- force decompression
       -oNAME  -- output filename/directory (default .rep/.unrep)
       -t      -- test (de)compression (redirect output to nul)
       -delete -- delete successfully (de)compressed input files
       -q      -- be quiet. -q[thpr]* disables title/header/progress/results individ
    ually
       -v      -- verbosity level (you should recompile program with -DDEBUG to enab
    le this feature)
       -cpu    -- compute raw CPU time (for becnhmarking)
       -h      -- display this help
       --      -- stop flags processing
     "-" used as filename means stdin/stdout
    
     Compression parameters:
       -b#     --  sliding window size (default: 128mb, recommended: half of total R
    AM)
       -l#     --  minimal match len
       -d#     --  barrier for smaller matches
       -s#     --  minimal match len after barrier
       -h#     --  hash bits
       -a#     --  coefficient of search "amplification" (1 - default, 99 - max. com
    pr.)

  6. #6
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    i tried using arc -mlzma:dXXXXm:mfht4 seems to not go above 1024M Dictionary size

    settings = Mem usage
    -d1024m = 1.877.044K
    -d2048m = 1.877.044K
    -d3072m = 1.877.044K

    I hardly think that if you triple the dict size that you remain within 1kilybytes of mem usage.

    Anyway going to let both 1024 and 3072 continue an have a look at the end file.
    Last edited by SvenBent; 5th June 2009 at 10:07.

  7. #7
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    lzma by itself doesn't support dicts >1gb

    rep is limited by largest memory block you can alloc. try it on linux! also rep+rep usually helps in these cases, not always though

    larger rep dicts may be avaliable only with 64-bit version, but i don't think that rep sources now are 64-bit ready

  8. #8
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    Thanx for the input Bulat. Just found out myself that "using" 3gb or 1GB dictionaries with lzma give exactly the same file (same size and same md5sum)


    Regarding rep i don't get the out of memory message. it actually starts but just uses less memory then what my logic would say. and since it limits at the 2gb point it was thinking it was a 32bit/windows limits in rep. (I'm using WinXP 64)

    Would it be hard to recompile a 64bit version which support larger dicts ?

    Org size = 7,88gb
    Using lzma 1gb dict = 6.52GB
    Using rep 2047mb dict = 3.35 gb

    is a rather odds case as i have never seen something (real life data) that likes these kind of big dictionaries

    i believe the reason for is it that the iso file contains a rerelease of several game CD's whicch contains the same install files, same acrobat reader, same DirectX, etc etc.

  9. #9
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    i don't think that rep sources now are 64-bit ready

  10. #10
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by SvenBent View Post
    Because this is what it says when you run REP without arguments

    Code:
    REP preprocessor v1.2 alpha (c) Bulat.Ziganshin@gmail.com  2008-11-18
    Is this REP version available for download ?
    Do you maybe have a change-log ?

    BTW: IIRC, current 7z (4.65 and 9.04) both support dictionaries up to 1 GB. (-md30)
    You'll need lot's of memory for this though!
    Last edited by pat357; 5th June 2009 at 15:07.

  11. #11
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    I got it from bulat because the version did use prior had a bug.
    I'm not sure if I'm allowed to reditribute it.


    an LZMA with 1gb dictionarys just doesn't cut it on this file


    -- edit ---
    Look here
    http://encode.ru/forum/showthread.ph....2alpha&page=4

    Bulat posts the link for the 1.2 alpha


    @Bulat.
    must have been blind when i asked you about the 64 bit version
    Last edited by SvenBent; 5th June 2009 at 15:31.

  12. #12
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by SvenBent View Post
    Bulat posts the link for REP 1.2 alpha
    Thanks, got it.

    If someone else wants it, the full link is :
    http://encode.ru/forum/attachment.ph...8&d=1232632247

  13. #13
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by SvenBent View Post
    Bulat posts the link for the 1.2 alpha
    Quote Originally Posted by Bulat Ziganshin
    REP preprocessor v1.2 alpha (c) Bulat.Ziganshin@gmail.com 2008-11-18
    Completely but pleasantly unexpected and fascinating ! I wonder how many pearls are in the depths of encode.ru

Similar Threads

  1. bzip2 dictionary size
    By Wladmir in forum Data Compression
    Replies: 3
    Last Post: 7th April 2010, 16:09
  2. Noob question about dictionary size (and about rep)
    By SvenBent in forum Data Compression
    Replies: 1
    Last Post: 23rd January 2009, 01:35
  3. Dictionary to Archive Size Question
    By GipFace in forum Data Compression
    Replies: 6
    Last Post: 21st January 2009, 18:03
  4. REP and Delta fails with big files
    By SvenBent in forum Data Compression
    Replies: 14
    Last Post: 23rd November 2008, 20:41
  5. Precomp v0.3.8 >2GB test version
    By schnaader in forum Data Compression
    Replies: 22
    Last Post: 15th July 2008, 12:47

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •