Results 1 to 7 of 7

Thread: Dictionary to Archive Size Question

  1. #1
    Member
    Join Date
    Jan 2009
    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Dictionary to Archive Size Question

    Hello! I was referred to here for this question because you people are very intelligent. ^_^

    I am preparing an archive which contains about 3GB of BMP files, many of which are slight modifications of a single BMP (there are several of these files). However, since this archive is going to be part of a public build, I cannot risk more than an 800MB dictionary size because some of my users will only have 1GB RAM. This dictionary size is less than the archive size. Would I get better compression if I were to split the archive into six 500MB parts and forcing an 800MB dictionary? I would split them so that all the key BMP files would be grouped together with the modified BMP files, allowing for the best possible compression.

    I'm looking at using CCMx (setting 6 for 786MB) as the compressor because it's 12-15% better than 7zip (ultra) and decompresses in about 20 minutes which is fine by me. FreeARC (-mx) performs consistently worse (about 3-5%) than 7zip, while PAQ is out of the question because it's not practical.

    Thank you for your time!

  2. #2
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    215
    Thanks
    0
    Thanked 0 Times in 0 Posts
    [offtopic]
    Did you try UHARC? It should do seriously well on such data. Better than any other LZ archiver, that's for sure ^^
    [/offtopic]

  3. #3
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    I think you should consider some preprocessing, for example only storing the key BMP files as they are and delta-encoding the others. If they're only slightly modified, you could save a lot of compression time and get a better ratio.
    http://schnaader.info
    Damn kids. They're all alike.

  4. #4
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    You may try PIM archiver as well. Although it has no solid mode...

  5. #5
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    RAR does well on BMPs and at compressing lots of files which are similar as it finds the differences between them and only compresses those.
    SBC is also cracking at BMP and another good choice.
    Both of those will use little memory.

  6. #6
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Intrinsic View Post
    RAR does well on BMPs and at compressing lots of files which are similar as it finds the differences between them and only compresses those.
    SBC is also cracking at BMP and another good choice.
    Both of those will use little memory.
    Probably much too small dictionary size.

  7. #7
    Member
    Join Date
    Jan 2009
    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I didn't try UHARC because of its very slow decompression time (twice as slow as CCMx) in all the benchmark charts, but I tried it today because nanoflooder suggested it. It simply ran away with the crown. What's funny is that -m3 is actually better than -mx (by ~1%) and decompresses in a fraction of the time! Sample data was 1200 BMP files (varying 24-bit and 32-bit), 2.1gb on a Core 2 Quad 2.7ghz.

    7-zip -md=27 -mfb=273 -mmc=10000 -mlc=8
    300 155 160, decode time 82s

    uharc -m3 -md32768 -b32768 -mm+
    239 137 085, decode time 218s [2 files to get past the 2gb limit]

    Thanks, nanoflooder! Under 4 minutes is definitely acceptable for a public build.
    Last edited by GipFace; 22nd January 2009 at 04:09.

Similar Threads

  1. bzip2 dictionary size
    By Wladmir in forum Data Compression
    Replies: 3
    Last Post: 7th April 2010, 16:09
  2. LZC Question
    By moisesmcardona in forum Data Compression
    Replies: 3
    Last Post: 16th August 2009, 22:33
  3. Does rep support >2GB Dictionary size ?
    By SvenBent in forum Data Compression
    Replies: 12
    Last Post: 6th June 2009, 00:08
  4. Brute forcing Delta block size
    By SvenBent in forum Data Compression
    Replies: 2
    Last Post: 2nd May 2009, 12:44
  5. Noob question about dictionary size (and about rep)
    By SvenBent in forum Data Compression
    Replies: 1
    Last Post: 23rd January 2009, 01:35

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •