Results 1 to 8 of 8

Thread: The 1MB Compression Benchmark

  1. #1
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    174
    Thanked 51 Times in 37 Posts

    The 1MB Compression Benchmark

    Good evening everyone!

    This benchmark is what the "DOS Archiver Benchmark" originally was supposed to be but then changed.

    This is just a fun benchmark to see how much we can compress a file with the following 2 requirements:

    1) The program can NOT exceed 1,999 K in Windows Task Manger during compression.
    2) The compression program must be a Win32 or Win64 executable.

    The file to be compressed is as follows:

    - Microsoft Office folder "Office12"
    - Size: 555 MB (582,474,448 bytes)
    - Total Files: 2,054

    The folder is archived in a single .shar file using Shelwien's SHAR archiver program with the following command:

    Code:
    shar a Office12.shar Office12\
    And now for the results:

    UPDATES
    12/19/14 - Updated UFA with "-m9" switch
    12/18/14 - Added UFA 0.04b1
    12/18/14 - Added ARJZ 0.15
    12/18/14 - Updated RAR with "-mm" switch
    12/18/14 - Added IMP 1.12
    12/18/14 - Added ASHFORD
    12/17/14 - Added LZOP
    12/17/14 - Added P5
    12/17/14 - Added SNAPPY 1.1.1.8

    Code:
    SIZE        | PROGRAM/SWITCHES
    ==============================================
    361,017,573 | SNAPPY 1.1.1.8
    339,895,453 | FCM1
    323,422,347 | P5
    316,368,134 | LZARI
    311,091,351 | LZOP 1.03 -9
    304,518,765 | ASHFORD 4
    295,034,882 | GZIP 1.3.5 -9
    294,847,400 | HA 0.999b a12
    292,049,003 | ARJZ 0.15 -mp9
    289,085,400 | ICT Universal Coder 2.1 em1
    287,445,924 | JAR32 1.02 -m3
    285,495,858 | QUANTUM 0.97 -c7 -t16
    285,197,211 | ACE32 1.2 -m5 -d64
    283,091,108 | CABARC -m LZX:15
    282,180,711 | RAR 2.00 -m5 -md128 -mm
    281,889,329 | IMP 1.12 -1 -m3 -mm -u200
    275,802,821 | PPMs J -o6
    265,063,803 | UFA 0.04b1 -mx -m9 -md64
    If you'd like to suggest or submit a program to be tested, please post or send me a PM. For submissions, please only submit Win32 or Win64 executables as I will not bother trying to compile sources.

    I will continue to update this first post when new file compressors are found (by me or by others) that meet the requirements for this benchmark.

    IMPORTANT NOTE: Please remember, this benchmark has absolutely no practical purpose--it is just for fun.
    Last edited by comp1; 19th December 2014 at 15:30.

  2. #2
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    If you have the patience you might want to try:

    zpaq add office12.zpaq office12.shar -method 57 -threads 1
    zpaq extract office12.zpaq office12.shar -to office12.out -threads 1
    fc/b office12.shar office12.out

    Or compress the folder instead of the shar file. It will probably compress better that way.

    Get zpaq.exe or zpaq64.exe at http://mattmahoney.net/dc/zpaq.html

  3. #3
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    856
    Thanks
    447
    Thanked 254 Times in 103 Posts
    lz4 -9 -B4D

    triggers the 64KB block mode,
    should remain below memory limit.

    Pre-compiled windows binary

  4. #4
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    174
    Thanked 51 Times in 37 Posts
    @Cyan

    Ahhh so close!!!



    @Matt



    And it was still going up...

    In regards to compressing the folder as opposed to one large file: The problem is that in some cases (like with ACE32), compressing a folder of 2,000+ files increases the memory usage past the limit. Also, some programs are not archivers but only compressors. I wanted to keep this competition as fair and consistent as possible.
    Last edited by comp1; 18th December 2014 at 14:36.

  5. #5
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    856
    Thanks
    447
    Thanked 254 Times in 103 Posts
    Pretty strange
    I made several tests, and consistently received 1884KB as peak memory usage
    (which is already too much, and shows that the underlying allocation practice is a bit too "generous").
    It would be interesting to understand why there is such a difference between both systems.

    In the meantime, it's also possible to directly compile from the open source code.
    Memory usage should be much more frugal on this version.

  6. #6
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    174
    Thanked 51 Times in 37 Posts
    Quote Originally Posted by Cyan View Post
    Pretty strange
    I made several tests, and consistently received 1884KB as peak memory usage
    (which is already too much, and shows that the underlying allocation practice is a bit too "generous").
    It would be interesting to understand why there is such a difference between both systems.

    In the meantime, it's also possible to directly compile from the open source code.
    Memory usage should be much more frugal on this version.
    That is quite a difference...Interesting.

    Well, I may look at compiling the source sometime but my plan was not to mess with compiling source code for this benchmark.

    Thanks.

  7. The Following User Says Thank You to comp1 For This Useful Post:

    just a worm (22nd December 2014)

  8. #7
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    Oh, I misread what you said. I thought you said 1,999 MB

  9. #8
    Member just a worm's Avatar
    Join Date
    Aug 2013
    Location
    planet "earth"
    Posts
    96
    Thanks
    29
    Thanked 6 Times in 5 Posts
    Quote Originally Posted by comp1
    Well, I may look at compiling the source sometime but my plan was not to mess with compiling source code for this benchmark.
    That's an attitute I sympathize with. Why should every user compile a program instead of the developer compiles it once? The developer usually knows the best how to compile his program and has all the required external libraries etc. But some people still distribute the source code only.

Similar Threads

  1. MONSTER OF COMPRESSION - New Benchmark
    By LovePimple in forum Data Compression
    Replies: 225
    Last Post: 23rd December 2009, 11:57
  2. New benchmark for generic compression
    By Matt Mahoney in forum Data Compression
    Replies: 20
    Last Post: 29th December 2008, 09:20
  3. MONSTER OF COMPRESSION - New Benchmark -
    By Nania Francesco in forum Forum Archive
    Replies: 222
    Last Post: 5th May 2008, 10:04
  4. Compression speed benchmark
    By Sportman in forum Forum Archive
    Replies: 104
    Last Post: 23rd April 2008, 16:38
  5. Synthetic compression benchmark
    By giorgiotani in forum Forum Archive
    Replies: 6
    Last Post: 3rd March 2008, 12:14

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •