Results 1 to 12 of 12

Thread: Archiver benchmarking framework

  1. #1
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts

    Archiver benchmarking framework

    I'm glad to present my new micro-framework for archiver benchmarking. It was developed with the following goals in mind:
    • Open-source, portable, interpreted code
    • Flexible enough to support any archiver with add/list/test/extract commands
    • Collect as much as possible information about archiver run (user/kernel times, virtual/workset memory, number of I/O operations)
    • Separate stages of information collection and report generation, with saving collected stats to JSON files
    • Flexible report generation in various formats with filtering and ordering
    Some goals aren't yet met: it's Windows-only, it collects only times, and it generates only simple reports. But it's still useful, so i share it with you.


    Installation instructions: since scripts are written in Lua, and use random libs from LfW, you need to:
    1. Install Lua for Windows
    2. Download my benchmark suite
    3. Copy winapi.dll to %Lua%\clibs directory
    Run "lua format.lua" to print formatted results of my benchmarks from logfile.txt, or edit testset/archiver configs in bench.lua and run "lua bench.lua" to perform benchmarks, saving results to logfile.txt
    Last edited by Bulat Ziganshin; 7th February 2013 at 18:37.

  2. #2
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    And now the first result - that's the data you can find in logfile.txt. These are results of compression of 100mb, 5000 files from RamDrive to RamDrive:
      globalprocessuserkernelsize
    Shar  0.3590.3580.0310.327104594025
    NanoZip 0.09-cn0.3290.3260.0460.28104531841
     -cf0.3590.9660.7170.24946000560
     -cF0.5541.7771.1850.59240967010
     -cd0.8812.7612.4180.34337338431
     -cD1.4455.3975.0230.37434537500
    7-zip 9.30-mx00.7080.7010.4210.28104276154
     -mx1 -tzip0.7373.9463.4940.45247801278
     -mx11.8788.478.190.2839651343
     -mx33.28314.39814.0080.3935855868
    zpaq 6.21-method 01.7052.7612.4490.31296501996
     -method 12.4167.3166.8480.46839799008
     -method 24.4920.02919.5460.48335267887
    FreeArc 0.67--nodata0.2060.2180.140.07830938
     -m01.992.0890.4831.606104265818
     -mx1 -tzip2.2494.6642.9481.71650432151
     -mxlz41.9862.2920.6861.60658122724
     -mrep:512m2.1092.2450.5141.73185881059
     -m12.1383.121.4821.63841178148
     -m22.5745.1633.3541.80935778085
     -m33.3329.0797.1291.9534263944
     -m45.81815.95813.9931.96533404252
    Exdupe 0.36-x03.5544.3361.8722.46499269422
     -x13.4854.321.7782.54258440448
     -x24.5426.9264.1182.80849932583
     -x310.32719.70215.1474.55545809268
    Rar 4.20-m02.0011.980.2491.731104682371
     -m1 -mda3.1083.9932.1841.80948150631
     -m13.8164.822.9171.90347521593
     -m24.76711.8249.7032.12143878705

  3. #3
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    how about adding checks for archiver features:
    - unicode names
    - windows paths (\\.\C:\ network paths, long paths/names which are not supported without \\?\ prefix)
    - links/junctions
    - attributes
    - timestamps (which of 3, plus precision)
    - order of file creation in a directory (lossless transformation from archive to FS and back)
    - windows streams

  4. #4
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Very interesting.
    Is it far from being portable?

  5. #5
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    856
    Thanks
    447
    Thanked 254 Times in 103 Posts
    That's a great start

  6. #6
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Shelwien, unfortunately, it's only about benchmarking, not feature-checking. i don't see anything in common except for list of archiver executables

    m^2, Lua by itself is portable, as well as most of its libraries. The only non-portable part is measuring itself - i.e. runtime() function that uses winapi library. it just need other OS-specific variants

  7. #7
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > it's only about benchmarking, not feature-checking

    However archiving/restoring speed is noticeably affected by attributes, timestamps, streams.
    So without checks for these features at least, store speed comparisons don't make sense.

  8. #8
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    So, you think that all existing benchmarks were useless?

  9. #9
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Somewhat.
    I think that the usual wall-of-text compression benchmarks are a legacy from old times,
    when files were small and only compression ratio really mattered.
    And now, imho, the much better format for benchmarks is something similar to cpu reviews -
    choose a practical task, find the tools which can perform it, and analyze their properties.
    Otherwise its all similar to me making a benchmark with specific testsets, where reflate/plzma
    would be best of all.

    As to this specific case, I just noticed that shar is much faster than rar/7z/fa in store modes,
    and thought that it probably can be explained by its ignorance of attributes and timestamps.
    Then, the same might apply to nanozip. Or maybe its something different in that case.
    That's why I asked about additional checks.

  10. #10
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    yes, it is exactly why i published its results. but the reason is different and can be found only by tracing winapi calls. so your idea doesn't work in the first case checked. i plan to make series of benchmarks highlighting various aspects of archiving, but analysis of reasons is left to the humans

  11. #11
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Uh, I forgot about computing crcs.
    fa's storing might be slower because of some sha256 or whatever you use there.

  12. #12
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    Shelwien, unfortunately, it's only about benchmarking, not feature-checking. i don't see anything in common except for list of archiver executables

    m^2, Lua by itself is portable, as well as most of its libraries. The only non-portable part is measuring itself - i.e. runtime() function that uses winapi library. it just need other OS-specific variants
    Sounds good.

Similar Threads

  1. Compression benchmarking: 64 bit images and 24 bit codecs
    By m^2 in forum The Off-Topic Lounge
    Replies: 6
    Last Post: 30th November 2011, 17:01
  2. paq8f w/ .DXEs (DJGPPv2, DOS, benchmarking)
    By Rugxulo in forum Data Compression
    Replies: 4
    Last Post: 2nd February 2010, 15:32
  3. Help beta testing QuickLZ 1.40 with the new test framework
    By Lasse Reinhold in forum Forum Archive
    Replies: 10
    Last Post: 19th April 2008, 16:16

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •