Page 1 of 2 12 LastLast
Results 1 to 30 of 36

Thread: Compressing millions of html

  1. #1
    Member
    Join Date
    May 2012
    Location
    usa
    Posts
    24
    Thanks
    5
    Thanked 0 Times in 0 Posts

    Cool Compressing millions of html files

    Hi there,
    I need suggestions to choose the right software & algorithm for my case.
    I have millions of .html files of archiving websites, each html file mostly contains text and few small images.
    I need a compression solution that can provide fast compression and decompression rates to such files.

    I have tested .7z, and it was a nightmare. any suggestion?

  2. #2
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts
    Take some of your files and generate a dictionary with Zstd then compress.

    Brotli is also good with older HTML files.

  3. #3
    Member
    Join Date
    May 2012
    Location
    usa
    Posts
    24
    Thanks
    5
    Thanked 0 Times in 0 Posts
    Thanks, I think Zstd will do, it's dictionary creation is helpful.

  4. #4
    Member
    Join Date
    Dec 2016
    Location
    Norway
    Posts
    18
    Thanks
    15
    Thanked 10 Times in 4 Posts
    Also worth checking if those images can be losslessly compressed. Try "Pinga" for instance, will do JPG and PNG.

  5. #5
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    772
    Thanks
    63
    Thanked 270 Times in 190 Posts
    Input:
    861,938,525 bytes - 20,000 HTML pages + headers

    Output:
    Code:
    107,282,956 bytes, 26.694 sec.,  1.015 + xx.xxx sec., xwrt -l0 -b255 +d & lzturbo -32 -b1024
    115,966,329 bytes, 28.265 sec.,  1.976 + xx.xxx sec., xwrt -l0 -b255 +d & tor -6
    116,778,388 bytes, 21.142 sec.,  2.127 sec., tor -7 -b1024
    117,479,089 bytes, 25.799 sec.,  2.454 sec., tor-small -9 -b1024
    117,691,077 bytes, 21.339 sec.,  2.051 sec., tor -7
    118,861,524 bytes, 26.275 sec.,  2.445 sec., tor-small -9
    119,356,953 bytes,  7.478 sec.,  1.312 sec., lzturbo -32 -b1024  
    120,900,667 bytes, 22.213 sec.,  2.521 sec., tor-small -8 -b1024
    121,138,361 bytes, 27.850 sec.,  2.386 + xx.xxx sec., xwrt -l0 -b255 +d & tor-small -6 -b1024
    124,452,325 bytes, 22.891 sec.,  2.476 sec., tor-small -8
    126,158,142 bytes, 15.736 sec.,  2.620 sec., tor-small -7 -b1024
    127,266,149 bytes, 22.487 sec.,  3.690 + xx.xxx sec., xwrt -l0 -b255 +d & nz -cF -m1024m
    127,635,533 bytes, 10.436 sec.,  2.273 sec., tor -6 -b1024
    129,992,276 bytes, 15.764 sec.,  2.720 sec., nz -cdP -m1024m
    131,064,250 bytes, 16.531 sec.,  2.526 sec., tor-small -7
    131,918,731 bytes, 15.605 sec.,  2.606 sec., nz -cdP
    132,919,600 bytes, 22.599 sec.,  1.568 sec., brotli -q 6 -w 24
    133,267,492 bytes, 11.009 sec.,  2.214 sec., tor -6
    133,751,721 bytes,  7.483 sec.,  1.322 sec., lzturbo -32
    134,896,907 bytes, 10.145 sec.,  2.707 sec., tor-small -6 -b1024
    136,558,579 bytes, 30.169 sec.,  0.861 sec., zstd -13
    137,484,365 bytes, 28.750 sec., xx.xxx sec., xwrt -l2 -b255 +d
    137,520,471 bytes, 28.605 sec., xx.xxx sec., xwrt -l2 -b255
    137,523,947 bytes, 17.004 sec.,  1.617 sec., brotli -q 5 -w 24
    137,655,975 bytes, 16.283 sec.,  0.813 sec., zstd -12
    138,291,201 bytes, 26.825 sec.,  3.710 sec., 7z -mx4 -t7z
    139,746,607 bytes, 26.719 sec.,  2.027 + xx.xxx sec., xwrt -l0 -b255 +d & rar -m1 -ma5
    140,307,711 bytes, 10.943 sec.,  2.403 sec., tor-small -6
    140,537,389 bytes, 14.895 sec., 20.804 sec., arc -m2
    140,944,857 bytes, 14.398 sec.,  0.835 sec., zstd -11
    141,165,390 bytes,  9.065 sec.,  1.507 sec., brotli -q 4 -w 24
    141,226,215 bytes, 25.752 sec.,  1.960 + xx.xxx sec., xwrt -l0 -b255 +d & 7z -mx1 -tzip
    141,452,371 bytes, 11.362 sec.,  0.838 sec., zstd -10
    142,207,481 bytes,  9.068 sec.,  0.876 sec., zstd -9
    143,480,522 bytes,  7.301 sec.,  0.835 sec., zstd -8
    143,613,139 bytes, 19.290 sec.,  3.925 sec., 7z -mx3 -t7z
    144,142,239 bytes,  6.922 sec.,  0.842 sec., zstd -7
    147,815,180 bytes,  4.905 sec.,  0.863 sec., zstd -6
    151,980,743 bytes, 18.480 sec.,  4.287 sec., 7z -mx2 -t7z
    153,003,725 bytes,  2.784 sec.,  3.105 sec., nz -cF -m1024m
    153,295,198 bytes,  2.676 sec.,  3.060 sec., nz -cF
    156,119,324 bytes,  4.009 sec.,  0.896 sec., zstd -5
    159,683,217 bytes,  5.575 sec.,  1.585 sec., brotli -q 3 -w 24
    160,316,452 bytes, 17.487 sec.,  4.590 sec., 7z -mx1 -t7z
    162,791,690 bytes,  4.888 sec.,  1.634 sec., brotli -q 2 -w 24
    163,971,199 bytes,  2.423 sec.,  0.902 sec., zstd -4
    164,415,042 bytes,  2.312 sec.,  0.897 sec., zstd -3
    169,893,453 bytes,  9.400 sec.,  2.588 sec., rar -m1 -ma5
    174,878,733 bytes,  1.911 sec.,  0.890 sec., zstd -2
    179,431,420 bytes,  9.460 sec.,  2.613 sec., 7z -mx1 -tzip
    181,138,030 bytes,  1.478 sec.,  0.869 sec., zstd -1
    204,712,777 bytes,  2.869 sec.,  1.872 sec., brotli -q 1 -w 24
    482,066,186 bytes, 19.271 sec., xx.xxx sec., xwrt -l0 -b255 +d
    tor 0.5
    lzturbo 1.2
    tor-small 0.4a
    brotli 1.0.4
    zstd 1.3.5
    nanozip 0.09
    7-zip 18.05
    freearc 0.67 (March 15 2014)
    rar 5.60
    xwrt 3.4

    All tests single core except zstd decompression.
    Last edited by Sportman; 17th July 2018 at 12:50. Reason: Added tor 0.5, rar, 7-zip, nanozip, xwrt

  6. The Following User Says Thank You to Sportman For This Useful Post:

    2pact (8th July 2018)

  7. #6
    Member
    Join Date
    May 2012
    Location
    usa
    Posts
    24
    Thanks
    5
    Thanked 0 Times in 0 Posts
    Your benchmark results are truly helpful. Thanks!!!!!!!!!
    Also, for zstd did you create a dictionary?

  8. #7
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    48
    Thanks
    25
    Thanked 10 Times in 7 Posts
    What is tor-small? Never heard of it.

  9. #8
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    772
    Thanks
    63
    Thanked 270 Times in 190 Posts
    Quote Originally Posted by 2pact View Post
    for zstd did you create a dictionary?
    No, I used default settings.
    Last edited by Sportman; 7th July 2018 at 22:35.

  10. #9
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    772
    Thanks
    63
    Thanked 270 Times in 190 Posts
    Quote Originally Posted by WinnieW View Post
    What is tor-small? Never heard of it.
    It was located in Bulat's standalone Tornado archive in the time freearc.org was up.

    Version 0.4:
    https://web.archive.org/web/20100922.../tornado04.zip

    Version 0.4a (version I used):
    https://web.archive.org/web/20140325...tornado04a.zip

    Version 0.5 (only tor):
    https://web.archive.org/web/20140325.../tornado05.zip

  11. The Following User Says Thank You to Sportman For This Useful Post:

    WinnieW (7th July 2018)

  12. #10
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts

  13. #11
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    772
    Thanks
    63
    Thanked 270 Times in 190 Posts
    Quote Originally Posted by Shelwien View Post
    Added.

  14. #12
    Member
    Join Date
    May 2012
    Location
    usa
    Posts
    24
    Thanks
    5
    Thanked 0 Times in 0 Posts
    I think mozjpeg is the best image compressor right now, atleast in my case.

  15. #13
    Member
    Join Date
    May 2012
    Location
    usa
    Posts
    24
    Thanks
    5
    Thanked 0 Times in 0 Posts
    What do you mean by xx.xxx sec for xwrt?

  16. #14
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    xwrt is actually a preprocessor, it can be used with any other codecs.
    Sportman somehow only tested it with built-in zlib compression.
    As to pictures (png/jpg), I'd suggest precomp; maybe even precomp/preflate.

  17. The Following 2 Users Say Thank You to Shelwien For This Useful Post:

    2pact (8th July 2018),Mike (9th July 2018)

  18. #15
    Member
    Join Date
    Apr 2018
    Location
    Indonesia
    Posts
    24
    Thanks
    7
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by 2pact View Post
    Your benchmark results are truly helpful. Thanks!!!!!!!!!
    Also, for zstd did you create a dictionary?
    Maybe you want to try paq8px or paq8pxd or cmix...they are also in this forum...

  19. #16
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    772
    Thanks
    63
    Thanked 270 Times in 190 Posts
    Quote Originally Posted by 2pact View Post
    What do you mean by xx.xxx sec for xwrt?
    I get xwrt decode error:
    File corrupted (s.size()>WORD_MAX_SIZE)!
    Not enough memory!

  20. The Following User Says Thank You to Sportman For This Useful Post:

    2pact (8th July 2018)

  21. #17
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    772
    Thanks
    63
    Thanked 270 Times in 190 Posts
    Quote Originally Posted by Shelwien View Post
    Sportman somehow only tested it with built-in zlib compression.
    Because I only include what compress till 30 seconds, otherwise it become to slow for very big HTML files or HTML big data sets.

    Added XWRT & Lzturbo and XWRT & Tornado and XWRT & tor-small and XWRT & Nanozip and XWRT & RAR and XWRT & 7-zip ZIP.
    Last edited by Sportman; 8th July 2018 at 14:23.

  22. #18
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    47
    Thanks
    5
    Thanked 22 Times in 14 Posts
    I've had good success with mcm on log files (compared to 7zip ppmd); not sure how it would perform here.

  23. #19
    Member
    Join Date
    Dec 2014
    Location
    Berlin
    Posts
    29
    Thanks
    35
    Thanked 26 Times in 12 Posts
    How about ZPAQ archiver http://mattmahoney.net/dc/zpaq.html which will dedupe and automatically choose a compression algorithm based on some short tests on the files.

  24. #20
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts
    XWRT is excellent and underexposed. It should've been the new web compressor (combined with a low overhead codec) instead of brotli. It beats brotli badly. It's weird that XWRT had just been sitting there for years and Google, Mozilla, Microsoft, and Apple ignored it.

    Unless Zstd with a good HTML, CSS, JS dictionary beats XWRT.

  25. #21
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by SolidComp View Post
    XWRT is excellent and underexposed.
    Is it streamable during decoding? What about encoding? How much ram does it use at decoding?

    (Browser developers would not integrate non-streamable decoders or decoders that allocate uncontrolled or large amounts of ram.)

    You are chaining it with other compressors -- what happens if you chain it with brotli or zstd?

    Quote Originally Posted by SolidComp View Post
    Unless Zstd with a good HTML, CSS, JS dictionary beats XWRT.
    Classic Brotli has user-defined dictionaries as well.

    In Shared Brotli there is an even more powerful dictionary modeling as we have added the transform modeling and unique length-distance encoding to user-defined dictionaries as well (the mechanisms we use for static dictionary). These simple mods typically make user-defined LZ77 dictionaries about 10-20 % more powerful -- for example if you would get 10 % savings from a custom LZ77 dictionary, you can get 11-12 % savings from a custom Shared Brotli dictionary. (We also added context-modeling for reordering the dictionary for some anticipated further savings, but no quantitative results on it yet.)

  26. #22
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Is it streamable during decoding? What about encoding? How much ram does it use at decoding?

    (Browser developers would not integrate non-streamable decoders or decoders that allocate uncontrolled or large amounts of ram.)

    You are chaining it with other compressors -- what happens if you chain it with brotli or zstd?



    Classic Brotli has user-defined dictionaries as well.

    In Shared Brotli there is an even more powerful dictionary modeling as we have added the transform modeling and unique length-distance encoding to user-defined dictionaries as well (the mechanisms we use for static dictionary). These simple mods typically make user-defined LZ77 dictionaries about 10-20 % more powerful -- for example if you would get 10 % savings from a custom LZ77 dictionary, you can get 11-12 % savings from a custom Shared Brotli dictionary. (We also added context-modeling for reordering the dictionary for some anticipated further savings, but no quantitative results on it yet.)
    You're right that we should be able to chain it to brotli. On the brotli user-defined dictionaries, how do we create them? Where are the instructions? Zstd has the instructions for training, generating, and using a dictionary right there on their homepage, but with brotli all I saw were build instructions with premake and all that jazz. Where are the user docs?

  27. #23
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    772
    Thanks
    63
    Thanked 270 Times in 190 Posts
    Input:
    4,104,832,596 bytes, 50,000 HTML pages + headers
    https://commoncrawl.s3.amazonaws.com...-00639.warc.gz

    Output:
    Code:
      638,628,491 bytes, 151.434 sec.,  9.531 sec., rar -m2 -ma5
      646,609,659 bytes,  33.043 sec.,  5.023 sec., lzturbo -32 -b1024
      649,879,350 bytes, 114.606 sec., 31.446 sec., arc -m3
      652,615,389 bytes, 176.503 sec.,  6.291 sec., brotli -q 7 -w 24
      675,786,114 bytes,  64.586 sec., 14.663 sec., arc -m2
      677,664,500 bytes, 154.458 sec.,  3.412 sec., zstd -14
      678,840,512 bytes,  85.041 sec.,  6.108 sec., brotli -q 6 -w 24
      678,960,551 bytes, 122.182 sec., 15.847 sec., 7z -mx4 -t7z
      679,967,079 bytes, 126.039 sec.,  3.445 sec., zstd -13
      683,375,896 bytes,  62.438 sec.,  3.476 sec., zstd -12
      684,497,537 bytes,  33.387 sec.,  4.478 sec., lzturbo -32
      692,992,930 bytes,  78.352 sec., 11.648 sec., nz -cdP -m1024m
      695,149,591 bytes,  63.323 sec.,  6.264 sec., brotli -q 5 -w 24
      697,952,704 bytes,  55.730 sec.,  3.471 sec., zstd -11
      699,764,448 bytes,  44.347 sec.,  3.464 sec., zstd -10
      701,111,732 bytes,  73.735 sec., 11.164 sec., nz -cdP
      701,702,322 bytes,  86.982 sec., 16.730 sec., 7z -mx3 -t7z
      702,414,691 bytes,  35.063 sec.,  3.492 sec., zstd -9
      707,020,525 bytes,  28.819 sec.,  3.461 sec., zstd -8
      709,451,933 bytes,  27.278 sec.,  3.486 sec., zstd -7
      722,729,124 bytes,  20.239 sec.,  3.601 sec., zstd -6
      734,191,921 bytes,  34.416 sec.,  6.434 sec., brotli -q 4 -w 24
      756,188,016 bytes,  12.405 sec., 13.413 sec., nz -cF -m1024m
      756,225,594 bytes,  87.151 sec., 18.418 sec., 7z -mx2 -t7z
      757,961,627 bytes,  12.209 sec., 13.228 sec., nz -cF
      759,237,267 bytes,  16.883 sec.,  3.685 sec., zstd -5
      763,168,591 bytes,  23.112 sec.,  6.472 sec., brotli -q 3 -w 24
      776,060,002 bytes,  19.364 sec.,  6.648 sec., brotli -q 2 -w 24
      788,313,067 bytes,   9.536 sec.,  3.754 sec., zstd -4
      790,161,648 bytes,   9.266 sec.,  3.685 sec., zstd -3
      818,253,204 bytes,  42.320 sec., 11.156 sec., rar -m1 -ma5
      843,409,165 bytes,  86.456 sec., 21.271 sec., 7z -mx1 -t7z
      851,100,961 bytes,   7.765 sec.,  3.753 sec., zstd -2
      878,916,253 bytes,   5.995 sec.,  3.652 sec., zstd -1
      930,837,437 bytes, 122.964 sec., 14.869 sec., tor -7 -b1024
      952,798,194 bytes,  89.072 sec., 17.272 sec., tor-small -7 -b1024
      955,192,051 bytes,  45.026 sec., 12.088 sec., 7z -mx1 -tzip
      956,171,387 bytes,  60.056 sec., 15.273 sec., tor -6 -b1024
    1,071,815,433 bytes,  12.787 sec.,  8.547 sec., brotli -q 1 -w 24
    tor 0.5
    lzturbo 1.2, beta, beta II
    tor-small 0.4a
    brotli 1.0.4
    zstd 1.3.5
    nanozip 0.09
    7-zip 18.05
    freearc 0.67 (March 15 2014)
    rar 5.60

    All tests single core except zstd decompression.
    Last edited by Sportman; 24th July 2018 at 15:58.

  28. The Following User Says Thank You to Sportman For This Useful Post:

    Mike (12th July 2018)

  29. #24
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by SolidComp View Post
    ... but with brotli all I saw were build instructions with premake and all that jazz. Where are the user docs?
    Would you consider filing an issue on the github repository about the unclarity?

  30. #25
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Would you consider filing an issue on the github repository about the unclarity?
    Done. https://github.com/google/brotli/issues/697

    I know you've asked me to do this before -- sorry it took me so long...

  31. #26
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Would you consider filing an issue on the github repository about the unclarity?
    By the way, is the custom dictionary part of the payload of the compressed file, or it a standalone dictionary that the recipient needs to have in order to decompress the file?

  32. #27
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    772
    Thanks
    63
    Thanked 270 Times in 190 Posts
    Input:
    57,269,226,154 bytes, Alexa top 1,000,000 HTML pages + headers

    Output:
    Code:
     6,682,500,126 bytes, 2683.736 sec.,  134.560 sec., rar -m2 -ma5 -md1g
     7,203,605,682 bytes, 2398.672 sec., 1494.298 sec., bsc -m0
     7,406,872,586 bytes, 1208.081 sec., 2010.735 sec., bsc -m6
     7,439,216,266 bytes,  301.624 + 469.578 sec., 85.830 + 68.479 sec., srep -m0 + lzturbo -32 -b1024
     7,442,736,590 bytes,  596.676 sec.,   86.695 sec., lzturbo (beta II) -32 -b1024
     7,462,762,823 bytes,  594.145 sec.,   86.675 sec., lzturbo (beta) -32 -b1024
     7,529,586,920 bytes, 1091.866 sec., 1890.670 sec., bsc -m5
     7,681,446,412 bytes, 2645.614 sec., 2458.265 sec., arc -m3
     7,689,994,916 bytes, 2629.968 sec.,  254.009 sec., nz -cD -m1024m
     7,721,814,081 bytes,  486.150 sec.,   95.146 sec., lzturbo (v1.2) -32 -b1024
     7,744,316,303 bytes, 2073.887 sec.,  132.313 sec., rar -m2 -ma5
     7,751,700,878 bytes,  966.680 sec., 1738.634 sec., bsc -m4
     7,708,706,413 bytes, 2633.326 sec.,  257.811 sec., nz -cDP -m1024m
     7,819,349,725 bytes, 2484.101 sec.,  256.802 sec., nz -cDp -m1024m
     7,968,006,316 bytes, 2693.067 sec.,   93.510 sec., brotli -q 7 -w 24
     8,306,530,796 bytes, 2232.525 sec.,   53.801 sec., ztsd -14
     8,347,490,076 bytes, 1843.954 sec.,   54.127 sec., ztsd -13
     8,395,993,618 bytes,  987.160 sec., 1549.477 sec., bsc -m3
     8,402,717,390 bytes, 1336.677 sec.,   90.671 sec., brotli -q 6 -w 24
     8,402,867,094 bytes, 1628.416 sec.,  226.449 sec., 7z -mx4 -t7z
     8,408,749,340 bytes,  986.207 sec.,   55.216 sec., ztsd -12
     8,458,196,886 bytes,  895.097 sec., 1258.219 sec., arc -m2
     8,576,544,152 bytes,  883.037 sec.,   54.046 sec., ztsd -11
     8,603,328,832 bytes,  709.001 sec.,   54.126 sec., ztsd -10
     8,634,534,227 bytes, 1160.808 sec.,  237.742 sec., 7z -mx3 -t7z
     8,637,690,336 bytes,  970.064 sec.,   93.386 sec., brotli -q 5 -w 24
     8,643,759,419 bytes,  569.279 sec.,   54.046 sec., ztsd -9
     8,722,553,984 bytes,  457.780 sec.,   54.144 sec., ztsd -8
     8,945,785,642 bytes, 1459.253 sec.,  256.516 sec., nz -cdP -m1024m
     8,974,062,112 bytes, 1099.432 sec.,  251.994 sec., 7z -mx2 -t7z
     8,986,125,666 bytes,  311.708 sec.,   56.016 sec., ztsd -6
     9,020,251,494 bytes,  998.225 sec.,  257.318 sec., nz -cd -m1024m
     9,105,934,610 byte,  1310.830 sec.,  259.954 sec., nz -cdp -m1024m
     9,276,336,130 bytes,  495.980 sec.,   95.928 sec., brotli -q 4 -w 24
     9,367,172,809 bytes,  227.601 sec.,  242.428 sec., nz -cF -m1024m
     9,397,787,657 bytes,  203.182 sec.,   81.220 sec., lzturbo (beta II) -31 -b1024
     9,409,591,496 bytes,  941.024 sec.,  360.078 sec., zpaq -m1
     9,436,992,864 bytes,  251.616 sec.,   57.529 sec., ztsd -5
     9,474,960,075 bytes, 1051.552 sec.,  275.146 sec., 7z -mx1 -t7z
     9,493,448,784 bytes,  206.122 sec.,   81.006 sec., lzturbo (beta) -31 -b1024
     9,684,434,905 bytes,  358.149 sec.,   97.541 sec., brotli -q 3 -w 24
     9,861,326,399 bytes,  287.412 sec.,  146.402 sec., arc -m1
     9,912,692,170 bytes,  299.844 sec.,  101.052 sec., brotli -q 2 -w 24
     9,917,117,752 bytes,  151.228 sec.,   59.004 sec., ztsd -4
     9,940,159,062 bytes,  144.961 sec.,   57.875 sec., ztsd -3
    10,230,258,393 bytes,  657.482 sec.,   52.604 sec., lzturbo (beta) -22 -b1024
    10,404,975,395 bytes,  133.783 sec.,   85.550 sec., lzturbo (beta II) -30 -b1024
    10,421,145,769 bytes,  589.954 sec.,  157.898 sec., rar -m1 -ma5 -md1g
    10,425,235,935 bytes,  588.781 sec.,  158.155 sec., rar -m1 -ma5
    10,464,649,227 bytes,  624.864 sec.,   57.328 sec., ztsd -5 --format=gzip
    10,549,597,390 bytes,  120.453 sec.,   58.298 sec., ztsd -2
    10,605,605,829 bytes, 1141.219 sec.,  205.593 sec., tor-small -7 -b1024
    10,688,066,823 bytes,  141.622 sec.,   84.939 sec., lzturbo (beta) -30 -b1024
    10,727,646,134 bytes,  580.659 sec.,  159.395 sec., 7z -mx1 -tzip
    10,913,491,962 bytes,   94.356 sec.,   57.048 sec., ztsd -1
    11,354,422,547 bytes,  196.824 sec.,   55.218 sec., lzturbo (beta) -21 -b1024
    11,629,817,134 bytes,  164.313 sec.,  174.498 sec., nz -cf -m1024m
    12,189,230,146 bytes,  336.405 sec.,   57.918 sec., ztsd -1 --format=gzip
    12,378,919,264 bytes,  174.560 sec.,  115.417 sec., brotli -q 1 -w 24
    12,791,392,694 bytes,  148.285 sec.,  121.207 sec., brotli -q 0 -w 24
    12,941,717,182 bytes,  623.304 sec.,   52.116 sec., lzturbo (beta) -12 -b1024
    12,966,122,553 bytes,  123.862 sec.,   59.483 sec., lzturbo (beta) -20 -b1024
    13,774,047,393 bytes,  185.359 sec.,   54.117 sec., lzturbo (beta) -11 -b1024
    14,558,206,392 bytes,  126.632 sec.,   55.053 sec., lzturbo (beta) -10 -b1024
    46,936,255,308 bytes,  301.624 sec.,   68.479 sec., srep -m0
    bsc 3.1.0
    lzturbo 1.2 + beta
    tor-small 0.4a
    brotli 1.0.4
    zstd 1.3.5
    nanozip 0.09
    7-zip 18.05
    freearc 0.67 (March 15, 2014)
    rar 5.60
    zpaq 7.15
    srep 3.93a beta (October 11, 2014)

    All tests single core except zstd decompression.
    Last edited by Sportman; 27th July 2018 at 23:47. Reason: Added srep

  33. #28
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    377
    Thanks
    139
    Thanked 198 Times in 108 Posts
    How srep performs on this set? +some compressor
    KZo


  34. #29
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    772
    Thanks
    63
    Thanked 270 Times in 190 Posts
    Quote Originally Posted by kaitz View Post
    How srep performs on this set? +some compressor
    Added.

  35. The Following User Says Thank You to Sportman For This Useful Post:

    kaitz (15th July 2018)

  36. #30
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    rar results seem pretty good... I want to suggest adding some options:
    Code:
    327,445,811 4.rar // rar560 a -ma5 -md1g 4.rar !arc
    275,572,136 3.rar // rar560 a -ma5 -md1g -oi1 3.rar !arc
    275,499,739 2.rar // rar560 a -ma5 -md1g -oi1 -ts- 2.rar !arc
    275,452,675 1.rar // rar560 a -ma5 -md1g -oi1 -ts- -qo- 1.rar !arc

Page 1 of 2 12 LastLast

Similar Threads

  1. Compressing mp3 offsets
    By Shelwien in forum Data Compression
    Replies: 13
    Last Post: 13th September 2017, 02:26
  2. Compressing pi
    By Matt Mahoney in forum Data Compression
    Replies: 24
    Last Post: 26th September 2016, 19:17
  3. compressing animation frames
    By Cosmin3 in forum Data Compression
    Replies: 24
    Last Post: 15th June 2016, 19:27
  4. XWRT (XML-WRT) - an efficient XML/HTML/text compressor
    By inikep in forum Data Compression
    Replies: 14
    Last Post: 17th November 2015, 21:49
  5. Compressing DNG
    By mohanohi in forum Data Compression
    Replies: 13
    Last Post: 9th July 2014, 20:25

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •