Results 1 to 22 of 22

Thread: 7-zip parameter optimization

  1. #1
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    181
    Thanks
    74
    Thanked 10 Times in 10 Posts

    7-zip parameter optimization

    just did quick compression test...used intel Lan drivers:
    Click image for larger version. 

Name:	oIwmBRw.png 
Views:	53 
Size:	7.3 KB 
ID:	6614

    seems like normal 7-zip does best compression atm.....if you dont consider time as a deciding factor.

    Is there a way to go beyond level 9 with 7-Zip ZS(LZMA2,fast)? to increase compression even more? Click image for larger version. 

Name:	5qVEjv4.png 
Views:	82 
Size:	12.8 KB 
ID:	6615

  2. #2
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    There're some ways:
    1. mx9 uses 64M window, you can specify up to 1.5G manually via -md
    2. Use lzma rather than lzma2, and no more than 1-2 threads
    3. Add -myx=9 switch
    4. manually specify more efficient BCJ2 scheme for exes; https://encode.ru/threads/15-7-Zip?p...ll=1#post57538
    5. optimize lc/pb/lp and fb settings
    6. Recompile lzma with tweaked parsing optimization options, like "#define kNumOpts (1 << 12)" to 1<<15

  3. #3
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    181
    Thanks
    74
    Thanked 10 Times in 10 Posts
    when i try to use -myx=9 i get error saying: Click image for larger version. 

Name:	JsDhzEn.png 
Views:	38 
Size:	8.2 KB 
ID:	6616

    I am using it like this: Click image for larger version. 

Name:	DmklVE8.png 
Views:	52 
Size:	22.6 KB 
ID:	6617 ...how do i add -myx=9 switch properly?

    Edit: i am using 7-Zip 19.00 64bit

  4. #4
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Just read the help on that parameters string... It says "omit -m", so write yx=9 there.

  5. #5
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    181
    Thanks
    74
    Thanked 10 Times in 10 Posts
    interestingly, yx=9 produces larger file size, by around 150kB
    If anyone wants to give it a try, here is file for testing: LAN_Intel_19030_LZMA_ultra_1536MB_wordSize273_soli d_1t.7z

  6. #6
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Code:
    7z a -mx=9 -bb3 -mqs=on ^
    -m0=bcj2:d24 ^
    -m1=lzma:mt2:d=384M:lc8:pb2:lp0:fb273:mc999 ^
    -m2=delta:4 ^
    -m3=lzma:mt2:d24:lc0:pb2:lp2:fb273 ^
    -m4=lzma:mt2:d24:lc0:pb2:lp2:fb273 ^
    -m5=lzma2:mt2:d24:lc0:pb0:lp0:fb273 ^
    -mb00s0:1 -mb00s1:2 -mb00s2:4 -mb00s3:5 -mb02s0:3 ^
    DP_LAN_Intel_19030.7z DP_LAN_Intel_19030
    
    21,852,665 DP_LAN_Intel_19030_LZMA_ultra_1536MB_wordSize273_solid_1t.7z0
    21,291,826 DP_LAN_Intel_19030.7z

  7. #7
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts
    In the 7-Zip file manager you can also add a=3 which should improve it a bit. Fast LZMA2 rarely compresses as well as LZMA2 at equal window sizes because the match-finder is a block algorithm.

    -ma=3 applies on the command line but it also sets the window size by the compression level: x1 = 1 Mb, x9 = 256 Mb

  8. #8
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    181
    Thanks
    74
    Thanked 10 Times in 10 Posts
    @Shelwien i have never tried all those switches at once....do i dropout "m" on all like for myx....or this is only exclusive for command line only?

    @Conor, just to clear for my self....is window size actually dictionary size? or those two are completely different parameters?

    edit: how do i make forum user names active in a post?

  9. #9
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    I tried inserting "x=9 qs=on 0=bcj2:d24 1=lzma:mt2:d=384M:lc8:pb2:lp0:fb273:mc999 2=delta:4 3=lzma:mt2:d24:lc0:pb2:lp2:fb273 4=lzma:mt2:d24:lc0:pb2:lp2:fb273 5=lzma2:mt2:d24:lc0:pb0:lp0:fb273 b00s0:1 b00s1:2 b00s2:4 b00s3:5 b02s0:3"
    into 7zFM parameters and it worked. "-bb3" only affects console output, so you can skip it.

    > is window size actually dictionary size? or those two are completely different parameters?

    7-zip/lzma calls the data that can be copied by LZ decoder as "dictionary".
    Its not exactly wrong, but with time, new things were added (like dictionary preprocessing and external dictionary for LZ),
    so now its better to use "window" and "window size" for this.

    > how do i make forum user names active in a post?

    With bbcode tags, but just don't - this forum engine is old and doesn't have these features really.

  10. The Following User Says Thank You to Shelwien For This Useful Post:

    dado023 (14th May 2019)

  11. #10
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    181
    Thanks
    74
    Thanked 10 Times in 10 Posts
    Quote Originally Posted by Shelwien View Post
    I tried inserting "x=9 qs=on 0=bcj2:d24 1=lzma:mt2:d=384M:lc8b2:lp0:fb273:mc999 2=delta:4 3=lzma:mt2:d24:lc0b2:lp2:fb273 4=lzma:mt2:d24:lc0b2:lp2:fb273 5=lzma2:mt2:d24:lc0b0:lp0:fb273 b00s0:1 b00s1:2 b00s2:4 b00s3:5 b02s0:3"
    into 7zFM parameters and it worked. "-bb3" only affects console output, so you can skip it.

    > is window size actually dictionary size? or those two are completely different parameters?

    7-zip/lzma calls the data that can be copied by LZ decoder as "dictionary".
    Its not exactly wrong, but with time, new things were added (like dictionary preprocessing and external dictionary for LZ),
    so now its better to use "window" and "window size" for this.

    > how do i make forum user names active in a post?

    With bbcode tags, but just don't - this forum engine is old and doesn't have these features really.
    As you recommend i wont use bbcode tags.

    Regarding parameters you provided for 7zFM, this is by far the highest compression with 7zip i ever saw:
    Click image for larger version. 

Name:	CsxC5jg.png 
Views:	48 
Size:	12.5 KB 
ID:	6618
    I feel i discovered something completely new , exciting...i just wonder how come 7zip doesnt have this setting maybe under different name, for example "Extreme", because even the compression speed is quite acceptable


    Another question: are these parameters good for all types of files(content) or you tuned this specifically for these hardware drivers?

  12. #11
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > or you tuned this specifically for these hardware drivers?

    Most of it would be useful for any exes. Except for BCJ2:d parameter, which depends on maximum exe size in archive - http://nishi.dreamhosters.com/u/method.htm#BCJ2
    and similarly the :d parameters in all lzma streams.

    Also, qs=on enables extension sort, which is usually better for compression, but NTFS uses name sort, so qs=on is kinda hard for the filesystem (fragmentation etc).

    As to lzma, in theory lzma2 should be better, but unfortunately along with improvements, Igor also decided to put a limit on lc+lp value
    for speed optimization, so best compression is now only possible for lzma, but not lzma2.

  13. The Following User Says Thank You to Shelwien For This Useful Post:

    dado023 (15th May 2019)

  14. #12
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    181
    Thanks
    74
    Thanked 10 Times in 10 Posts
    you mention LZMA can achieve better compression......do i use same parameters you provided before in 7zFM with just LZMA method selected in the dialog box.....or different parameters are needed in this case?

  15. #13
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    181
    Thanks
    74
    Thanked 10 Times in 10 Posts
    @Conor, i tried using a=3 but it doesn't accept this parameter in 7zFM

  16. #14
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > you mention LZMA can achieve better compression......
    > do i use same parameters you provided before in 7zFM with just LZMA method selected in the dialog box.....
    > or different parameters are needed in this case?

    Parameters can be tweaked based on data type.
    As example, for tables of 32-bit values, its better to use lp2:pb2 and lower lc, maybe lc0 (like what I did for bcj2 addr streams).
    For text and exes its usually best to use lc8, but there're cases where different lc value can be optimal
    (default is lc3; lc0 means using order0 CM for literals, lc8 means order1).
    Also, if lc+lp<=4, its better to use lzma2.

    Btw, .7z format also supports ppmd codec, which has its own parameters and can compress texts better than lzma.

  17. The Following 2 Users Say Thank You to Shelwien For This Useful Post:

    dado023 (16th May 2019),introspec (16th May 2019)

  18. #15
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    181
    Thanks
    74
    Thanked 10 Times in 10 Posts
    after reading this https://www.sami-lehtinen.net/blog/l...nd-compression i am even more interested in "window size"...if i understand correct, windows size works as a sort of deduplication technique

    @Shelwien you mention windows size can be up to 1.5G using -md parameter, right? ...if m is omitted i assume its actually d

    Code:
    "x=9 qs=on 0=bcj2:d24 1=lzma:mt2:d=384M:lc8:pb2:lp0:fb273:mc999 2=delta:4 3=lzma:mt2:d24:lc0:pb2:lp2:fb273 4=lzma:mt2:d24:lc0:pb2:lp2:fb273 5=lzma2:mt2:d24:lc0:pb0:lp0:fb273 b00s0:1 b00s1:2 b00s2:4 b00s3:5 b02s0:3"
    d24 ? or d=384M ? ...do i replace those number in above parameter set with for example d31 and d=1536M to make it work and get a bit better compression ratio?
    Last edited by dado023; 19th May 2019 at 15:11.

  19. #16
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > if i understand correct, window size works as a sort of deduplication technique

    Technically, deduplication methods are specialized compression algorithms tuned
    to use less memory comparing to LZ with similar window size.
    The most popular one is probably srep by Bulat Ziganshin.
    http://web.archive.org/web/201612231...arch/SREP.aspx
    http://freearc.dreamhosters.com/srep393a.zip

    > you mention windows size can be up to 1.5G using -md parameter, right? ...if m is omitted i assume its actually d

    Yes, although its 7-zip that limits the "dictionary size" to 2^n and 3*2^n values.
    lzma format supports match distances up to 4G, standard lzma encoder supports up to ~2000M due to double buffer.

    > d24 ? or d=384M ?

    d24 is 7-zip syntax for setting window size to 2^n bytes, ie 2^24=16M in this case.

    > ...do i replace those number in above parameter set with for example d1024
    > and d=1024M to make it work and get a bit better compression ratio?

    d30 or d=1G or d=1024M would work, d1024 is probably not what you want.
    Just read the 7-zip help: http://nishi.dreamhosters.com/u/meth...DictionarySize

    But larger "dictionary" won't improve compression if its larger than data size.

  20. #17
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    181
    Thanks
    74
    Thanked 10 Times in 10 Posts
    @Shelwien

    thank you so much for your reply. You are very informative.

    >But larger "dictionary" won't improve compression if its larger than data size.
    Now i understand,.....there is no point of using 1024M if all files together are for example 700M, right?
    However, does it reduce compression if dict size is above total file size?


    I am not sure how to use srep with 7zFM, if its usable at all with it....i guess that's out of the scope of 7z

  21. #18
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > Now i understand,.....there is no point of using 1024M if all files together are for example 700M, right?

    7-zip allows a limited set of dictionary sizes (because :d value has to be stored in archive),
    so unfortunately its not as simple as setting dictionary size to data size.
    So for 700M of data, you can use d=768M, but for 800M of data it might be better to use 1024M.

    > However, does it reduce compression if dict size is above total file size?

    No, but it might affect memory usage at compression.
    lzma wants 10.5*d of memory _per instance_, so its easy enough to make it use more memory than you have.

    > I am not sure how to use srep with 7zFM,
    > if its usable at all with it....i guess that's out of the scope of 7z

    7-zip supports external codecs, so its possible.
    If it doesn't already exist, I can probably make one, but it would be better
    to check if you need it first, by testing compression with standalone srep.

    There's also freearc which has it integrated (along with lzma and other codecs).

  22. #19
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    181
    Thanks
    74
    Thanked 10 Times in 10 Posts
    @Shelwien

    now i understand dict size and memory tradeoff much better. Thank you so much.

    >If it doesn't already exist, I can probably make one, but it would be better

    i don't need it personally as a must nor professionally, i am just learning a bit more about compression and how to improve it with better parameters, ...thanks to you, now i know how to compress at much higher ratios with 7zFM. I am grateful for that.
    I am not sure how whole thing with external codecs works, but it would be awesome if you could improve compression ratio even more with srep combined with 7zFM, the whole world is always looking for higher compression

    As for freearc, it kinda doesn't look maintained, so for long term, i don't want to have my files in that format

  23. #20
    Member
    Join Date
    Jul 2016
    Location
    Russia
    Posts
    21
    Thanks
    13
    Thanked 7 Times in 6 Posts
    7-ZIP finetuner
    http://krzyrio.blogspot.com/2017/01/...finetuner.html

    automatic search for the best parameters, but alas, in 1 thread mode (search), because of what is slow.
    Last edited by zubzer0; 29th May 2019 at 21:31.

  24. #21
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    181
    Thanks
    74
    Thanked 10 Times in 10 Posts
    @zubzer0

    Does it work with latest 7zip 19.0 ?

  25. #22
    Member
    Join Date
    Jul 2016
    Location
    Russia
    Posts
    21
    Thanks
    13
    Thanked 7 Times in 6 Posts
    @dado023

    Yes, seems no problem. But as it was said, it lacks a multithreaded search ...
    I tried to contact the author to contribute this idea, but he disappeared.
    "7-ZIP finetuner" author https://encode.ru/members/3822-Mr_KrzYch00 (Last Activity 18th September 2018 )

    p.s. Perhaps someone else will take up writing alternatives to this program?

Similar Threads

  1. The 7-zip compression selection optimization thread
    By SvenBent in forum Data Compression
    Replies: 14
    Last Post: 1st March 2017, 13:18
  2. GIF optimization
    By lorents17 in forum Data Compression
    Replies: 9
    Last Post: 27th July 2013, 19:57
  3. C++ optimization guide
    By Bulat Ziganshin in forum The Off-Topic Lounge
    Replies: 4
    Last Post: 25th May 2012, 12:23
  4. Parameter optimization
    By toffer in forum Data Compression
    Replies: 42
    Last Post: 30th June 2008, 00:53
  5. QuickLZ ZIP - new zip/deflate library
    By Lasse Reinhold in forum Forum Archive
    Replies: 23
    Last Post: 1st October 2007, 22:08

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •