Page 1 of 2 12 LastLast
Results 1 to 30 of 55

Thread: 7-Zip binaries using the Fast LZMA2 lib

  1. #1
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts

    7-Zip binaries using the Fast LZMA2 lib

    Attached are 64-bit 7-Zip binaries which use the new library. I'll put the source on github soon. They are v 17.01 because I haven't merged v 18 yet.

    In the GUI, the initial word size setting may be loaded from the registry. It's best to initialize by changing the level because the default values are higher than normal 7z.

    The compression levels use the same dictionary size as normal 7z, which results in a little less compression because the MF is a block algorithm. Bumping the level up one tends to even it out. Size at level 8 & 9 is higher than normal 7z at 128Mb.

    Note the low multithread memory usage

    EDIT: attached an updated version due to a bug in my 7-zip stream handler which caused it to infinite loop after the library finished compressing, if dict > input size. This version has smoother progress updates too.

    EDIT 2: Fixed a crash when progress is set to NULL, e.g. with filters like BCJ2

    EDIT 3: It has now passed plenty of automated testing and is released on github for x86 and x64: https://github.com/conor42/7-Zip-FL2/releases/
    Last edited by Conor; 9th March 2018 at 15:26.

  2. The Following 6 Users Say Thank You to Conor For This Useful Post:

    Bulat Ziganshin (16th January 2018),Gonzalo (16th January 2018),JaegerFox (17th September 2018),load (5th March 2018),Stephan Busch (1st March 2018),Zonder (17th January 2018)

  3. #2
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    imho, it will be better to put it into your original thread. otherwise, a few months later it will become a mess, Or, crosslink these two topics in their header posts

    About modified 7-zip package - i propose to publish your codec as separate one, distinct from original lzma2. This will allow people to easily drop your dll into mainline 7-zip installations as well as other programs (freearc/powerarchiver can compress to .7z, RAR/WinZip can extract .7z and so on), compare lzma2 and your codec and choose which one to use for each run. It's especially important taking into account that your codec isn't absolute win so sometimes original lzma2 is preferable

    I.e. instead of providing modified 7-zip package i would prefer if you will provide just libflzma2-x64.dll which one can drop into any 7z.dll-based program and immediately get access to your codec without sacrificing the original lzma2

    If you don't know how to arrange that - i can help

  4. #3
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    imho, it will be better to put it into your original thread. otherwise, a few months later it will become a mess, Or, crosslink these two topics in their header posts

    About modified 7-zip package - i propose to publish your codec as separate one, distinct from original lzma2. This will allow people to easily drop your dll into mainline 7-zip installations as well as other programs (freearc/powerarchiver can compress to .7z, RAR/WinZip can extract .7z and so on), compare lzma2 and your codec and choose which one to use for each run. It's especially important taking into account that your codec isn't absolute win so sometimes original lzma2 is preferable

    I.e. instead of providing modified 7-zip package i would prefer if you will provide just libflzma2-x64.dll which one can drop into any 7z.dll-based program and immediately get access to your codec without sacrificing the original lzma2

    If you don't know how to arrange that - i can help
    Or merge it with milky's 7z-zstd which already includes several alternative algorithms. I guess/hope that it will opaque the original 7z in due time.
    Maybe publish it as LZMA3 or something like that. Perhaps change ari to ANS so it won't remain compatible with LZMA2 and boost a little more the speed?

  5. #4
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts
    Maybe the threads can be merged. Not sure if the forum software supports that.

    It will be added to 7-Zip-zstd. We already tried to add Radyx but there were compilation problems due to use of C++14. I gave the codec a unique id and added code to replace it with the LZMA2 id when writing, so the resulting .7z file can be decoded with mainline. After some more testing it should be okay to add it to milky's code.

    Right now I'm adding progress metering to the library because updates in 7z are granular when using a large dict.

  6. #5
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Maybe publish it as LZMA3 or something like that.
    using names like 7-zip pro, zstd2 and lzma3 is unfair to authors of original algos. I rather prefer anything showing strong relation to original algorithm, yet the difference. In particular, Radyx name tells a lot to me, but not to most of users. Given algorithm strong points, i can suggest something with names multi, par, block, f.e. ParLzma2. I.e. the name should let people know that this algo
    1) produces lzma2-compatible output
    2) employs all cores in more efficient manner than original algo

    Perhaps change ari to ANS so it won't remain compatible with LZMA2 and boost a little more the speed?
    lzma employs binary ari coders. In order to replace that with ANS-based coders one need to completely redesign compression format, which may require several months of work and skills available only in few specialists (may be, E.Shelwien?). Switching binary ari to binary ans can't buy us anything

    I gave the codec a unique id and added code to replace it with the LZMA2 id when writing, so the resulting .7z file can be decoded with mainline.
    do you mean it's already implemented in the provided executable or will be in future versions? my english is weak..

    After some more testing it should be okay to add it to milky's code.
    7z-zs is great project, but i insist that it will be great to publish your codec as separate dll, that can be dropped into other programs too. I think, the main reason behind 7z-zs is GUI support for choosing compression parameters of new codecs, but i think this problem should be solved other way - by establishing an API that allows codecs to expose their parameters to GUIs.

    Aniskin already developed first version of such API. If codecs included in 7z-zs and 7z-zs GUI will adopt such API, it will open the way to develop GUI-controlled codecs and codec-controlling GUIs independently instead of packaging both into a single monolithic product

  7. The Following 2 Users Say Thank You to Bulat Ziganshin For This Useful Post:

    Gonzalo (16th January 2018),hunman (16th January 2018)

  8. #6
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts
    do you mean it's already implemented in the provided executable or will be in future versions? my english is weak.
    It's not implemented that way in the provided exe. It was in my 7z-zstd dev branch which was never released, and will be in the new one.

    If I get a new codec id from Igor Pavlov and make a drop-in dll, I prefer to make an improved algorithm, e.g. Charles Bloom found that literal encoding is better if always done in match mode. Also some format changes to match it better with the block algorithm. I agree the algorithm should have a unique name, like RLZMA or similar.

  9. #7
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    I somewhat lost. So
    1) your current codec always generate data compatible with lzma2 format? Or it depends on some parameters?
    2) your current codec reuses the lzma2 codec id? i.e. programs using your codec cannot use original lzma2 anymore?
    3) what is the way, if any, 7-zip proposes for codecs that have original compression algo, but can be decompressed by existing (standard) codecs?

    One more possibility is to extend lzma2 with your match finder, so user can choose bt4/hc4 for original algos, and Radyx MF for a new algo. It's what i have done in my freearc which adds HT4 mf to lzma

    Sp please elaborate a bit of lzma2<->rlzma interaction, existing and planned, so we can understand how your codec can be used and how it can't

  10. #8
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    I somewhat lost. So
    1) your current codec always generate data compatible with lzma2 format? Or it depends on some parameters?
    Always compatible.
    2) your current codec reuses the lzma2 codec id? i.e. programs using your codec cannot use original lzma2 anymore?
    Correct unless the workaround is used, but that's not a plug-in dll. It needs a mod to the .7z database writing stage.
    3) what is the way, if any, 7-zip proposes for codecs that have original compression algo, but can be decompressed by existing (standard) codecs?
    It is not supported because 7-Zip identifies codecs by id after reading the id from the registered info. I had this problem when doing the original 7-Zip-zstd work. I registered the codec with the name Radyx, 7-Zip would read the id 0x21 from the registration info, then when setting up compression it would use the original LZMA2 encoder because it was registered first with 0x21.
    One more possibility is to extend lzma2 with your match finder, so user can choose bt4/hc4 for original algos, and Radyx MF for a new algo. It's what i have done in my freearc which adds HT4 mf to lzma
    The problem is the matchfinder completes its work before the encoder begins, and multiple encoders run in parallel on the result. Also, rep matches are checked after the match, because if the match table has no match, there's no rep. It works better with a separate encoder.

    It will be a while before I try to make a new format.

  11. #9
    Member
    Join Date
    Oct 2009
    Location
    usa
    Posts
    56
    Thanks
    1
    Thanked 9 Times in 6 Posts
    May we please get a 32-bit compile for x86 systems? It is worth trying. Thanks.

  12. #10
    Member
    Join Date
    Aug 2016
    Location
    Russia
    Posts
    79
    Thanks
    6
    Thanked 51 Times in 27 Posts
    Quote Originally Posted by Conor View Post
    I registered the codec with the name Radyx, 7-Zip would read the id 0x21 from the registration info, then when setting up compression it would use the original LZMA2 encoder because it was registered first with 0x21.
    Did you try to ask Igor about this problem?

  13. #11
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts
    Quote Originally Posted by zyzzle View Post
    May we please get a 32-bit compile for x86 systems? It is worth trying. Thanks.
    I've only just compiled an x86 version of the library so will need to do some testing. Maybe in a day or two.

    Quote Originally Posted by Aniskin View Post
    Did you try to ask Igor about this problem?
    No but it was pretty clear what was happening. It's unusual to have two ways of making the same type of output that are different enough to need separate implementations.

  14. #12
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts
    Attached an updated version to my original post due to an infinite loop bug. Not a library bug but one in the 7-zip encoder object.

  15. #13
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    Quote Originally Posted by Conor View Post
    Attached an updated version to my original post due to an infinite loop bug. Not a library bug but one in the 7-zip encoder object.
    Than you. I have also a different problem in 7zFM. I cannot navigate between folders. It gives me an error saying that the system cannot allocate the right amount of memory, then crashes. Maybe is WINE's fault but I don't have the same issue with vanilla 7z nor zstd-modified

  16. #14
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts
    I can't reproduce the problem in Windows 10. Maybe it's something to do with the build options. I'm using the 7z makefiles, which is what 7z-zs uses too but they may have modified them.

  17. #15
    Member
    Join Date
    Aug 2016
    Location
    Russia
    Posts
    79
    Thanks
    6
    Thanked 51 Times in 27 Posts
    Quote Originally Posted by Conor View Post
    No but it was pretty clear what was happening. It's unusual to have two ways of making the same type of output that are different enough to need separate implementations.
    I asked Igor and he said: Probably it's possible to change that code in 7-Zip. I'll look it later.

  18. #16
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts
    Thanks for asking. I found a way to add it as a new matchfinder option into the existing lzma2 compressor class, but it would be neater to make it separate.

    I just released a new version with the latest library v0.9.1.

  19. #17
    Member
    Join Date
    Sep 2011
    Location
    uk
    Posts
    237
    Thanks
    186
    Thanked 16 Times in 11 Posts
    Conor, thanks for this new code.

    Just few qs:
    0) Be good to integrate with 7z zstd -is this proposed?
    1) I've downloaded 64 bit The 7z.exe and 7z.dll clash with those of the 'real' 7zip. If I overwrite these can I still get standard 7z functionality as in 7zip 18.01?
    2) Once I've overwritten these can I still use 7z.exe from the standard 18.01 dist or do I need to copy the standard dll?
    3) to get max compression do I still use -mx

    john

  20. #18
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts
    John,

    You're welcome

    I added the old c++ code from Radyx to 7z-zstd but C++11 build problems meant it was never merged. When the new C library has had some more testing it will be ready to add.

    The binaries should go in a separate folder from standard 7z 18.01 because the build didn't include all of the compression codecs and container formats in the full 7-Zip. For the next build I'll look at enabling all of those, and see if I can use the installer too.

    The -mx switch should work as before.

  21. The Following User Says Thank You to Conor For This Useful Post:

    avitar (7th February 2018)

  22. #19
    Member
    Join Date
    Sep 2011
    Location
    uk
    Posts
    237
    Thanks
    186
    Thanked 16 Times in 11 Posts
    Put it in separate folder and tried it compressing a directory from MS Office 2007 799Mbytes. With -mx2 compresses to 679M or so, similar speed to real 7-zip and to arc. rz is a lot slower.

    What tells me I'm using the new stuff? Should it just be same compression, but bit eg 5% faster?
    john

  23. #20
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts
    7-Zip adds its install folder to the search path, so if the modified 7z.exe fails to find the modified 7z.dll, it will find and use the standard one instead. Also if standard 7z.dll is already in memory it will use it. I will look at using a manifest or something to prevent this.

    The simplest way to tell is use -mmf=RMF which will error out in the standard dll. Also the -bt option should show under 100Mb RAM usage at level 5.

    The lower levels are closer to 7z performance. Level 1 is actually slightly worse in speed vs ratio. The algorithm starts to work really well at level 5 (similar to level 8 in the library dll).

  24. #21
    Member
    Join Date
    Aug 2016
    Location
    Russia
    Posts
    79
    Thanks
    6
    Thanked 51 Times in 27 Posts
    Quote Originally Posted by Conor View Post
    I found a way to add it as a new matchfinder option into the existing lzma2 compressor class, but it would be neater to make it separate.
    It looks like Igor fixed the problem in 18.02. I created dummy codec with ID=0x21 and now 7zip calls my code during compression.

  25. #22
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts
    Quote Originally Posted by Aniskin View Post
    It looks like Igor fixed the problem in 18.02. I created dummy codec with ID=0x21 and now 7zip calls my code during compression.
    Does LZMA2 still work? 7z used to use the first codec that was registered with 0x21, so it's possible that it always uses your dummy codec if it is registered first.

  26. #23
    Member
    Join Date
    Aug 2016
    Location
    Russia
    Posts
    79
    Thanks
    6
    Thanked 51 Times in 27 Posts
    Quote Originally Posted by Conor View Post
    Does LZMA2 still work?
    Yes. When I select original LZMA2 codec - 7z uses original LZMA2 codec, When I select my dummy 0x21 codec - 7z uses my dummy codec.

  27. #24
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts
    That's great. Thanks for letting me know.

  28. #25
    Member
    Join Date
    Aug 2016
    Location
    Russia
    Posts
    79
    Thanks
    6
    Thanked 51 Times in 27 Posts
    Is there a chance that you will create a separate codec for 7Zip?

  29. #26
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts
    It's the neatest way to add it to 7-zip, so I'll update my 7z fork to 18.02 and modify it.

  30. #27
    Member
    Join Date
    Sep 2011
    Location
    uk
    Posts
    237
    Thanks
    186
    Thanked 16 Times in 11 Posts
    18.03 out

  31. #28
    Member
    Join Date
    Feb 2015
    Location
    Australia
    Posts
    73
    Thanks
    13
    Thanked 65 Times in 23 Posts
    New release, updated to 18.03 and the radix method is now specified as FLZMA2. It's the default method in the command line exe. Compilation includes Igor's optimized decoder.

    https://github.com/conor42/7-Zip-FL2/releases/

  32. The Following 7 Users Say Thank You to Conor For This Useful Post:

    Aniskin (9th March 2018),avitar (9th March 2018),Bulat Ziganshin (9th March 2018),joerg (9th March 2018),load (9th March 2018),Zonder (9th March 2018),zubzer0 (9th March 2018)

  33. #29
    Member
    Join Date
    Jul 2016
    Location
    Russia
    Posts
    21
    Thanks
    13
    Thanked 7 Times in 6 Posts
    debug version - need specific dll's

  34. The Following User Says Thank You to zubzer0 For This Useful Post:

    load (9th March 2018)

  35. #30
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    wonderful - thank you very much!

Page 1 of 2 12 LastLast

Similar Threads

  1. Fast LZMA2 library with radix matchfinder
    By Conor in forum Data Compression
    Replies: 27
    Last Post: 28th May 2019, 03:28
  2. Give me newest Win64 lz4.exe & lz5.exe binaries
    By lz77 in forum Data Compression
    Replies: 3
    Last Post: 21st July 2017, 10:47
  3. SLZ - stateless zip - fast zlib-compatible compressor
    By willy in forum Data Compression
    Replies: 48
    Last Post: 28th August 2016, 21:39
  4. lzma2 stream detector
    By Shelwien in forum Data Compression
    Replies: 0
    Last Post: 4th June 2016, 19:28
  5. Compiling Fastest Binaries
    By comp1 in forum The Off-Topic Lounge
    Replies: 8
    Last Post: 3rd April 2016, 18:54

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •