Results 1 to 7 of 7

Thread: demo of lzma file alignment effect

  1. #1
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts

    demo of lzma file alignment effect

    http://nishi.dreamhosters.com/u/lzma_align_v0.rar

    Code:
    file1   262151
    file1a       9
    file2   786432
    
    1.7z    827248 // 7z a 1 file1 file2
    2.7z    809983 // 7z a 2 file1 file1a file2
    It might be even better if lc2 (lzma parameter) is used.

    In short, lzma is weird like that, so all files in solid archives have to be padded to filesize%32==0

  2. The Following User Says Thank You to Shelwien For This Useful Post:

    RamiroCruzo (6th February 2017)

  3. #2
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    have to be
    but how they can be?

  4. #3
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > but how they can be?

    By padding them like I demonstrated?
    Or just by taking it into account in a new format implementation?
    (The padding doesn't have to be actually stored anywhere, its just needed to fix up the data position in the codec)

  5. #4
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    without a new format, we will end up with unwanted files on extraction, i think.

  6. #5
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    yes, but for installers and such it doesn't matter.
    Also we can use "nul" for names of pad files

  7. #6
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Could you explain what's the reason for the effect?
    Does it impact lzmarec too?
    How came Igor didn't do it by himself?

  8. #7
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > Could you explain what's the reason for the effect?

    lzma doesn't treat the match distances as a single "alphabet".
    It only has a model for log2(dist) + high bit and low 4 bits,
    while all intermediate bits are stored without compression.

    Thus, any changes in the data that affect the distance alignment
    would also affect compression.

    Of course, my example here is intentionally designed to amplify
    the difference (its still not the worst possible case though),
    but the same effect is also noticeable with "natural" files.

    > Does it impact lzmarec too?

    Yes, lzmarec uses a very similar distance decomposition:
    Code:
    1.7z.rec 755038
    2.7z.rec 737363
    > How came Igor didn't do it by himself?

    lzma is basically the only codec (besides its clones) which has such a quirk,
    so maybe he simply didn't think about this.

    But its also a fact that there're many filetypes for which
    lzma compression would be affected by alignment of their
    start position (eg. exes), and that all existing archive formats
    with lzma support and solid compression don't fix up
    the stream position on transition between files.

    Anyway, there're many possible improvements for lzma which are not
    currently implemented (eg. http://encode.ru/threads/1546-init-f...terals-in-lzma )
    so its nothing special.

  9. The Following 2 Users Say Thank You to Shelwien For This Useful Post:

    Bulat Ziganshin (13th April 2018),RamiroCruzo (6th February 2017)

Similar Threads

  1. Simple binary rangecoder demo
    By Shelwien in forum Data Compression
    Replies: 35
    Last Post: 17th June 2019, 16:21
  2. comparing file transfer speed with ccm and lzma compression
    By Shelwien in forum Data Compression
    Replies: 19
    Last Post: 10th March 2011, 12:49
  3. M1 - Optimized demo coder
    By toffer in forum Data Compression
    Replies: 189
    Last Post: 21st July 2010, 23:49
  4. Comparison of the recent CM demo coders
    By Shelwien in forum Data Compression
    Replies: 38
    Last Post: 13th June 2008, 13:21
  5. QUAD-SFX DEMO
    By encode in forum Forum Archive
    Replies: 17
    Last Post: 26th April 2007, 13:57

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •