Results 1 to 11 of 11

Thread: Linux Kernel community phases out Bzip2 for source bundles

  1. #1
    Member
    Join Date
    Nov 2012
    Location
    Bangalore
    Posts
    114
    Thanks
    9
    Thanked 37 Times in 22 Posts

    Linux Kernel community phases out Bzip2 for source bundles

    A slightly old news but shows a shift towards LZMA within the open source community:
    https://www.kernel.org/happy-new-year-and-good-bye-bzip2.html

    The unfortunate thing is that XZ is very bad on performance. It is significantly slower than 7-Zip. I feel, performance aspect in XZ was forgotten in the name of file format and API standardization and code restructuring.

  2. #2
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    why it's slower than 7-zip if both use the same lzma algo?

  3. #3
    Member
    Join Date
    Nov 2012
    Location
    Bangalore
    Posts
    114
    Thanks
    9
    Thanked 37 Times in 22 Posts
    The code is completely different from original 7-Zip implementation. I have found it to be quite slow in my tests: http://moinakg.github.io/pcompress/results1.html
    I do not know the exact cause. Maybe it uses different parameters.

  4. #4
    Member
    Join Date
    Oct 2010
    Location
    New York
    Posts
    21
    Thanks
    9
    Thanked 2 Times in 1 Post
    I'm not really seeing a difference between xz and 7-zip in terms of decompression speed on my machines. Especially when compared to bzip2.

    One of the big hurdles for 7zip in the *nix community is that the current commandline interface doesn't adhere to the POSIX standards. This makes it more difficult to insert into the full toolchains that are already in place.

    Code:
    ~$ time unxz linux-3.13-rc7.tar.xz 
    
    real    0m7.294s
    user    0m6.716s
    sys    0m0.576s
    
    ~$ time 7z x linux-3.13-rc7.tar.7z 
    
    7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
    p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,8 CPUs)
    
    Processing archive: linux-3.13-rc7.tar.7z
    
    Extracting  linux-3.13-rc7.tar
    
    Everything is Ok
    
    Size:       549806080
    Compressed: 80265748
    
    real    0m7.143s
    user    0m6.564s
    sys    0m0.576s
    
    ~$ time bunzip2 linux-3.13-rc7.tar.bz2 
    
    real    0m15.859s
    user    0m15.281s
    sys    0m0.580s

  5. #5
    Member
    Join Date
    Oct 2010
    Location
    New York
    Posts
    21
    Thanks
    9
    Thanked 2 Times in 1 Post
    Testing compression speeds using the flags from your benchmark seems about the same on my machine. Maybe your version was compiled differently? Or maybe a strong machine dependency?

    Code:
    ~$ time 7za a -t7z -mx6 linux-3.13-rc7.tar.7z linux-3.13-rc7.tar
    
    7-Zip (A) [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
    p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,8 CPUs)
    Scanning
    
    Creating archive linux-3.13-rc7.tar.7z
    
    Compressing  linux-3.13-rc7.tar      
    
    Everything is Ok
    
    real    3m9.577s
    user    5m21.212s
    sys    0m1.864s
    
    ~$ time xz -z -6 -k linux-3.13-rc7.tar -v
    linux-3.13-rc7.tar (1/1)
      100 %        76.0 MiB / 524.3 MiB = 0.145   1.7 MiB/s       5:13             
    
    real    5m13.160s
    user    5m12.584s
    sys    0m0.572s

  6. #6
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by moinakg View Post
    The code is completely different from original 7-Zip implementation. I have found it to be quite slow in my tests: http://moinakg.github.io/pcompress/results1.html
    I do not know the exact cause. Maybe it uses different parameters.
    it may be because it doesn't implement lzma multithreading. check CPU times. anyway, compression time isn't important for kernel.org purposes - they have classic compress-once-extract-many usage

  7. #7
    Member
    Join Date
    Jun 2008
    Location
    G
    Posts
    372
    Thanks
    26
    Thanked 22 Times in 15 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    it may be because it doesn't implement lzma multithreading. check CPU times. anyway, compression time isn't important for kernel.org purposes - they have classic compress-once-extract-many usage
    yes i agree compressiontiom is nearly doubled in your tests, but i though that was already implemented........hmmhm

  8. #8
    Member
    Join Date
    Jun 2013
    Location
    USA
    Posts
    98
    Thanks
    4
    Thanked 14 Times in 12 Posts
    Multi-threading is implemented in the latest xz alpha(5.1.3) but not in the stable.

  9. #9
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    I thought I would test the two concatenated copies of silesia.tar on zpaq, but I get 423,896,064 bytes for the original size, not 413,224,960. Anyway, on a 2 GHz T3200 (2 cores, Win32), which is slower than their test machine:

    136,470,367 silesia2.tar.gz 37.6 s
    109,140,158 silesia2.tar.bz2 144.3 s
    99,458,006 silesia2.7z 262.6 s
    69,603,834 silesia-1.zpaq 21.3 s
    67,175,106 silesia-2.zpaq 38.7 s
    59,226,485 silesia-3.zpaq 62.5 s
    48,343,287 silesia-4.zpaq 163.9 s
    43,598,975 silesia-5.zpaq 337.8 s
    40,138,239 silesia-6.zpaq 1161.0 s
    Last edited by Matt Mahoney; 10th January 2014 at 01:26. Reason: added silesia-6.zpaq

  10. #10
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    190
    Thanks
    8
    Thanked 62 Times in 33 Posts
    Quote Originally Posted by moinakg View Post
    The code is completely different from original 7-Zip implementation.
    Is it related to LZMA2?

  11. #11
    Member
    Join Date
    Jun 2008
    Location
    G
    Posts
    372
    Thanks
    26
    Thanked 22 Times in 15 Posts
    no xz tools make just did programmed from scratch according spec

Similar Threads

  1. BZIP2 1.0.6 for Windows
    By encode in forum Download Area
    Replies: 13
    Last Post: 18th October 2018, 02:01
  2. Concurrent kernel execution in OpenCL implementations
    By Piotr Tarsa in forum The Off-Topic Lounge
    Replies: 0
    Last Post: 16th April 2011, 16:04
  3. bzip2 dictionary size
    By Wladmir in forum Data Compression
    Replies: 3
    Last Post: 7th April 2010, 16:09
  4. Data decompression on in-memory kernel
    By cregd in forum Data Compression
    Replies: 8
    Last Post: 27th January 2009, 18:24
  5. SouceForge's community choiche awards
    By giorgiotani in forum Forum Archive
    Replies: 2
    Last Post: 20th June 2007, 01:12

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •