Results 1 to 7 of 7

Thread: IPP7 beta

  1. #1
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts

    IPP7 beta

    These days, even Intel tries to show us how to write compressors
    IPP7 now includes implementations for zlib,gzip,bzip2 and lzo,
    there's no prebuilt binary for lzo though, so I skipped it.

    The main point here is that bzip2 was there since long ago, and
    it was based on some standard sort, which tended to freeze on
    redundant files.

    This time they optimized it again and it doesn't take 20s to
    sort 1.5M file anymore, but I'd appreciate if BWT experts here
    could verify it with some more problematic data.

    ipp7 compression utils rip:
    http://nishi.dreamhosters.com/u/ipp70beta_dc.rar
    ipp 6.1 bzip2 rip:
    http://nishi.dreamhosters.com/u/ipp_...61u4_win32.rar

    IPP7 beta links:
    http://software.intel.com/en-us/arti...gram-homepage/
    http://registrationcenter-download.i...0.030_ia32.exe
    http://registrationcenter-download.i..._7.0.0.020.zip

    Links for "originals" utils:
    gzip: http://downloads.sourceforge.net/pro...3.12-1-bin.zip
    bzip2: http://downloads.sourceforge.net/pro...-1.0.5-bin.zip
    zlib: http://www.winimage.com/zLibDll/zlib125.zip + self-compiled

    Code:
    Q9450 @ 3.52Ghz, ramdrive
    
    test.html:
             orig   ipp7    orig   ipp7
    bzip2    0.328s 0.796s  342516 342792  
    gzip     0.188s 0.094s  593932 599175  
    minigzip 0.187s 0.219s  597025 596673  
    
    enwik8:
             orig   ipp7    orig     ipp7
    bzip2    13.453s 8.750s 29041862 29008758
    gzip      9.782s 2.484s 36457542 36445251
    minigzip  9.453s 9.969s 36524739 36548933
    
    P4 @ 2.8Ghz. hdd
    
    test.html
             orig   ipp7    orig   ipp7
    bzip2    1.282s 5.468s  342792 342516
    gzip     0.313s 0.391s  594293 593932
    minigzip 0.329s 0.375s  596673 597025

  2. #2
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Appears that my rips won't work without MS CRT 9.0 installed.
    In such a case it can be fixed by copying this (contents of)
    into \WINDOWS directory: http://nishi.dreamhosters.com/u/msvcr90redist.rar

    Also, Sami posted a comparison: http://compressionratings.com/s_bzip2.html

  3. #3
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    I noticed that somehow IPP versions don't seem to utilize all the
    cores, tested and observed the same behaviour myself.
    Then tried to compile it, and... apparently they have some
    macros there and forgot to define them for their demo executable %)

    For now I removed dlls, enabled MT, and compiled separate versions
    for Itoh-Tanaka, "suffix sort" and "auto" switches:
    http://nishi.dreamhosters.com/u/ipp70b_bzip2_sh.rar

    Note: these are unpatched intelc SSE2 versions...
    Would likely require some kind of icc_patch for usage on amd cpus.

  4. #4
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    @Shelwien:

    ask 1:

    as you surely know the current bzip2-version is 1.0.5 (http://www.bzip.org/)

    why intel and you are using the old version 1.0.4 and not the current one ?


    ask 2:

    as you surely know there is a parallel version pbzip2 version 1.1.1 (Apr 17, 2010)

    on http://compression.ca/pbzip2/
    and a first win32-compile on http://www.leszer.net/index.php/my-software.html

    Would it be possible to make such a optimized ipp7-version from the new pbzip2 version 1.1.1 ?


    best regards

  5. #5
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > as you surely know the current bzip2-version is 1.0.5 (http://www.bzip.org/)
    > why intel and you are using the old version 1.0.4 and not the current one ?

    I'm using that because its what intel included into their ipp releases.
    And as to why - they probably don't care. Their previous bzip modifications
    had some major performance problems, and this version is still beta.
    Btw the only difference between 1.0.4 and 1.0.5 is this:

    Code:
    diff -rw bzip2-1.0.4\bzlib_private.h bzip2-1.0.5\bzlib_private.h
    11,12c11,12
    <    bzip2/libbzip2 version 1.0.4 of 20 December 2006
    <    Copyright (C) 1996-2006 Julian Seward <jseward@bzip.org>
    ---
    >    bzip2/libbzip2 version 1.0.5 of 10 December 2007
    >    Copyright (C) 1996-2007 Julian Seward <jseward@bzip.org>
    39c39
    < #define BZ_VERSION  "1.0.4, 20-Dec-2006"
    ---
    > #define BZ_VERSION  "1.0.5, 10-Dec-2007"
    444a445,446
    >     /* c_tPos is unsigned, hence test < 0 is pointless. */ \
    >     if (s->tPos >= (UInt32)100000 * (UInt32)s->blockSize100k) return True; \
    449a452,453
    >     /* c_tPos is unsigned, hence test < 0 is pointless. */ \
    >     if (c_tPos >= (UInt32)100000 * (UInt32)ro_blockSize100k) return True; \
    471a476,477
    >     /* c_tPos is unsigned, hence test < 0 is pointless. */ \
    >     if (s->tPos >= (UInt32)100000 * (UInt32)s->blockSize100k) return True; \
    So it shouldn't be hard to fix manually in ipp source - not that I see
    any sense to do that myself, as it doesn't affect performance.

    > as you surely know there is a parallel version pbzip2 version 1.1.1 (Apr 17, 2010)

    Yes, I've seen your post and notified Sami, let's hope that he'll eventually
    find time to do more tests.

    > Would it be possible to make such a optimized ipp7-version
    > from the new pbzip2 version 1.1.1 ?

    I didn't really look, but ipp is not some kind of a trick applicable to anything.
    Its just a collection of implementations of popular algorithms by intel developers -
    not even quite open-source, as the actual library is distributed in the form
    of .lib + .dll files - btw you may find ipp*.dll binaries in some of your installed
    apps probably.
    Anyway, there's intel's own multi-threaded bzip2 codec implementation in ipp, and
    I don't see how's that compatible with pbzip2.
    But I guess it might be possible to add some speed to pbzip2 by experimenting with
    compiler settings - both intelc and gcc 4.5, but that's time-consuming and
    I'm not really interested.

  6. #6
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    on www.bzip.org is a new version 1.0.6 (sourcecode) online
    ---
    The current version is 1.0.6, released 20 Sept 2010.
    Version 1.0.6 removes a potential security vulnerability, CVE-2010-0405, so all users are recommended to upgrade immediately.
    ---
    anyone has binaries ?
    anyone has seen updates of derivative works ?

  7. #7
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Quote Originally Posted by joerg View Post
    on www.bzip.org is a new version 1.0.6 (sourcecode) online
    anyone has binaries ?
    anyone has seen updates of derivative works ?
    http://www.heise.de/newsticker/meldu...n-1082667.html (german)
    http://www.h-online.com/open/news/it...l-1082837.html (english)

    The article only mentions updated ClamAV software and Linux distributions, but I guess there are/will be further updates.

    As for binaries, I did a quick Windows compilation using MinGW gcc 4.5.0. Makefiles didn't work, so I used the following calls:

    Code:
    gcc -Wall -Winline -O2 -g -D_FILE_OFFSET_BITS=64 blocksort.c huffman.c crctable.c randtable.c compress.c decompress.c bzlib.c bzip2.c -obzip2.exe
    gcc -Wall -Winline -O2 -g -D_FILE_OFFSET_BITS=64 blocksort.c huffman.c crctable.c randtable.c compress.c decompress.c bzlib.c bzip2recover.c -obzip2recover.exe
    Attached Files Attached Files
    http://schnaader.info
    Damn kids. They're all alike.

  8. The Following User Says Thank You to schnaader For This Useful Post:

    Jaff (10th May 2014)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •