Results 1 to 7 of 7

Thread: Intel IPP compression support

  1. #1
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts

    Lightbulb Intel IPP compression support

    Intel Integrated Performance Primitives (IPP) includes a nice set of compression-related stuff - Deflate, BZip2, LZO, LZSS, BWTFwd/BWTInv, ... Nice read:
    http://software.intel.com/sites/prod...3_0_Intro.html

  2. The Following User Says Thank You to encode For This Useful Post:

    nburns (3rd February 2014)

  3. #2
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    I wonder if Intel plans to extend the instruction set or add hardware optimizations to support their library.

  4. #3
    Member
    Join Date
    Jun 2008
    Location
    G
    Posts
    372
    Thanks
    26
    Thanked 22 Times in 15 Posts
    Quote Originally Posted by Matt Mahoney View Post
    I wonder if Intel plans to extend the instruction set or add hardware optimizations to support their library.
    i think that is the logical consequense, its just a matter of time, but maybe we will see in the next tock, what will happen

  5. #4
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    There are hardware Deflate solutions already, so Intel is late to the party :P

    http://en.wikipedia.org/wiki/DEFLATE#Hardware_encoders

    And bzip2 is already being obsoleted by lzma/ lzma2.

  6. #5
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    I think I recall a conversation a year or two ago about Intel asking how they could optimize compression. My response was that for high end algorithms like PPM or CM that the bottleneck is random memory access, so the best thing they could do was design faster memories and larger caches.

    But it seems to me that the most logical target would be to optimize video decompression. Maybe instructions for fast IDCT.

    Another possibility is hardware and OS support for compressed memory. For example, blocks of zero bytes don't have to be physically allocated.

  7. #6
    Member just a worm's Avatar
    Join Date
    Aug 2013
    Location
    planet "earth"
    Posts
    96
    Thanks
    29
    Thanked 6 Times in 5 Posts
    If I understood thouse "Integrated Performance Primitives" correctly then this is a bunch of libraries for Linux and Windows. I think the libraries for Windows where *.dll files which only had a few functions (they split all the stuff up so there are many *.dll files) and every *.dll file is something like 2 MB in size. The whole package for Windows is something like 400 MB. So I don't think that this library package is beeing used widely.

    Did anyone ever work with thouse libraries? How are they doing regarding speed and memory consumption?

    My response was that for high end algorithms like PPM or CM that the bottleneck is random memory access, so the best thing they could do was design faster memories and larger caches.
    Yes, that's right. If you are trying to do a BWT detransform, then you have the same problem.

    But I don't think that larger caches really fixes the problem. It helps but it rather reduces the problem to the data that still doesn't fit in the cache because of it's still to big. Usually the data you need to process is also several times bigger than the actual payload.

    One thing that intel has done in the near past was adding support to non-blocking memory accesses to the Atom-series. The Atom-series is probably the most sold processor serie at the time. Maybe this also helps a little to increase the speed. But you would probably need to do some adjustments in the code to make good use of the "out-of-order"-capabilities of a processor because it's somewhat different to an "in-order"-processor (like the older generation of the Atom-processors are).

    The most other processors already had support for non-blocking memory accesses. I don't think that AMD ever released another "in-order execution"-processor after their first "out-of-order execution"-processor.

    In the last months I tried to do some speed improvements by using the instruction set extensions. I used the SSE2 instruction set extension with the "16 byte"-XMM-registers to improve the detransformation speed of a BWT. I could only use it for some trivial tasks instead of anything serious so the speed improvement was less than 10% but it was very much work to look into all this stuff. So my personal opinion about thouse instruction set extensions is, that they are somewhat disappointing.
    Last edited by just a worm; 4th February 2014 at 14:01.

  8. #7
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    437
    Thanks
    137
    Thanked 152 Times in 100 Posts
    I tested ipp_gzip recently and while faster, it was also poorer at compression. There was still some speed up over and above the slightly lower compression (tested against zlib using lower compression levels to get a similar size), but not a particularly large one.

  9. The Following User Says Thank You to JamesB For This Useful Post:

    just a worm (11th February 2014)

Similar Threads

  1. Intel AVX-512, MPX and SHA Extension
    By Bulat Ziganshin in forum Download Area
    Replies: 7
    Last Post: 1st June 2015, 21:43
  2. Intel Avoton
    By Sportman in forum Data Compression
    Replies: 0
    Last Post: 4th September 2013, 01:57
  3. gzip - Intel IPP
    By M4ST3R in forum Download Area
    Replies: 5
    Last Post: 2nd June 2010, 15:09
  4. Winzip v12.0 with JPG recompression & 7z support
    By maadjordan in forum Data Compression
    Replies: 3
    Last Post: 12th September 2008, 23:58
  5. GCC mmx support
    By toffer in forum Forum Archive
    Replies: 6
    Last Post: 19th January 2008, 14:20

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •