Results 1 to 5 of 5

Thread: TurboPFor (Integer Compression) vs blosc (Binary compression)

  1. #1
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    456
    Thanks
    46
    Thanked 164 Times in 118 Posts

    TurboPFor (Integer Compression) vs blosc (Binary compression)

    First of all blosc is claiming to be faster than memcpy.
    This is impossible, simply because blosc is no more
    than a shuffle (or transpose) + lz77.
    Only the shuffle alone cannot be faster than memcpy, because
    shuffle is a blocked memcpy with zero compression.
    The Shuffle is applied before compression and after decompression.


    A multithread blosc muss be also compared to a multithreaded memcpy.


    We can see from the benchmarks at TurboPFor that "Integer Compression" is superior to "binary compression"
    like blosc in terms of compression ratio and speed.
    Blosc is tested w. lz4 as compressor and vectorized shuffle.


    Another advantage of Integer Compression is direct access to individual values without decompressing entire blocks.

  2. #2
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    First of all blosc is claiming to be faster than memcpy.
    This is impossible, simply because blosc is no more
    than a shuffle (or transpose) + lz77.
    Only the shuffle alone cannot be faster than memcpy, because
    shuffle is a blocked memcpy with zero compression.
    The Shuffle is applied before compression and after decompression.
    Why? If stream is divided into small enough blocks then all intermediate data can fit in caches so we can get faster than memcpy. Memcpy has to read uncompressed input and write uncompressed output, that's 2x input size. Codec that keeps all intermediate data within caches does less memory I/O than 2x input size as the compressed stream is smaller than uncompressed one.

    A multithread blosc muss be also compared to a multithreaded memcpy.
    Yep.

    We can see from the benchmarks at TurboPFor that "Integer Compression" is superior to "binary compression"
    like blosc in terms of compression ratio and speed.
    Blosc is tested w. lz4 as compressor and vectorized shuffle.
    It's possible that blosc authors can provide other benchmarks in which they win. We need independent benchmarkers to make objective opinion.

    Another advantage of Integer Compression is direct access to individual values without decompressing entire blocks.
    That's nice, although it also depends on use case - what if the user actually reads input data in small blocks?

  3. #3
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    456
    Thanks
    46
    Thanked 164 Times in 118 Posts
    Quote Originally Posted by Piotr Tarsa View Post
    Why? If stream is divided into small enough blocks the...
    Well, as i have explained the shuffle function alone is a variant of memcpy. Copy the same amount of bytes only to different locations. Addionally you need lz77 after preprocessing.
    You have also Blosc = shuffle + lz77.
    This is independent from the data or caching. "shuffle + lz77" cannot be faster than memcpy, because "memcpy speed" >= "shuffle speed" as the same amount of bytes is transferred.
    Last edited by dnd; 21st June 2015 at 22:38.

  4. #4
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    But the intermediate copy is done within caches. Authors didn't claim that blosc is faster than memcpy for all input sizes and input caching status. But even with small input they could achieve speed over memcpy if the input is uncached prior to compression/ decompression.

  5. #5
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    Okay. If I would say that blosc decompression is faster than system-RAM to system-RAM transfer of uncompressed data, then would that be possible? And why (or why not)?

Similar Threads

  1. TurboPFor: Integer Compression
    By dnd in forum Data Compression
    Replies: 40
    Last Post: 15th July 2019, 20:46
  2. Simple binary rangecoder demo
    By Shelwien in forum Data Compression
    Replies: 35
    Last Post: 17th June 2019, 16:21
  3. Crook, a new binary PPM compressor
    By valdmann in forum Data Compression
    Replies: 25
    Last Post: 19th March 2012, 17:12
  4. lzma's binary tree matchfinder
    By Shelwien in forum Data Compression
    Replies: 7
    Last Post: 13th October 2011, 08:38
  5. Bytewise vs Binary
    By Shelwien in forum Forum Archive
    Replies: 9
    Last Post: 30th March 2008, 16:51

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •