Results 1 to 3 of 3

Thread: fastest open source integer compression algorithms

  1. #1
    Member
    Join Date
    Jan 2015
    Location
    chennai,Tamilnadu,India
    Posts
    19
    Thanks
    2
    Thanked 0 Times in 0 Posts

    fastest open source integer compression algorithms

    I am in need of fastest integer (de)compression implementation. Suggest any of that kind with liberal licensing schemes like Apache or BSD. I am looking forward to adopt that into my codes.

  2. #2
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts

  3. #3
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts

    Fast integer compression with brotli

    Quote Originally Posted by mitra View Post
    I am in need of fastest integer (de)compression implementation. Suggest any of that kind with liberal licensing schemes like Apache or BSD. I am looking forward to adopt that into my codes.
    If you have relatively dense filling of the range of integers and no correlations, like in storing a bloom filter, just use a binary arithmetic coder for them. If you have something more complex, then you can benefit from a more advanced compressor.

    Given that you have monotonically rising unique integers, you should consider codifying their deltas, and possibly store values of delta - X in your stream, where X is the smallest diff between two consequent integers. It is possibly a good idea to transpose the four bytes (or two 16-bit values, eight nibbles, ...) of the deltas, too, before compressing. Transposing the bytes means that you store all most significant bytes first for all values, then next most significant, etc. Transposing is more likely to be helpful when used with a fast decompressor such as deflate or brotli and less helpful or even possibly harmful with more advanced (but slower) statistical modeling.

    If you try out brotli, you can get roughly the same decoding speed as you see with deflate, but with a greatly improved compression ratio. When configuring brotli to store such data you may get slightly denser compression ratio using the configuration WOFF 2.0 uses (MODE_FONT in BrotliParams).

Similar Threads

  1. TurboPFor: Integer Compression
    By dnd in forum Data Compression
    Replies: 40
    Last Post: 15th July 2019, 20:46
  2. Silesia Open Source Compression Benchmark
    By Alexander Rhatushnyak in forum Data Compression
    Replies: 15
    Last Post: 22nd May 2016, 15:34
  3. Open Source Streaming API for Compression
    By harry in forum Data Compression
    Replies: 7
    Last Post: 30th September 2014, 07:28
  4. Why not open source?
    By nemequ in forum Data Compression
    Replies: 65
    Last Post: 25th November 2013, 23:05
  5. Non open source Data compression Tools
    By ehsansad in forum Data Compression
    Replies: 9
    Last Post: 22nd September 2011, 00:41

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •