Results 1 to 11 of 11

Thread: fastest and user friendly entropy encoder

  1. #1
    Member
    Join Date
    Apr 2012
    Location
    London
    Posts
    232
    Thanks
    9
    Thanked 0 Times in 0 Posts

    fastest and user friendly entropy encoder

    lzturbo (Hamid Buzidi, LZ+tANS)- focused on speed in large range of compression levels: https://sites.google.com/site/powturbo/


    This among the fastest ? Is this also user friendly allows user input multiplicities values of distinct input symbols to compress ? barest minimum possible Header in compressed file ( just minimally the input multiplicity values of distinct symbols in input file to be compressed ) ? Integrable within C# ( call C++ / C etc within C# )

    Likewise
    allows user input multiplicities values of distinct input symbols to decompress ?

    Otherwise any other recommendations ?


  2. #2
    Member
    Join Date
    Apr 2012
    Location
    London
    Posts
    232
    Thanks
    9
    Thanked 0 Times in 0 Posts
    Forget the LZ part, was only primarily interested in pure entropy coder part only

    Nevertheless obvious alreadythe tANS within LZTurbo
    seems among fastest

  3. #3
    Member
    Join Date
    Apr 2012
    Location
    London
    Posts
    232
    Thanks
    9
    Thanked 0 Times in 0 Posts
    Only then ANS could start to become widely used by everyone, not limited to ZStandard and few experts

  4. #4
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    612
    Thanks
    192
    Thanked 192 Times in 115 Posts
    Last time I checked, tANS was about ~500MB/s decoding on 1 CPU core, while rANS has reached crazy ~1500MB/s in 4 interleaving x 8 SIMD = 32 independent streams setting: https://sites.google.com/site/powturbo/entropy-coder
    On ARM they are much closer, tANS has some advantages for ASIC/FPGA, especially if working with static distributions. It can also include simultaneous encryption, for example offering cheap compression+encryption layer for IoT - which currently usually have none.

  5. #5
    Member
    Join Date
    Apr 2012
    Location
    London
    Posts
    232
    Thanks
    9
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Jarek View Post
    On ARM they are much closer, tANS has some advantages for ASIC/FPGA, especially if working with static distributions. It can also include simultaneous encryption, for example offering cheap compression+encryption layer for IoT - which currently usually have none.

    Is the simultaneous encryption strong like professionals dedicated encryption at all ?

    it may even seems like performing basic fisher-yates jumble on data couple of times with different secret keys is already closest to 'unbreakable' ?

  6. #6
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    612
    Thanks
    192
    Thanked 192 Times in 115 Posts
    No, its single round within compression is not the security level of 10 rounds of AES - which is just too costly from hardware and energy perspective for many applications, like IoT sensor which should cost cents, and gather and send data for 10 years from a single AAA battery - requiring what is called "lightweight cryptography".

    However, tANS provides very different type of data scrambling: using short blocks of practically unpredictable varying lengths - adding synchronization problem to cryptanalysis. They are extremely tough to handle, e.g. we still don't know even the capacity of the simplest synchronization channel: https://en.wikipedia.org/wiki/Deletion_channel
    Also, while AES uses only permutations and XORs, tANS behavior is extremely nonlinear, depends on chosen symbol spread (based on the key) defining evolution of hidden state.

    Anyway, tANS and AES have complementary types of data scrambling - wanting to get level of protection of 10 round AES, it should be sufficient to use e.g. tANS + 1-2 rounds of AES ... but it just needs serious research.

  7. #7
    Member
    Join Date
    Apr 2015
    Location
    Greece
    Posts
    68
    Thanks
    31
    Thanked 22 Times in 15 Posts
    Also check FPC which is faster than FSE and FSE Huff https://github.com/algorithm314/FPC

  8. #8
    Member
    Join Date
    Apr 2012
    Location
    London
    Posts
    232
    Thanks
    9
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by algorithm View Post
    Also check FPC which is faster than FSE and FSE Huff https://github.com/algorithm314/FPC
    This is surprising fast

    Is it simple feasible to modify compressor to output only compressed body ( no header at all), instead user supply these removed parameters to decode the 'pure' compressed body?

  9. #9
    Member
    Join Date
    Apr 2012
    Location
    London
    Posts
    232
    Thanks
    9
    Thanked 0 Times in 0 Posts
    How does FPC handle widely divergent static frequencies of eg 20 symbols ( non powers of 2 probabilities) and gives optimum entropy encode?

  10. #10
    Member
    Join Date
    Apr 2015
    Location
    Greece
    Posts
    68
    Thanks
    31
    Thanked 22 Times in 15 Posts
    It is a length limited Huffman coder. It does not have optimal entropy code. But it has a parser that chooses best block size(FPC 0). If probabilities are.not very skewed it has very good compression ratio(usually better than block based ANS due to.smaller.block size). Also it is very simple to use .

    Include fpc.h use FPC_compress/decompress and compile.with fpc.c

  11. #11
    Member
    Join Date
    Apr 2012
    Location
    London
    Posts
    232
    Thanks
    9
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by algorithm View Post
    It is a length limited Huffman coder. It does not have optimal entropy code. If probabilities are.not very skewed it has very good compression ratio(usually better than block based ANS due to.smaller.block size). Also it is very simple to use .
    Supposes input file with 1,000,000 symbols ( eg 20 distinct symbols) each of divergent probabilities ( eg 0.4 0.15 0.7... So forth). Will FPC still give comparable compressed output size compared to ANS/AC?

Similar Threads

  1. Image compression and end-user display resolution
    By SolidComp in forum Data Compression
    Replies: 3
    Last Post: 2nd February 2019, 11:28
  2. Replies: 11
    Last Post: 6th October 2014, 11:47
  3. Replies: 85
    Last Post: 2nd August 2014, 20:23
  4. user - video compression.
    By WebWalker in forum Data Compression
    Replies: 15
    Last Post: 8th July 2013, 17:13
  5. What is best for a pure Entropy Encoder?
    By biject.bwts in forum Data Compression
    Replies: 4
    Last Post: 28th September 2011, 12:45

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •