Results 1 to 4 of 4

Thread: TurboHF: Huffman Coding second reincarnation beats asymmetric numeral systems+FSE

  1. #1
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    456
    Thanks
    46
    Thanked 164 Times in 118 Posts

    Exclamation TurboHF: Huffman Coding second reincarnation beats asymmetric numeral systems+FSE

    for the commonly used uniform distribution.
    See: Entropy Coding Benchmark

  2. #2
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Brotli's entropy encoder might be interesting for such comparison. A quick try on it gives 560795962 bytes (with -q 9). I suspect -q 11 will give 5-10 % more.

    I got this result by removing the LZ77 and static dictionary part from the brotli encoder and the remaining code encodes at 40 MB/s (-q 9) and the result decoded at 110 MB/s. I suspect that one could get twice the speed by cutting it out more thoroughly.

  3. #3
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    456
    Thanks
    46
    Thanked 164 Times in 118 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Brotli's entropy encoder might be interesting for such comparison.
    Yes, but there is no direct interface to the huffman coder in brotli.
    I think, your result is not based on a order-0 huffman coder.

  4. #4
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by dnd View Post
    Yes, but there is no direct interface to the huffman coder in brotli.
    I think, your result is not based on a order-0 huffman coder.
    Yes, the literal coding in Brotli is typically order-1 (or two, depending how one counts) based on binary-oring LUT-values of previous two bytes. With quality 9 there is only limited dynamic analysis of the context, and the context table is filled with small number of contexts (I think I used 15 contexts to be used in such a setting). With quality 10 and 11 the context count can go up to 64.

    Also, there is a dynamic block length dance going on. That is also somewhat limited at quality 9 and way more advanced at quality 10 and 11.

Similar Threads

  1. Replies: 33
    Last Post: 25th June 2019, 12:15
  2. List of Asymmetric Numeral Systems implementations
    By Jarek in forum Data Compression
    Replies: 13
    Last Post: 1st March 2018, 00:06
  3. Replies: 32
    Last Post: 8th January 2016, 10:47
  4. TurboHF - 1GB/s Huffman Coding Reincarnation
    By dnd in forum Data Compression
    Replies: 13
    Last Post: 10th September 2015, 20:06
  5. LzTurbo Byte coding vs. Zhuff FSE
    By dnd in forum Data Compression
    Replies: 57
    Last Post: 24th January 2014, 04:25

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •