Page 1 of 4 123 ... LastLast
Results 1 to 30 of 93

Thread: Open source Kraken/Mermaid/Selkie/LZNA/BitKnit decompression

  1. #1
    Member
    Join Date
    Aug 2016
    Location
    Europe
    Posts
    19
    Thanks
    0
    Thanked 25 Times in 6 Posts
    Last edited by powzix; 1st September 2016 at 19:48.

  2. The Following 8 Users Say Thank You to powzix For This Useful Post:

    algorithm (16th August 2016),Bulat Ziganshin (16th August 2016),comp1 (16th August 2016),encode (16th August 2016),ne0n (18th August 2016),RamiroCruzo (13th September 2016),Shelwien (17th August 2016),Sportman (16th August 2016)

  3. #2
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    772
    Thanks
    63
    Thanked 270 Times in 190 Posts
    0.08 sec. 121.90 MB/s dickens
    0.28 sec. 183.14 MB/s mozilla
    0.08 sec. 121.44 MB/s mr
    0.08 sec. 439.43 MB/s nci
    0.06 sec. 110.52 MB/s ooffice
    0.06 sec. 176.43 MB/s osdb
    0.04 sec. 164.67 MB/s reymont
    0.09 sec. 230.04 MB/s samba
    0.06 sec. 131.85 MB/s sao
    0.30 sec. 137.73 MB/s webster
    0.12 sec. 69.01 MB/s x-ray
    0.02 sec. 353.22 MB/s xml

  4. #3
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,130
    Thanks
    179
    Thanked 919 Times in 467 Posts
    1. Seems compatible with dll 2.20 only, files created with 2.13 and 2.30 produce "size error"
    2. Writes 4-byte filesize at start of the stream, while oodle_api writes 8 bytes; either remove bytes 4-7 or modify lzenc.cpp
    3. Had to modify the source to make it compatible with IntelC/older VS

  5. The Following 2 Users Say Thank You to Shelwien For This Useful Post:

    comp1 (16th August 2016),RamiroCruzo (16th August 2016)

  6. #4
    Member
    Join Date
    Aug 2016
    Location
    Europe
    Posts
    19
    Thanks
    0
    Thanked 25 Times in 6 Posts
    Quote Originally Posted by Shelwien View Post
    1. Seems compatible with dll 2.20 only, files created with 2.13 and 2.30 produce "size error"
    2. Writes 4-byte filesize at start of the stream, while oodle_api writes 8 bytes; either remove bytes 4-7 or modify lzenc.cpp
    3. Had to modify the source to make it compatible with IntelC/older VS
    I added a pre-built unkraken.exe to the github repo. I get the following performance with it:
    dickens.kraken : 2906822 => 10192446 (0.05 seconds, 199.19 MB/s)
    mozilla.kraken : 14236921 => 51220480 (0.15 seconds, 334.50 MB/s)
    mr.kraken : 2910974 => 9970564 (0.04 seconds, 261.88 MB/s)
    nci.kraken : 1894091 => 33553445 (0.04 seconds, 857.14 MB/s)
    ooffice.kraken : 2559090 => 6152192 (0.02 seconds, 258.22 MB/s)
    osdb.kraken : 3080743 => 10085684 (0.03 seconds, 351.99 MB/s)
    reymont.kraken : 1385574 => 6627202 (0.02 seconds, 367.62 MB/s)
    samba.kraken : 4001067 => 21606400 (0.05 seconds, 430.64 MB/s)
    sao.kraken : 4666289 => 7251944 (0.03 seconds, 285.11 MB/s)
    webster.kraken : 8748984 => 41458703 (0.14 seconds, 295.86 MB/s)
    xml.kraken : 484282 => 5345280 (0.01 seconds, 783.98 MB/s)
    x-ray.kraken : 4842659 => 8474240 (0.05 seconds, 163.43 MB/s)


    Note that threaded decoding is not implemented and it doesn't do as much bounds checking as regular kraken.
    Last edited by powzix; 16th August 2016 at 17:18.

  7. #5
    Member
    Join Date
    Aug 2016
    Location
    Europe
    Posts
    19
    Thanks
    0
    Thanked 25 Times in 6 Posts
    I added support for Kraken 2.30 files to the github repo.

  8. #6
    Member
    Join Date
    Aug 2016
    Location
    Europe
    Posts
    19
    Thanks
    0
    Thanked 25 Times in 6 Posts
    Performance with latest version built in x64 mode:

    dickens.kraken : 2901312 => 10192446 (0.03 seconds, 328.03 MB/s)
    mozilla.kraken : 14199727 => 51220480 (0.11 seconds, 458.47 MB/s)
    mr.kraken : 2923528 => 9970564 (0.03 seconds, 359.33 MB/s)
    nci.kraken : 1764475 => 33553445 (0.04 seconds, 855.54 MB/s)
    ooffice.kraken : 2561349 => 6152192 (0.02 seconds, 324.22 MB/s)
    osdb.kraken : 3007629 => 10085684 (0.02 seconds, 429.78 MB/s)
    reymont.kraken : 1387086 => 6627202 (0.01 seconds, 461.50 MB/s)
    samba.kraken : 3994454 => 21606400 (0.04 seconds, 606.30 MB/s)
    sao.kraken : 4677118 => 7251944 (0.02 seconds, 357.47 MB/s)
    webster.kraken : 8658979 => 41458703 (0.13 seconds, 328.90 MB/s)
    xml.kraken : 474924 => 5345280 (0.01 seconds, 875.90 MB/s)
    x-ray.kraken : 4887079 => 8474240 (0.04 seconds, 207.59 MB/s)

  9. #7
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    437
    Thanks
    137
    Thanked 152 Times in 100 Posts
    How do those numbers compare with zstd, brotli, etc on your same machine? I'm also wondering how they compare to the real Kraken implementation. It seems pretty fast for some data sets, but they're all so small it's hard to get a judge on the reliability of that timing. I tried webster with zstd on a local machine (3.2GHz i5) with ZStd and it got around 440MB/s.

    I'm curious how you did this too. Was it simply feeding data in and investigating how the output format changes, or actual reverse engineering of the dll? The latter may have implications to the usability of your code in some countries. Regardless knowing the techniques used means the opensource community can use similar techniques in their own tools, avoiding actual code implementations. Eg the idea of having a quantum of 2 blocks using shared history buffers is presumably a method for interleaving two decoders together and thus making memory fetch delays less severe.

  10. #8
    Member
    Join Date
    Dec 2015
    Location
    US
    Posts
    57
    Thanks
    2
    Thanked 112 Times in 36 Posts
    It's clearly reverse engineered from Oodle 2.2.0 32-bit x86, most likely the Windows DLL. Let's just say I recognize a lot of the code fragments, and they replicate a bunch of random implementation details exactly, in one case appearing to mention instruction selection choices made by VC++ that are different from what GCC does for the same source on Linux.

  11. The Following 3 Users Say Thank You to fgiesen For This Useful Post:

    Bulat Ziganshin (17th August 2016),JamesB (17th August 2016),RamiroCruzo (17th August 2016)

  12. #9
    Member
    Join Date
    Aug 2016
    Location
    Europe
    Posts
    19
    Thanks
    0
    Thanked 25 Times in 6 Posts
    Quote Originally Posted by fgiesen View Post
    It's clearly reverse engineered from Oodle 2.2.0 32-bit x86, most likely the Windows DLL. Let's just say I recognize a lot of the code fragments, and they replicate a bunch of random implementation details exactly, in one case appearing to mention instruction selection choices made by VC++ that are different from what GCC does for the same source on Linux.
    Yes you are right that I analyzed the Oodle 2.2.0 DLL. I have not looked at GCC output on Linux, but if you're referring to the two CMOVs I spent some time getting VC++ to actually use a CMOV.
    Last edited by powzix; 1st September 2016 at 18:57.

  13. #10
    Member
    Join Date
    Aug 2016
    Location
    Europe
    Posts
    19
    Thanks
    0
    Thanked 25 Times in 6 Posts

    Open source Mermaid / Selkie decompressor

    The decompressor is on the below link. It uses the same code base as the kraken decompressor because a lot of the code is shared.

    http://github.com/powzix/kraken

    Also not fuzz safe and might crash on invalid input (unlike the real Selkie / Mermaid decompressor).

  14. The Following 4 Users Say Thank You to powzix For This Useful Post:

    algorithm (17th August 2016),Bulat Ziganshin (17th August 2016),comp1 (17th August 2016),Shelwien (17th August 2016)

  15. #11
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts
    Quote Originally Posted by powzix View Post
    Here's my open source kraken / mermaid / selkie decompressor:

    http://github.com/powzix/kraken

    (Only for Windows and is not safe for invalid input)
    I've been confused by the third-party benchmarks, and now codecs – I thought these were secret, proprietary codecs. Did RAD release the specs for K/M/S? Where is it? Did they release source code for decompression too?

  16. #12
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,130
    Thanks
    179
    Thanked 919 Times in 467 Posts
    No, somebody else just made a good decompiler.

    As to benchmarks, there're now PC games using oodle.

  17. #13
    Member
    Join Date
    Aug 2016
    Location
    Europe
    Posts
    19
    Thanks
    0
    Thanked 25 Times in 6 Posts
    I added LZNA support too.

  18. The Following 3 Users Say Thank You to powzix For This Useful Post:

    algorithm (25th August 2016),RamiroCruzo (24th August 2016),Shelwien (24th August 2016)

  19. #14
    Member
    Join Date
    Aug 2016
    Location
    Europe
    Posts
    19
    Thanks
    0
    Thanked 25 Times in 6 Posts
    Added Bitknit support too!

  20. The Following 2 Users Say Thank You to powzix For This Useful Post:

    algorithm (25th August 2016),RamiroCruzo (25th August 2016)

  21. #15
    Member
    Join Date
    Apr 2015
    Location
    Greece
    Posts
    68
    Thanks
    31
    Thanked 22 Times in 15 Posts
    powzix your work is awesome!

  22. #16
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    645
    Thanks
    205
    Thanked 196 Times in 119 Posts
    I see BitKnit adapts probabilities for rANS every 1024 symbols, but there is some strange weight choice.
    E.g. for literals it uses weight 31 for symbols in the stream ... but then weight 725 for the last symbol - I understand it is to simulate exponential forgetting, but still it could be a bit smoother, e.g. using a table like weight[(model->adapt_interval) >> 3].

    LZNA has very nice symbol-by-symbol rANS adaptation for 8 (3 bit) or 16 (4 bit) size alphabet - with beautiful SIMD for both searching of the symbol and adapting probabilities using CDF only.
    This is exactly what they need in VP10/AV1 (alphabet is limited by 16) ...

  23. #17
    Member
    Join Date
    Aug 2016
    Location
    Europe
    Posts
    19
    Thanks
    0
    Thanked 25 Times in 6 Posts
    Quote Originally Posted by Jarek View Post
    LZNA has very nice symbol-by-symbol rANS adaptation for 8 (3 bit) or 16 (4 bit) size alphabet - with beautiful SIMD for both searching of the symbol and adapting probabilities using CDF only.
    Yes I agree, it's amazingly beautiful. Very good job by fgiesen or cbloom or whoever it was that came up with it..

    BitKnit looks terrible in comparison. Also BitKnit's recent match distance table is weird, it doesn't really keep track of the 8 most recent distances, but only the 2 most recent, and then 6 more... Maybe it remembers only the ones that are actually referenced again at least once. I didn't quite figure out how it works in detail.

  24. #18
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    437
    Thanks
    137
    Thanked 152 Times in 100 Posts
    Quote Originally Posted by Jarek View Post
    LZNA has very nice symbol-by-symbol rANS adaptation for 8 (3 bit) or 16 (4 bit) size alphabet - with beautiful SIMD for both searching of the symbol and adapting probabilities using CDF only.
    This is exactly what they need in VP10/AV1 (alphabet is limited by 16) ...
    The toggling between two states is intriguing. I'm assuming this is an alternative method to get interleaving as flip-flopping between two rans states the throughput vs latency differences in cpu instructions can be avoided. That's if I understand the intention correctly. It makes more sense than my approach of having multiple states with multiple start coordinates in the buffer. That does the same job, but doesn't lend itself well to adapting the probabiltiies.

    The proper renormalisation rather than using static lookup tables makes it easier to exploit SIMD too. I tried with SIMD but it's largely crippled due to the matrix lookups needing slow and expensive gather operations. However it's only viable with small alphabet sizes (hence lznib I guess). A nice algorithm design.

  25. #19
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,130
    Thanks
    179
    Thanked 919 Times in 467 Posts
    Maybe fgiesen.wordpress.com/2016/03/07/repeated-match-offsets-in-bitknit/ ?
    Anyway, they optimize codecs for compression of game resources, so results can be a bit weird.

  26. #20
    Member
    Join Date
    Dec 2015
    Location
    US
    Posts
    57
    Thanks
    2
    Thanked 112 Times in 36 Posts
    Quote Originally Posted by Jarek View Post
    I see BitKnit adapts probabilities for rANS every 1024 symbols, but there is some strange weight choice.
    That's just an oversight, not an intentional feature of the model. Quoting the actual code:

    Code:
            kIncrement = kAvailable / kRenormEvery,
            kFinalBoost = kAvailable % kRenormEvery,
    It's just trying to redistribute kAvailable slots of the probability space between renormalization intervals. To be honest I didn't notice how huge the final boost was relative to the normal increments until months after the bitstream had been frozen!

    A much more uniform way is to just boost the last "kFinalBoost" symbols before a renormalization takes place by a single increment:
    Code:
      symfreq[sym] += kIncrement + (renorm_counter >= kFinalBoost ? 1 : 0);
      if (--renorm_counter == 0) renorm();
    Alas, not the kind of change you make post-bitstream-freeze.

    Quote Originally Posted by Jarek View Post
    LZNA has very nice symbol-by-symbol rANS adaptation for 8 (3 bit) or 16 (4 bit) size alphabet - with beautiful SIMD for both searching of the symbol and adapting probabilities using CDF only.
    This is exactly what they need in VP10/AV1 (alphabet is limited by 16) ...
    This is why I've been insisting that this approach is really quite SIMD-friendly and you might wanna have a look at that for more than a year.

  27. The Following 2 Users Say Thank You to fgiesen For This Useful Post:

    JamesB (30th August 2016),Jarek (26th August 2016)

  28. #21
    Member
    Join Date
    Dec 2015
    Location
    US
    Posts
    57
    Thanks
    2
    Thanked 112 Times in 36 Posts
    Quote Originally Posted by powzix View Post
    BitKnit looks terrible in comparison. Also BitKnit's recent match distance table is weird, it doesn't really keep track of the 8 most recent distances, but only the 2 most recent, and then 6 more... Maybe it remembers only the ones that are actually referenced again at least once. I didn't quite figure out how it works in detail.
    I talked about it in depth here: https://fgiesen.wordpress.com/2016/0...ts-in-bitknit/

    New distances are inserted into the second-to-last slot and only get "promoted" to one of the first 6 if they occur as a repeat offset. BitKnit was designed for a very specific application (mesh and animation data in RAD Granny), not for Oodle. That data is full of structs and the repeats are extremely frequent. Hence the 8 repeat offsets. Inserting the new distances further down is to avoid contaminating the distance set with transient match offsets that only get used once. The match distance coder is intentionally dumb and cheap, optimized for speed more than compression ratio. Was a better trade-off for the data types and decoding speeds it's targeting.

    BitKnit has a few other wrinkles that stem from its origins as special-purpose compressor for one type of file. It turned out to be unexpectedly (and unintendedly!) good for general-purpose compression as well, and it was at a point on the space-speed trade-off curve that nothing else shipping in Oodle at the time was close to, which is how it ended up in there. Post-Kraken, it's back to its original intended role: high compression ratios for specific struct-heavy data (and substantially faster than LZMA/LZNA in that application), OK but not great for anything else.

  29. #22
    Member
    Join Date
    Dec 2015
    Location
    US
    Posts
    57
    Thanks
    2
    Thanked 112 Times in 36 Posts
    Quote Originally Posted by JamesB View Post
    The toggling between two states is intriguing. I'm assuming this is an alternative method to get interleaving as flip-flopping between two rans states the throughput vs latency differences in cpu instructions can be avoided.
    That is yet another topic I've written about publicly before: https://fgiesen.wordpress.com/2015/1...s-in-practice/

    Covers lots of implementation details for the LZNA and BitKnit rANS coders, including rationale. Published last year, something like 4 months after BitKnit was first released (in Granny), and just after it got included in Oodle.

    I have to admit that this whole thing is starting to seriously get on my nerves. Charles and me really do write a lot about this stuff, and usually within months of the release of our implementations; sometimes even before the stuff we're talking about is released. It's not exactly encouraging that detailed write-ups like that will get ignored (with some accusations of "you're keeping everything to yourself and giving nothing back to the community!" on the side), but as soon as someone has reverse-engineered the code it's suddenly interesting. I don't think we need to apologize for actually trying to make back the development costs of software that we've spent several full-time months on before we spill the beans on everything completely. Come on!

  30. The Following 7 Users Say Thank You to fgiesen For This Useful Post:

    Bulat Ziganshin (25th August 2016),Cyan (26th August 2016),JamesB (30th August 2016),Jarek (26th August 2016),nemequ (27th August 2016),Stefan Atev (25th August 2016),Turtle (25th August 2016)

  31. #23
    Member
    Join Date
    Aug 2016
    Location
    Europe
    Posts
    19
    Thanks
    0
    Thanked 25 Times in 6 Posts
    Fgiesen, in kraken 2.2, you decode 3 huffman streams simultaneously (and in 2.3 it's 6..).

    Two going forwards and one going backwards from the end:
    |--> ......................|-->.............................<--|

    Is there a reason why you chose not to go backwards also from the middle position?

  32. #24
    Member
    Join Date
    Dec 2015
    Location
    US
    Posts
    57
    Thanks
    2
    Thanked 112 Times in 36 Posts
    The reverse streams need an extra byte swap on LE platforms, and we care more about LE than BE right now.The original idea with the reverse streams was that forward/reverse pairs provide safety padding for each other, and each cursor can act as "end pointer" for the other. The second turned out to be really useful, the first, not as much as expected.

  33. #25
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    645
    Thanks
    205
    Thanked 196 Times in 119 Posts
    Quote Originally Posted by fgiesen View Post
    To be honest I didn't notice how huge the final boost was relative to the normal increments until months after the bitstream had been frozen!
    From the perspective of compressors for games, freezing bitstream doesn't seem to be an issue (?) - you don't need forward or backward compatibility, just within a given game?
    Quote Originally Posted by fgiesen View Post
    A much more uniform way is to just boost the last "kFinalBoost" symbols before a renormalization takes place by a single increment(...)
    The problem of optimal probability adaptation is a tough one, I've tried to find something better like trend-seeking predicting e.g. linear trend of evolution, but it was more complex and not really better - the exponential forgetting seems just the best way.
    Increasing by one until some kFinalBoost is reasonable, but you could getter better behavior at similar cost by just tabling the weighs, e.g. weight[(model->adapt_interval) >> 3].
    Quote Originally Posted by fgiesen View Post
    This is why I've been insisting that this approach is really quite SIMD-friendly and you might wanna have a look at that for more than a year.
    So personally I am alone at my University with information theory, I have really tried to find a scientific collaboration on a related topics, but it seems hopeless here. Sure I could spend a year to write a great compressor and make it open source ... but so what? I am evaluated by the number of papers and teaching. Nobody is interested in ANS here, I didn't get a dime from it.

    A few months ago I had many discussions with VP10 and Daala team, e.g. here: http://lists.xiph.org/pipermail/daal...il/thread.html
    I haven't followed it later, but originally Daala was advocating their implementation of Moffat's approximation of range coding:
    https://people.xiph.org/~tterribe/tm...c%20Coding.pdf
    there is no multiplication, but the inaccuracy means 1-3% ratio loss.
    They emphasize advantage as working on non-power-of-two-denominator (impractical for rANS), because they use adaptation of type: Pr(s) ~ #s / total.
    But this adaptation has poor forgetting (far symbols have the same impact as close ones) - tests show that exponential forgetting is essentially better ... and use power-of-two-denominator as you describe.
    As their alphabets are limited to 16 (for each of hundreds of contexts), indeed I was advocating exactly what you have in LZNA: using SIMD for both search and symbol update (but I haven't found motivation to learn programming with SIMD).
    Currently both rANS and Daala entropy coders are being developed for AV1: https://aomedia.googlesource.com/aom/+/master/aom_dsp/
    It would be great if RAD Game Tools would joint the AOM ...

  34. #26
    Member
    Join Date
    Dec 2015
    Location
    US
    Posts
    57
    Thanks
    2
    Thanked 112 Times in 36 Posts
    Quote Originally Posted by Jarek View Post
    From the perspective of compressors for games, freezing bitstream doesn't seem to be an issue (?) - you don't need forward or backward compatibility, just within a given game?
    Backwards compatibility is non-negotiable. Otherwise every single format change would create what is effectively a new fork of the project that needs to be maintained for, at the very least, several years. So if we were to say at some point discover a buffer overrun that's been in the code base for a year, we would now need to issue separate patches for every format variant we've supported in that time, and we'd have a lot of support issues with customers accidentally getting the wrong version etc.

    The case for forwards compatibility is things like DLC (downloadable content) - new content for an existing game that's released several months after the original game came out. If we can get encoder improvements without changing the format, that's good. Any breaking changes are really bad, and anything that is exclusive (you can either use the old library version _or_ the new one) is a show-stopper if the new library version also happens to add something that the DLC parts depend on.

    Quote Originally Posted by Jarek View Post
    Currently both rANS and Daala entropy coders are being developed for AV1: https://aomedia.googlesource.com/aom/+/master/aom_dsp/
    It would be great if RAD Game Tools would joint the AOM ...
    We're not a big enough company to be able to work on that kind of project "on the side"! We have a total of 9 devs working on currently 8 different products.

  35. The Following User Says Thank You to fgiesen For This Useful Post:

    Jarek (28th August 2016)

  36. #27
    Member
    Join Date
    Jan 2014
    Location
    Bothell, Washington, USA
    Posts
    685
    Thanks
    153
    Thanked 177 Times in 105 Posts
    Quote Originally Posted by fgiesen View Post
    We're not a big enough company to be able to work on that kind of project "on the side"! We have a total of 9 devs working on currently 8 different products.
    Sometimes the world is much smaller than it seems. I looked up Rad Game Tools and found it's only five miles from my house.

  37. #28
    Member
    Join Date
    Nov 2014
    Location
    Earth
    Posts
    38
    Thanks
    0
    Thanked 77 Times in 19 Posts
    fgiesen and cbloom, I am wondering if you've considered other options for opening up these compression formats, given the high level of interest in the formats and the benefit to the community in making them more open? I feel that if your work remains locked up, it will not gain the level of appreciation and recognition that it deserves.

    One idea is to release the code as GPL but also sell commercial licenses. Then it could be used in GPL-ed open source projects, but commercial games would still have to pay RAD for a commercial license.

    Another idea is to release a format specification, but no code. Then people could, over time, write their own compressors and decompressors. The idea is that this would indirectly protect RAD's particular implementation, since then it wouldn't be as worthwhile for anyone to bother reverse engineering it.

    Actions like these might actually *increase* revenue for RAD by virtue of generating more free advertising for their work. RAD should retain an advantage anyway: I understand that their offering includes excellent compressors *and* decompressors, is professionally tested, is professionally supported, has been optimized and tested for gaming platforms, comes with the rest of Oodle, and presumably is easy to license together with other RAD products that games developers may want.

    Anyway, I'm just throwing some ideas out there for ways everyone might come out ahead. I'm not really interested in getting into arguments about any specific idea.

  38. #29
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    437
    Thanks
    137
    Thanked 152 Times in 100 Posts
    Quote Originally Posted by fgiesen View Post
    That is yet another topic I've written about publicly before: https://fgiesen.wordpress.com/2015/1...s-in-practice/

    Covers lots of implementation details for the LZNA and BitKnit rANS coders, including rationale. Published last year, something like 4 months after BitKnit was first released (in Granny), and just after it got included in Oodle.

    I have to admit that this whole thing is starting to seriously get on my nerves. Charles and me really do write a lot about this stuff, and usually within months of the release of our implementations; sometimes even before the stuff we're talking about is released. It's not exactly encouraging that detailed write-ups like that will get ignored (with some accusations of "you're keeping everything to yourself and giving nothing back to the community!" on the side), but as soon as someone has reverse-engineered the code it's suddenly interesting. I don't think we need to apologize for actually trying to make back the development costs of software that we've spent several full-time months on before we spill the beans on everything completely. Come on!
    I did read that article, but managed to forget it apparently! One day I'll get around to exploring adaptive rANS and I'm certain your blog posts will come in handy for that, but like many others I spend most of my days doing other work so it's hard to fit it all in.

    So I for one though are really happy to see your posts. They're definitely informative and I think you and Charles have been surprisingly open given you're a commercial outfit. Keep up the great work!

  39. #30
    Member
    Join Date
    Nov 2015
    Location
    ?l?nsk, PL
    Posts
    81
    Thanks
    9
    Thanked 13 Times in 11 Posts
    The situation reminds me of Chris Martelock.
    Hope it ends differently.

Page 1 of 4 123 ... LastLast

Similar Threads

  1. Mermaid and Selkie join Kraken
    By SolidComp in forum Data Compression
    Replies: 29
    Last Post: 1st February 2019, 07:54
  2. Why not open source?
    By nemequ in forum Data Compression
    Replies: 65
    Last Post: 25th November 2013, 23:05
  3. MCM open source
    By Mat Chartier in forum Data Compression
    Replies: 12
    Last Post: 29th August 2013, 20:22
  4. Open source JPEG compressors
    By inikep in forum Data Compression
    Replies: 8
    Last Post: 22nd October 2011, 00:16
  5. PeaZip - open source archiver
    By squxe in forum Data Compression
    Replies: 1
    Last Post: 3rd December 2009, 22:01

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •