Results 1 to 14 of 14

Thread: Looking for powerefficient, lossy, raw photocompression on RPI3

  1. #1
    Member
    Join Date
    Dec 2016
    Location
    the Netherlands
    Posts
    2
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Looking for powerefficient, lossy, raw photocompression on RPI3

    Hello,

    For part of my photography workflow I want to use my Raspberry Pi 3 to create a compressed copy from a RAW file. In my case CR2 files from a Canon EOS 6D, measuring around 25MB per file.
    Goal is to get very small (under 1MB) files with 'reasonable' quality files that still can be edited (white balance correction etc) so the full dynamic range needs to be retained.
    So far I'v been using dcraw to convert CR2 to 16bit-per-channel (rgb48le, 5496x3670) PPM files, then converted using jpeg2000(grok) to J2K. It's amazing how good the jpeg2000 compression is at low bitrates! Only problem I have on the battery powered RPI is that the whole conversion takes ages:

    30 seconds: full raw file dcraw > ppm
    344 seconds: ppm > jpeg2000

    so all together it takes more than 6 minutes, that's too long in my case.
    When I use dcraw's half size option (rgb48le, 2748x1835) I can live with the loss of resolution and times get better:

    11 seconds: 'half size' option dcraw > ppm
    35 seconds: ppm > jpeg2000

    While already better, I'm looking/hoping for JPEG-LS kind of speed (less than 2 seconds) but FFMPEG tells me that JPEG-LS does not support rgb48le encoding

    So any suggestions are very welcome!

    Thank you in advance,

    Mike.

  2. #2
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts
    Have you tried JPEG-XR? It's supposed to be more computationally efficient than JPEG-2000. I don't know if it supports RGB48le, but it does support a 16-bit floating point format.

  3. #3
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by Embee View Post
    16bit-per-channel
    Store it as two separate rgb24 images. Then you can use PNG or WebP for the resulting 8 big images. This is usually much better than just storing a single PNG with 16 bit channels, because PNG stores the lower and higher bits with the same entropy code and the entropy in them is (in photographs) as very different.

  4. #4
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    437
    Thanks
    1
    Thanked 96 Times in 57 Posts
    Quote Originally Posted by Embee View Post
    While already better, I'm looking/hoping for JPEG-LS kind of speed (less than 2 seconds) but FFMPEG tells me that JPEG-LS does not support rgb48le encoding
    JPEG-LS supports up to 16bits per component. If FFMPEG does not implement this, then shame on them. This code here https://github.com/thorfdbg/libjpeg also includes JPEG-LS and as such supports up to 16bpp. However, note that JPEG-LS is lossless or near-lossless, and hence compression performance is limited to about 2:1, so it's probably not satisfying your demands. Additional suggestions beyond JPEG 2000 are JPEG XT (lossy or lossless, up to 16bpp, same codec) or JPEG XR (already stated). All of them support 16bpp input. You find the JPEG XR reference implementation on www.jpeg.org, btw, and I already pointed you to the JPEG XT reference implementation which you can also find on www.jpeg.org.

  5. #5
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by thorfdbg View Post
    JPEG-LS
    How does JPEG-LS compare in decoding speed with PNG and WebP?

  6. #6
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    437
    Thanks
    1
    Thanked 96 Times in 57 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    How does JPEG-LS compare in decoding speed with PNG and WebP?
    I did not compare with PNG, but I would expect it to be considerably faster than WebP. JPEG LS is pretty lightweight.

  7. #7
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by thorfdbg View Post
    I did not compare with PNG, but I would expect it to be considerably faster than WebP. JPEG LS is pretty lightweight.
    I thought JPEG LS does context modeling. Wouldn't it make it decode slower than both PNG and WebP lossless (both of which don't do context modeling)?

  8. #8
    Member
    Join Date
    Dec 2016
    Location
    Cambridge, MA
    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    I thought JPEG LS does context modeling. Wouldn't it make it decode slower than both PNG and WebP lossless (both of which don't do context modeling)?
    JPEG LS uses context-adaptive Golomb-Rice coding. The context is determined by the neighboring pixels (http://www.labs.hp.com/research/info...L-98-193R1.pdf).

    In PNG, I assume the bulk of the complexity is the Lempel-Ziv coder (please correct me if I'm wrong). To me, intuitively, the dictionary lookup of the LZ coder would still be slower than Golomb-Rice.

  9. #9
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by taliho View Post
    To me, intuitively, the dictionary lookup of the LZ coder would still be slower than Golomb-Rice.
    Why do you expect LZ to be slower than Golomb-Rice?

    LZ is just a memcpy at decoding time, and often the fastest possible way to decode data. Most of the worlds fastest decoders are LZ77 based. LZ is typically much faster to decode than Huffman, Huffman is faster than Golomb-Rice (adaptive or not).

    LZ is of course not at all successful with noisy photographs, and WebP lossless typically turns it of for photographs, and the LZ77 phase is replaced by an improved rle-encoding. It estimates the resulting entropy using a few strategies and uses the strategy that makes most sense.

  10. #10
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    437
    Thanks
    1
    Thanked 96 Times in 57 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Why do you expect LZ to be slower than Golomb-Rice? LZ is just a memcpy at decoding time, and often the fastest possible way to decode data.
    Unfortunately, it is a non-local data lookup, which can cause caching issues, and it requires searching and comparions at encoding times, so it is not quite as symmetric as LS.
    Quote Originally Posted by Jyrki Alakuijala View Post
    Most of the worlds fastest decoders are LZ77 based. LZ is typically much faster to decode than Huffman, Huffman is faster than Golomb-Rice (adaptive or not).
    I really wonder why Huffman should be faster than G-R. G-R is some special form of Huffman coding, the difference being that for the latter the code is actually parametrized and not "free" as for Huffman coding. Thus, from this follows that Golomb-Rice can be implemented at least as fast as Huffman.

  11. #11
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by thorfdbg View Post
    Unfortunately, it is a non-local data lookup, which can cause caching issues, and it requires searching and comparions at encoding times, so it is not quite as symmetric as LS. I really wonder why Huffman should be faster than G-R. G-R is some special form of Huffman coding, the difference being that for the latter the code is actually parametrized and not "free" as for Huffman coding. Thus, from this follows that Golomb-Rice can be implemented at least as fast as Huffman.
    Prefix decoding speed is related to the density of the codes. Non-adaptive Golomb-Rice is a less dense prefix code than Huffman. Less dense code needs more memory bandwidth, and this typically makes it slower even when the same prefix decoding algorithm can do the bulk processing.

    If Golomb-Rice is used adaptively -- even with the simplest possible context models there is a dependency chain on previous pixel, it can reduce internal latency hiding and out-of-ordering that the processor would otherwise be able to do. This gives a small further degradation for decoding speed. With more complex context models one gets more decoding speed reduction.

    Is there a high quality open sourced implementation of JPEG LS? Perhaps it would be possible to just try it against PNG and WebP lossless?

  12. #12
    Member
    Join Date
    Dec 2016
    Location
    Cambridge, MA
    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by thorfdbg View Post
    Unfortunately, it is a non-local data lookup, which can cause caching issues, and it requires searching and comparions at encoding times, so it is not quite as symmetric as LS. I really wonder why Huffman should be faster than G-R. G-R is some special form of Huffman coding, the difference being that for the latter the code is actually parametrized and not "free" as for Huffman coding. Thus, from this follows that Golomb-Rice can be implemented at least as fast as Huffman.
    Thomas, any chance you could clarify the 'non-local' part? Are you saying the LZ dictionary is not going to fit in the L1/L2 cache?

    Quote Originally Posted by thorfdbg View Post
    I really wonder why Huffman should be faster than G-R. G-R is some special form of Huffman coding, the difference being that for the latter the code is actually parametrized and not "free" as for Huffman coding. Thus, from this follows that Golomb-Rice can be implemented at least as fast as Huffman..
    I agree that G-R encoding can be implemented just as fast Huffman if we create a look-up table of the G-R codewords. But the usual implementation of G-R should be a bit slower than Huffman

  13. #13
    Member
    Join Date
    Dec 2016
    Location
    Cambridge, MA
    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Prefix decoding speed is related to the density of the codes. Non-adaptive Golomb-Rice is a less dense prefix code than Huffman. Less dense code needs more memory bandwidth, and this typically makes it slower even when the same prefix decoding algorithm can do the bulk processing.

    If Golomb-Rice is used adaptively -- even with the simplest possible context models there is a dependency chain on previous pixel, it can reduce internal latency hiding and out-of-ordering that the processor would otherwise be able to do. This gives a small further degradation for decoding speed. With more complex context models one gets more decoding speed reduction.

    Is there a high quality open sourced implementation of JPEG LS? Perhaps it would be possible to just try it against PNG and WebP lossless?
    Jyrki, any chance you could clarify the 'density of the code' part?

    The decoding procedure for G-R is very different to Huffman. G-R does not use a look-up table.
    The decoding procedure for G-R is:
    1. Read quotient q (via unary coding).
    2. Read M residual bits, and convert to decimal (M_decimal)
    3. Output symbol : q^M + M_decimal

  14. #14
    Member
    Join Date
    Dec 2016
    Location
    Cambridge, MA
    Posts
    7
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Why do you expect LZ to be slower than Golomb-Rice?

    LZ is just a memcpy at decoding time, and often the fastest possible way to decode data. Most of the worlds fastest decoders are LZ77 based. LZ is typically much faster to decode than Huffman, Huffman is faster than Golomb-Rice (adaptive or not).

    LZ is of course not at all successful with noisy photographs, and WebP lossless typically turns it of for photographs, and the LZ77 phase is replaced by an improved rle-encoding. It estimates the resulting entropy using a few strategies and uses the strategy that makes most sense.
    Jyrki, I very much agree that the decoding of LZ is extremely fast
    Last edited by taliho; 4th January 2017 at 23:08.

Similar Threads

  1. lossless recompression of camera raw
    By Stephan Busch in forum Data Compression
    Replies: 74
    Last Post: 22nd March 2016, 23:40
  2. JPEG Lossy (new algorithm)
    By lorents17 in forum Data Compression
    Replies: 1
    Last Post: 25th September 2015, 00:22
  3. Lossy DEFLATE, lossy PNG
    By porneL in forum Data Compression
    Replies: 11
    Last Post: 15th August 2013, 19:00
  4. Lossless vs. Lossy
    By WillatSMU in forum Data Compression
    Replies: 7
    Last Post: 11th May 2012, 17:29
  5. Where can I get best Raw video samples?
    By mark in forum Data Compression
    Replies: 4
    Last Post: 17th October 2011, 22:43

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •