Results 1 to 16 of 16

Thread: Diverse third-party ecosystem for optimization of webp and JPEG-XR images

  1. #1
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts

    Diverse third-party ecosystem for optimization of webp and JPEG-XR images

    Hi all,

    As you're well aware, there are lots of JPEG and PNG optimizers and encoders, both lossy and lossless. However, newer image formats like webp and JPEG-XR each have only one reference encoder, from Google and Microsoft, respectively, and no GUI tools for working with these formats.

    Do the webp and JPEG-XR formats have the sort of headroom for optimization that JPEG and PNG do? Is there anything about the formats that has prevented the emergence of third-party encoders and optimizers? Have any of you looked at building new encoders or optimizers?

    Cheers,

    Joe

  2. #2
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by SolidComp View Post
    Hi all,

    As you're well aware, there are lots of JPEG and PNG optimizers and encoders, both lossy and lossless. However, newer image formats like webp and JPEG-XR each have only one reference encoder, from Google and Microsoft, respectively, and no GUI tools for working with these formats.

    Do the webp and JPEG-XR formats have the sort of headroom for optimization that JPEG and PNG do? Is there anything about the formats that has prevented the emergence of third-party encoders and optimizers? Have any of you looked at building new encoders or optimizers?
    WebP engineering team responds to issues and proposals at: https://bugs.chromium.org/p/webp/issues/list -- so it is possible to ask them directly why they don't use a guetzli like approach.

    About why we build Guetzli and ZopfliPNG:

    We built guetzli on top of jpeg, because jpeg performed better at high quality than other alternatives, particularly the lack of a good integral transform and the YUV420 color space reduce the performance of WebP at highest quality (break even with libjpeg was around quality 77, and worse above 85 in libjpeg quality). Better meaning here: faster decode (up to around 6x faster) and less bytes needed at high quality (in comparison to butteraugli scores). New codecs are often far superior at very low bit rates, but that was not what we were interested of. At very low bit rates the images will always be distorted in various ways, typically they are made smoother and the observable surface properties are replaced by plastic feel.

    We built ZopfliPNG just because we could -- after building Zopfli and Lode being the author of LodePNG, it was a natural thing to build ZopfliPNG, too. No big strategic thinking. (And naturally ZopfliPNG is a worse solution than our previous work with the WebP lossless format. Also, ZopfliPNG is worse than sending an uncompressed but filtered PNG with brotli content encoding.)

  3. #3
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    437
    Thanks
    1
    Thanked 96 Times in 57 Posts
    Quote Originally Posted by SolidComp View Post
    Do the webp and JPEG-XR formats have the sort of headroom for optimization that JPEG and PNG do? Is there anything about the formats that has prevented the emergence of third-party encoders and optimizers? Have any of you looked at building new encoders or optimizers?
    Actually, the JPEG XR reference software is *not* from Microsoft. Microsoft funded its development. The HDPhoto Developer Kit is from Microsoft, but this is a different software, and it is neither exactly JPEG XR. The differences are subtle, though, and mostly address defects we found during standardization. As for JPEG XR: Yes, there is a little bit of headroom. It offers variable quantization which is not used by the reference software by default. This alone can give you a PSNR advantage of 0.2dB (depending on the image, if I recall correctly, but I can dig out the publication). Also, there is *some* possibility to tune it towards other quality indices like SSIM - though it is clearly not as flexible (nor as good) as JPEG 2000 is.

  4. #4
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts
    Quote Originally Posted by thorfdbg View Post
    Actually, the JPEG XR reference software is *not* from Microsoft. Microsoft funded its development. The HDPhoto Developer Kit is from Microsoft, but this is a different software, and it is neither exactly JPEG XR. The differences are subtle, though, and mostly address defects we found during standardization. As for JPEG XR: Yes, there is a little bit of headroom. It offers variable quantization which is not used by the reference software by default. This alone can give you a PSNR advantage of 0.2dB (depending on the image, if I recall correctly, but I can dig out the publication). Also, there is *some* possibility to tune it towards other quality indices like SSIM - though it is clearly not as flexible (nor as good) as JPEG 2000 is.
    Thanks for the info. This is interesting. For some reason I thought JPEG-XR was supposed to be better than JPEG 2000 – I think I saw a research paper to that effect. JPEG-XR will probably die off soon since Microsoft has been so lazy with it, and no one else is exploiting its headroom, HDR and other features, etc. or building better encoders, e.g. a libjpegxr-turbo or MozJPEG-XR or a Guetzli approach.

    If JPEG 2000 is better, why don't people use it? It's always seemed like a confusing and difficult format to adopt. Just viewing JPEG 2000 images on Windows 10 is difficult. I had to do some hacking to help my girlfriend view her MRIs in some format called DICOM where some of the images were JPEG 2000. Is it patent encumbered? I remember hearing something about that.

    Ancillary question: Do you happen to know if Safari on macOS and iOS actually displays JPEG 2000 files? It's supposed to, but I've never been able to find any mention of it in Apple's documentation, and I have no devices to test. That also makes me wonder if patent issues have prompted Apple to keep quiet about it.

  5. #5
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by SolidComp View Post
    If JPEG 2000 is better, why don't people use it?
    Better is a complicated word.

    JPEG 2000 is better than JPEG at very low bit rates for both human viewers and for simple metrics like PSNR.

    JPEG 2000 is better (10 % less bloaty) than JPEG at high quality for simple metrics like PSNR.

    JPEG 2000 is more feature-rich.

    JPEG 2000 is worse (10 % more bloaty) than JPEG at high quality for human viewers.

    Most of the digital images overall -- and particularly the images in the internet are high quality images consumed by human viewers. JPEG fits this purpose better than JPEG 2000.

  6. #6
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Better is a complicated word.

    JPEG 2000 is better than JPEG at very low bit rates for both human viewers and for simple metrics like PSNR.

    JPEG 2000 is better (10 % less bloaty) than JPEG at high quality for simple metrics like PSNR.

    JPEG 2000 is more feature-rich.

    JPEG 2000 is worse (10 % more bloaty) than JPEG at high quality for human viewers.

    Most of the digital images overall -- and particularly the images in the internet are high quality images consumed by human viewers. JPEG fits this purpose better than JPEG 2000.
    I meant better than JPEG-XR, which is what thorfbdg had said. Of course JPEG-2000 is better than JPEG – I mean it has 2000 in the name That's what people did in the 1990s when they wanted to mark something as futuristic and high-tech.

    I also thought that JPEG-XR came after JPEG-2000, which implied to me that it would have to be an improvement over it. And it wasn't as messy on the spec side – have you seen all the JPEG-2000 specs?

  7. #7
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts
    By the way, this reminds me that the plethora of available tools to optimize JPEGs and PNGs makes the case for webp (and JPEG-XR) weaker, or at least more complicated. In reality, webp is not competing with libjpeg or even libjpeg-turbo. webp is competing with the gauntlet of tools that I can put JPEGs through, including lossy optimizers. Think of everything that's in FileOptimizer, plus TinyPNG.com (which does JPEGs too), etc. The same applies to PNGs.

    Google must have beautiful, rich, and useful data on webp performance in all aspects, but for some reason it refuses to release it. All I've seen are old comparisons between webp and PNGs where they used pngcrush and pngout, for example. But in the wild we're not limited to pngcrush and pngout, and those aren't even the most powerful tools. Does webp actually produce smaller files than JPEG and PNG if we use all the tools available for those legacy formats? I don't know. I'm not sure if anyone knows. My hunch is that it probably does, but the difference isn't as big as Google claims in its dated tests on the webp website. I'm more confident that webp can beat superoptimized PNGs than superoptimized JPEGs, but the JPEG case is the most complicated because of the difficulty of apples-to-apples comparison in terms of "quality". We have quality integers that we're throwing around that aren't commensurable across formats and most web development shops don't have enough staff to deep dive into the finer points of image optimization.

  8. #8
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts
    I've also wondered about combining raster and vector approaches in the same format, since it's clear that some images would benefit much more from one or the other.

    I also want to see a format that explicitly accounts for image sets in addition to standalone images, since a lot of groups of images on webpages, social media, etc. feature the same settings and subjects and have lots of encodable overlap. It sounds sort of like video encoding, but it's going to be much less similar and more complicated than adjacent frames in video.

    The DNN researchers at Google have done some impressive work with images and image recognition, and I wonder if that could be applied to an approach that uses reference images for specific people, places, logos, etc. and does something like delta encoding for a bunch of images that feature those people, places, logos, etc. At first blush, this seems unlikely to work out – to lead to significant savings – but I think my initial intuition is wrong and that it's just a matter of much better algorithms and constructs.

    Another idea I had was what sorts of things we could do at the time of image acquisition – when we're composing and taking photographs – that would help achieve smaller file sizes at a given level of quality. I've never read anyone talk about this before, but there are so many things going on in, say, headshots, product photography, etc. in terms of backgrounds, lighting, colors, color gradients, angles, reflectance, etc. It might be possible to subtly manipulate these things in ways that lead to surprisingly smaller JPEGs, webps, or PIKs without significantly impacting (or even improving) how humans perceive the image.

  9. #9
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by SolidComp View Post
    I've also wondered about combining raster and vector approaches in the same format, since it's clear that some images would benefit much more from one or the other.

    I also want to see a format that explicitly accounts for image sets in addition to standalone images, since a lot of groups of images on webpages, social media, etc. feature the same settings and subjects and have lots of encodable overlap. It sounds sort of like video encoding, but it's going to be much less similar and more complicated than adjacent frames in video.

    The DNN researchers at Google have done some impressive work with images and image recognition, and I wonder if that could be applied to an approach that uses reference images for specific people, places, logos, etc. and does something like delta encoding for a bunch of images that feature those people, places, logos, etc. At first blush, this seems unlikely to work out – to lead to significant savings – but I think my initial intuition is wrong and that it's just a matter of much better algorithms and constructs.

    Another idea I had was what sorts of things we could do at the time of image acquisition – when we're composing and taking photographs – that would help achieve smaller file sizes at a given level of quality. I've never read anyone talk about this before, but there are so many things going on in, say, headshots, product photography, etc. in terms of backgrounds, lighting, colors, color gradients, angles, reflectance, etc. It might be possible to subtly manipulate these things in ways that lead to surprisingly smaller JPEGs, webps, or PIKs without significantly impacting (or even improving) how humans perceive the image.
    That is a lot of good ideas.

  10. The Following User Says Thank You to Jyrki Alakuijala For This Useful Post:

    SolidComp (9th February 2018)

  11. #10
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by SolidComp View Post
    Google must have beautiful, rich, and useful data on webp performance in all aspects, but for some reason it refuses to release it.
    Post a request to release such data on https://bugs.chromium.org/p/webp/issues/list

    My current understanding that WebP lossy is justified by PSNR, Y-SSIM and RGB-SSIM plots.

    Guetzli is justified by Butteraugli and our somewhat outrageous claims are made slightly more plausible by a human comparison against libjpeg.

    Quote Originally Posted by SolidComp View Post
    All I've seen are old comparisons between webp and PNGs where they used pngcrush and pngout, for example. But in the wild we're not limited to pngcrush and pngout, and those aren't even the most powerful tools.
    In the wild many people use the default settings. Back then when I did that study 4 % of PNGs in the internet were 16-bits per channel. Definitely they were not optimized at all, and needed to be reoptimized to be fairly comparable with WebP lossless.

    Back then WebP lossless with default settings gave a 40+ % reduction from PNG file size considering PNGs as they were on the internet (without recompression).

    Quote Originally Posted by SolidComp View Post
    Does webp actually produce smaller files than JPEG and PNG if we use all the tools available for those legacy formats? I don't know.
    At high quality JPEG wins over WebP, but PNG wins over WebP for more random reasons. About 2 % of PNGs in the internet cannot be further compressed by WebP lossless. Some part of that 2 % may have gone through a near-lossless PNG conversion (possibly about 0.5 %).

    Quote Originally Posted by SolidComp View Post
    I'm not sure if anyone knows. My hunch is that it probably does, but the difference isn't as big as Google claims in its dated tests on the webp website.
    Lossless testing is not difficult. Nowadays you get 26-27 % gain from WebP lossless against ZopfliPNG. ZopfliPNG has improved PNGs (over from the PNGOUT days), but also WebP lossless has improved a few percent in density.

    Quote Originally Posted by SolidComp View Post
    I'm more confident that webp can beat superoptimized PNGs than superoptimized JPEGs, but the JPEG case is the most complicated because of the difficulty of apples-to-apples comparison in terms of "quality".
    Yeah. I'm fully with you.

    Try compressing a big one megabyte photo down to 15 kB, and you will be convinced that there are some use cases where WebP lossy is far superior to JPEG. If you take graphics images, WebP can be competitive even at higher quality settings.

    For high quality photography, I (and butteraugli) believe that JPEG is actually a better than WebP.

    Quote Originally Posted by SolidComp View Post
    We have quality integers that we're throwing around that aren't commensurable across formats and most web development shops don't have enough staff to deep dive into the finer points of image optimization.
    This is why I think it is important that default settings don't spoil the image and still compress relatively well.

    Last time I checked with butteraugli WebP lossy at quality 85 was somewhat similar to JPEG YUV444 at quality 77, i.e., that was the place where the size and quality curves met. Below JPEG quality 77 WebP lossy wins, above JPEG quality 77 JPEG wins (for photography). At that quality WebP quality 85 would match with JPEG quality 77, so there may indeed be a difference on how to use the quality settings in these two encoders. This was based on the maximum compression artefact in an image -- averaging from 1000 images.

    Depending on methodology, which version of butteraugli one uses, or if one uses ssimulacra or ms-ssim on comparison, one can get different equivalence numbers.

    WebP's artefacts tend to be more friendly to the eye than JPEG's, so when things start to go south, WebP may be more pleasant.

  12. The Following User Says Thank You to Jyrki Alakuijala For This Useful Post:

    SolidComp (9th February 2018)

  13. #11
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts
    Thanks for rich and thoughtful reply Jyrki. On PNG, do you have any idea how webp fares against Kornel's PNG quant? When it comes to PNG in general, I think people get distracted by the fact that it's "lossless". Yes it's defined and specced as a lossless compression of bitmap data, but I think lossy PNG optimization is often undetectable by users, and is much less problematic than lossy JPEGs. PNGs are simple. (And the hybrid vector/raster image format I have in mind would significantly shrink a lot of PNGs.)

    On my idea of reference images and delta encoding, it would be awesome if a few prior photos of a person or mountain made subsequent photos much smaller. This is going to be complicated and hard, but I think in the end it will work.

    The DNN team recently did some work on security camera footage enhancement. Security cameras suck, seem to be a decade or more old, with fuzzy pictures. It would be interesting if any specific crime case, they could take reference images of employees (of a store, bank, etc.) and get various footage of those employees from the security cameras. Then analyze the delta between known, high quality photos and grainy security cam footage of those same people. They could then build a model of those specific security cameras' behavior in how they capture and compress a person's face, in great detail, with specific features, morphological patterns, etc. Then they could use this work to help identify criminals by taking a lineup of suspects, getting the kind of quality reference photos previously mentioned, and have their model predict how that specific security camera in the bank or whatever would tend to encode that person's face. If this could be made to be sufficiently reliable, it would be a great tool.

  14. #12
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Quote Originally Posted by SolidComp View Post
    I've also wondered about combining raster and vector approaches in the same format, since it's clear that some images would benefit much more from one or the other.
    Well, there's PDF, SVG and such where you can combine both, though they can't be used to compress a given image automatically.

    Quote Originally Posted by SolidComp View Post
    I also want to see a format that explicitly accounts for image sets in addition to standalone images, since a lot of groups of images on webpages, social media, etc. feature the same settings and subjects and have lots of encodable overlap. It sounds sort of like video encoding, but it's going to be much less similar and more complicated than adjacent frames in video.
    WaveOne made some progress to this last year using machine learning, see the paragraph "Domain-adaptive compression" on this page. Additionally, the algorithm compresses "common" objects like faces good at very low bitrates, see the result images in the paper.
    http://schnaader.info
    Damn kids. They're all alike.

  15. #13
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    437
    Thanks
    1
    Thanked 96 Times in 57 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    JPEG 2000 is worse (10 % more bloaty) than JPEG at high quality for human viewers.
    Sorry, that's simply not correct. It is a matter of how you use it and how you drive it. In the language of JPEG, "it is a matter of the quantization matrix". We did all the exercise for the ICIP challenge 2016, and the results should still be available on www.jpeg.org. If you compare a "default JPEG" with its "human oriented quantization matrix" with a PSNR-driven "default JPEG 2000", then you are correct, of course. But using JPEG 2000 this way is just folly. You need to drive the latter in a way that is optimized for human vision, not for PSNR. Equally well, you can tune JPEG towards PSNR of course, with quite some improvements in performance, though it does not become then better visually (actually, worse) and neither better than JPEG 2000.

  16. #14
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    437
    Thanks
    1
    Thanked 96 Times in 57 Posts
    Quote Originally Posted by SolidComp View Post
    Thanks for the info. This is interesting. For some reason I thought JPEG-XR was supposed to be better than JPEG 2000 – I think I saw a research paper to that effect.
    Yes, I know the paper, and that was simply due to MS not understanding what they were doing: They measured PSNR of a PSNR-optimized JPEG XR codec against PSNR of a visually optimized JPEG 2000 codec (Kakadu). So yes, results are of course "as expected" (or "as wished"), but that is certainly not a scientifically valid comparison. Again, see the ICIP challenge.
    Quote Originally Posted by SolidComp View Post
    JPEG-XR will probably die off soon since Microsoft has been so lazy with it, and no one else is exploiting its headroom, HDR and other features, etc. or building better encoders, e.g. a libjpegxr-turbo or MozJPEG-XR or a Guetzli approach.
    Well, not really. MS is using it (at least according to Rico Malvar, head of MS research) internally in their RDP products, and there is also recently an activity going on on the MPEG side to allow a royalty free codec in the HEIF container. HEVC I-frame has IP problems, so they are including JPEG 2000 and JPEG XR there.
    Quote Originally Posted by SolidComp View Post
    If JPEG 2000 is better, why don't people use it? It's always seemed like a confusing and difficult format to adopt. Just viewing JPEG 2000 images on Windows 10 is difficult. I had to do some hacking to help my girlfriend view her MRIs in some format called DICOM where some of the images were JPEG 2000. Is it patent encumbered? I remember hearing something about that.
    People *do* use it. Just not in digital photography, where it is more advantageous for the vendors to push their own proprietary codecs to "lock in" their customers to their products. This is "raw" (which is not a format). JPEG 2000 is used a lot in digital cinema and the professional market.
    Quote Originally Posted by SolidComp View Post
    Ancillary question: Do you happen to know if Safari on macOS and iOS actually displays JPEG 2000 files? It's supposed to, but I've never been able to find any mention of it in Apple's documentation, and I have no devices to test. That also makes me wonder if patent issues have prompted Apple to keep quiet about it.
    I would believe Apple products are more likely to support JPEG 2000 these days, more than any other vendor.

  17. #15
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by thorfdbg View Post
    We did all the exercise for the ICIP challenge 2016, and the results should still be available on www.jpeg.org.
    Yes, we are both correct. All depends on what constitutes 'high quality'. For me, high quality is something like JPEG YUV444 at quality 95-99. In this range it is possible to find settings that allow for no visual degradation.

    In rough terms of bpp, I can get high quality at 3.5 bpp with libjpeg JPEG YUV444, 2.5 bpp with Guetzli, and (currently) with 1.58 bpp with PIK.

    ICIP 2016 challenge showed subjective results only up to 1 bpp. For JPEG, 1 BPP corresponds to quality range ~80 and YUV420. I don't consider this high quality, quality 85 YUV444 is already what I consider 'middle' image quality where compression artefacts are universally present.
    Last edited by Jyrki Alakuijala; 10th February 2018 at 20:21.

  18. #16
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by SolidComp View Post
    Thanks for rich and thoughtful reply Jyrki. On PNG, do you have any idea how webp fares against Kornel's PNG quant?
    Kornel's PNG quant is the best-in-class tool for palettization -- great balance of features and superior quality. Unfortunately there is nothing like it for WebP, and possibly the best you can do is to apply it for a PNG and then convert that PNG to a WebP.

    It is possible that one could use Zoltan's greedy palettization and top that with butteraugli to select between the palettized and non-palettized variations for even some more savings in WebP lossless -- but no such tool exists for now.

Similar Threads

  1. WebP (lossy image compression)
    By Arkanosis in forum Data Compression
    Replies: 62
    Last Post: 12th April 2019, 18:45
  2. Replies: 7
    Last Post: 24th June 2016, 15:07
  3. WEBP - how to improve it?
    By Stephan Busch in forum Data Compression
    Replies: 38
    Last Post: 4th June 2016, 13:43
  4. WebP (Lossless April 2012)
    By caveman in forum Data Compression
    Replies: 32
    Last Post: 19th April 2013, 15:53
  5. Detector for ex-JPEG images
    By Alexander Rhatushnyak in forum Data Compression
    Replies: 22
    Last Post: 13th September 2012, 03:18

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •