Results 1 to 14 of 14

Thread: ICIP Grand Challenge on Image Compression - Results

  1. #1
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    437
    Thanks
    1
    Thanked 96 Times in 57 Posts

    ICIP Grand Challenge on Image Compression - Results

    Hi folks,

    ICIP was last month, and as advertised earlier on this forum, we run a challenge on lossy and lossless image compression there. Amongst the lossy candidates we had JPEG, JPEG 2000, JPEG XR, HEVC, Daala and WebP in the test plan. For lossless, we had many variants of JPEG (including hierarchical), JPEG (predictive), JPEG XR, JPEG 2000, JPEG LS, PNG and FLIF.

    By resolution of the committee, we are happy to make the ICIP presentation publicly available here:

    https://jpeg.org/downloads/aic/wg1n7..._challenge.pdf

    Some reviews:

    For lossy, HEVC is the winner, followed closely (and sometimes even left behind by) Daala, which performs very well on face images. Daala also improved from the previous test on the bike test due to smarter filtering.

    Surprisingly, "JPEG on Steroids" can even compete with these methods for higher bitrates. It is a JPEG variant with uses some advanced coding tricks similar to mozjpeg. Luckely, I got the permission from the owner to make it available on my github pages under GPLv3, I just have to do it. It will appear there in a couple of days, I will post again as soon as I uploaded it.

    WebP is not really any better than existing technology.

    JPEG 2000 was unfortunately misconfigured due to an oversight on my side - I run it with the 5/3 wavelet rather than the 9/7 wavelet. My fault. Sorry about that, so please disregard its graphs for now.

    Worst performing is JPEG. So what can we learn? It depends a lot on what precisely you do.

    For lossless, the winner is FLIF. However, the distance to JPEG-LS part-2 is not very big. FLIF is rather high-complexity, JPEG-LS 2 a low-complexity codec. I guess it's just the usual with lossless compression: If you try something simple, you get 2:1. If you try very hard, you get 2:1. Or something that is a little bit better. After all, you are mostly compressing noise.

    I want to thank all participants for submitting their software, and for the test labs for running the subjective tests.

    Greetings,

    Thomas
    Last edited by thorfdbg; 29th October 2016 at 12:41.

  2. The Following 9 Users Say Thank You to thorfdbg For This Useful Post:

    Embee (27th December 2016),encode (28th August 2017),hexagone (26th October 2016),Intrinsic (28th October 2016),JamesB (27th October 2016),load (26th October 2016),Mike (26th October 2016),schnaader (27th October 2016),SolidComp (27th October 2016)

  3. #2
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Which objective measurement correlates most with the subjective measurements (for example by spearman rank)?

    Would it be possible to include butteraugli in the objective measurements and correlate it against the subjective measurements, too?

    It seems to me that the highest quality (1-1.5 bits per pixel) subjective tests are still in the working range of butteraugli.

  4. #3
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    437
    Thanks
    1
    Thanked 96 Times in 57 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Which objective measurement correlates most with the subjective measurements (for example by spearman rank)?
    Unfortunately, I'm one of the few participants without a vision lab, so I cannot tell, and I do not have the raw scores. But if you are interested, please drop me a private mail and I'll get you in touch with Touradj. He should have this data. Actually, also from Switzerland, EPFL.

  5. #4
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    I can't access that PDF atm? 404 not found.

  6. #5
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    437
    Thanks
    1
    Thanked 96 Times in 57 Posts
    Quote Originally Posted by Intrinsic View Post
    I can't access that PDF atm? 404 not found.
    Might be because the ISO livelink system is on maintenance until Monday. So it will be back (hopefully) after the weekend. Unfortunately, I don't have a copy, I could also only pull it out of livelink - which is right now unfortunately down. Update: It seems that Tim has moved the document, so the new link is here: https://jpeg.org/downloads/aic/wg1n7..._challenge.pdf Sorry for the confusion. You also get it by first clicking on "News" on www.jpeg.org, there go to the Chengdu meeting, then scroll down until you reach the section on the grand challenge, and there you find the link.

  7. The Following User Says Thank You to thorfdbg For This Useful Post:

    Intrinsic (29th October 2016)

  8. #6
    Member
    Join Date
    Nov 2011
    Location
    france
    Posts
    38
    Thanks
    2
    Thanked 26 Times in 18 Posts
    Hi,

    Quote Originally Posted by thorfdbg View Post
    WebP is not really any better than existing technology.
    well, fwiw, i consider it a satisfying result that it consistently ranked ~3rd, including for lossless, given the constraints.

    my .02c
    skal/

  9. #7
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    645
    Thanks
    205
    Thanked 196 Times in 119 Posts
    skal, WebP was originally a single frame from VP8 - do you plan to switch to a single frame from VP10/AV1?

  10. #8
    Member
    Join Date
    Nov 2011
    Location
    france
    Posts
    38
    Thanks
    2
    Thanked 26 Times in 18 Posts
    Jarek,

    Quote Originally Posted by Jarek View Post
    skal, WebP was originally a single frame from VP8 - do you plan to switch to a single frame from VP10/AV1?
    As it is right now, from tests, VP10/AV1 could give a 20-30% size gain over WebP, depending on how you count. So, yes, it's definitely in the 'worth it' zone to have a look at next-gen compression techniques. How and when to deploy it is a separate issue.
    Re-using a video-codec as-is ( like was done for VP8 ) is asking for troubles, as you can't really fit both requirements for video and images simultaneously. Video want smaller dimension than images, but can afford more memory. Images could use larger dimension but memory is constrained and parallelism should be at pixel level, not frame level. etc. etc.
    Lastly, note that some Daala techniques are being retrofitted into AV1 already.

  11. The Following User Says Thank You to skal For This Useful Post:

    Jarek (1st November 2016)

  12. #9
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    232
    Thanks
    38
    Thanked 80 Times in 43 Posts
    Quote Originally Posted by thorfdbg View Post
    It seems that Tim has moved the document, so the new link is here: https://jpeg.org/downloads/aic/wg1n7..._challenge.pdf Sorry for the confusion.
    May I host this document on LPCB site?

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

  13. #10
    Member
    Join Date
    Oct 2015
    Location
    Belgium
    Posts
    29
    Thanks
    9
    Thanked 10 Times in 8 Posts
    Quote Originally Posted by thorfdbg View Post
    For lossless, the winner is FLIF. However, the distance to JPEG-LS part-2 is not very big. FLIF is rather high-complexity, JPEG-LS 2 a low-complexity codec. I guess it's just the usual with lossless compression: If you try something simple, you get 2:1. If you try very hard, you get 2:1. Or something that is a little bit better. After all, you are mostly compressing noise.
    That is true for photographs (or in general, for any image that is the result of noisy sensors), but not for other kinds of images which do not contain noise. E.g. rendered images, screenshots, icons, logos, plots, webcomics, pixel art, etc. Those weren't tested at the ICIP'16 challenge, but for those type of images, lossless compression can usually achieve much better results, often even producing smaller files than lossy methods. And the gap between FLIF and other lossless formats tends to be larger for such images too.

  14. #11
    Member
    Join Date
    Dec 2016
    Location
    the Netherlands
    Posts
    2
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Very interesting to see some recent measurements lined up. I'm particularly interested in the power efficiency (simple put: the time it takes) of lossy algorithms, any place to find information on that? It's not in the pdf.

  15. #12
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    437
    Thanks
    1
    Thanked 96 Times in 57 Posts
    Quote Originally Posted by Embee View Post
    Very interesting to see some recent measurements lined up. I'm particularly interested in the power efficiency (simple put: the time it takes) of lossy algorithms, any place to find information on that? It's not in the pdf.
    We didn't look into this problem at this time, and it is much more a matter of the implementation. I did something in this direction a couple of years ago in an SPIE publication where I looked at computation time over compression factor. Unsurprisingly, JPEG 2000 was much more dependent on the compression factor compared to JPEG, but for high compression rates, JPEG 2000 was actually better in terms of quality/complexity compared to JPEG. If there is interest, I can probably dig this paper out.

  16. #13
    Member
    Join Date
    Aug 2017
    Location
    Israel
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts

    HEVC as image compressor?

    Isn't HEVC a video compression codec? I'm pretty confused what is the meaning of using this codec as image compressor...
    Can you shade light abuot this?
    Thanks!

    Quote Originally Posted by thorfdbg View Post
    Hi folks,

    ICIP was last month, and as advertised earlier on this forum, we run a challenge on lossy and lossless image compression there. Amongst the lossy candidates we had JPEG, JPEG 2000, JPEG XR, HEVC, Daala and WebP in the test plan. For lossless, we had many variants of JPEG (including hierarchical), JPEG (predictive), JPEG XR, JPEG 2000, JPEG LS, PNG and FLIF.

    By resolution of the committee, we are happy to make the ICIP presentation publicly available here:

    https://jpeg.org/downloads/aic/wg1n7..._challenge.pdf

    Some reviews:

    For lossy, HEVC is the winner, followed closely (and sometimes even left behind by) Daala, which performs very well on face images. Daala also improved from the previous test on the bike test due to smarter filtering.

    Surprisingly, "JPEG on Steroids" can even compete with these methods for higher bitrates. It is a JPEG variant with uses some advanced coding tricks similar to mozjpeg. Luckely, I got the permission from the owner to make it available on my github pages under GPLv3, I just have to do it. It will appear there in a couple of days, I will post again as soon as I uploaded it.

    WebP is not really any better than existing technology.

    JPEG 2000 was unfortunately misconfigured due to an oversight on my side - I run it with the 5/3 wavelet rather than the 9/7 wavelet. My fault. Sorry about that, so please disregard its graphs for now.

    Worst performing is JPEG. So what can we learn? It depends a lot on what precisely you do.

    For lossless, the winner is FLIF. However, the distance to JPEG-LS part-2 is not very big. FLIF is rather high-complexity, JPEG-LS 2 a low-complexity codec. I guess it's just the usual with lossless compression: If you try something simple, you get 2:1. If you try very hard, you get 2:1. Or something that is a little bit better. After all, you are mostly compressing noise.

    I want to thank all participants for submitting their software, and for the test labs for running the subjective tests.

    Greetings,

    Thomas

  17. #14
    Member
    Join Date
    May 2017
    Location
    Slovenia
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Tamar View Post
    Isn't HEVC a video compression codec? I'm pretty confused what is the meaning of using this codec as image compressor...
    Should a video codecs be able to compress a video with only 1 frame? If yes - there you go... that's essentially the same as an image.

    Essentially all video codecs are image codecs with additional logic to exploit the similarities in future and/or past frames/images, however specifically made image codecs can be more efficient as the same logic can hold the image codec down (complexity vise).

Similar Threads

  1. Compression Challenge
    By Dimitri in forum Data Compression
    Replies: 21
    Last Post: 7th October 2016, 04:09
  2. Calgary Compression Challenge is closing on May 20
    By leob in forum Data Compression
    Replies: 6
    Last Post: 20th May 2016, 09:18
  3. Replies: 0
    Last Post: 3rd December 2015, 13:30
  4. Image Compression Challenge at PCS 2015
    By thorfdbg in forum Data Compression
    Replies: 0
    Last Post: 8th February 2015, 15:20
  5. Calgary compression challenge
    By Matt Mahoney in forum Data Compression
    Replies: 3
    Last Post: 25th March 2012, 12:18

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •