Results 1 to 16 of 16

Thread: New video, image and audio codecs you might not be aware of

  1. #1
    Member
    Join Date
    Jan 2017
    Location
    uk
    Posts
    8
    Thanks
    0
    Thanked 5 Times in 1 Post

    New or upcoming video, image and audio codecs you might not be aware of

    VIDEO:

    You are probably aware of VP9 and h265 but did you know that the alliance for open media (AOM) will be ratifying the successor to VP9 called AV1 in January? It will offer ~30% improvement over VP9 and around 20% better than x265 and will be patent-free. Hardware decoders will be available ~12 months afterwards, youtube, netflix, facebook etc will be adopting it.

    AUDIO:

    You are probably aware of Opus, the patent-free audio codec that offers good or great compression from 6-512kbps. Did you know that xHE-AAC was released a while back and uses the USAC algorithm, it offers the best compression for 16-64kbps audio and support for it has been added to the DRM+ digital radio broadcast standard.
    Did you know that EVS (Enhanced voice services) is a new patented audio codec that is the best for phone calls, it was ratified 3yrs ago and support has been added in iphone8, iphone8 plus and iphone x. This works for voice-over-lte calls, wifi calls and also voip landline phones too once landline voip phones start rolling out.

    IMAGE:

    You are probably aware that WebP is a decent image codec that was released a few years ago. Did you know that some google employees are creating PIK, which offers 55% performance improvement currently and will be finalised when it reaches 65%?
    Did you know that the Jpeg Alliance has announced they are creating JPEG-XL a patent-free successor (base profile only, other profiles paid) to jpeg that will offer 60%+ better compression and will be ratified in 2020/2021?
    Did you know that the Alliance for Open Media (AOM), the creators of the upcoming AV1 video codec will be creating a new image codec called AVIF, likely using AV1 as the backend. It is likely that Google's PIK will be incorporated into AVIF like how the development version of the VP10 codec was incorporated into AV1.
    Did you know HEIF was ratified a while back and was added in ios11 ~1month ago and offers 50% better compression than jpeg, it uses h265 as the backend.
    Last edited by spaceship9876; 29th November 2017 at 23:23.

  2. The Following 5 Users Say Thank You to spaceship9876 For This Useful Post:

    boxerab (30th November 2017),Bulat Ziganshin (12th January 2018),encode (29th November 2017),hunman (30th November 2017),Jyrki Alakuijala (1st December 2017)

  3. #2
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    174
    Thanked 51 Times in 37 Posts
    Did you know that xhe-aac was released a while back and uses the USAC algorithm, it offers the best compression for 16-64kbps audio and support for it has been added to the DRM+ digital radio broadcast standard.
    What a coincidence, I've been trying to find an encoder/decoder for xHE-AAC.Does anyone have this?

  4. #3
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    48
    Thanks
    25
    Thanked 10 Times in 7 Posts
    Quote Originally Posted by comp1 View Post
    Does anyone have this?
    There are no free implementations of the xHE-AAC codec, afaik. You even have to pay for the specifiction of the format. It's heavly patented.
    xHE-AAC is not backwards compatible to HE-AAC. The arrangement of data is different. (some improvements: better SBR, better PS, better entropy coding, AMR-WB speech codec for low bitrates)
    Intended for bitrates between 16 and 64 kbit/s. No advantage or benefit for bitrates > 64 kbit/s.

    Use opus codec instead.

  5. The Following User Says Thank You to WinnieW For This Useful Post:

    pothos2 (1st December 2017)

  6. #4
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    174
    Thanked 51 Times in 37 Posts
    @Winnie - I'm only interested in extremely low bitrates, 12-24kbps, and HE-AACv2 is better than Opus at those bitrates. I am only interested in xHE-AAC. Thanks.

  7. #5
    Member
    Join Date
    May 2017
    Location
    Slovenia
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by spaceship9876 View Post

    Did you know that the Alliance for Open Media (AOM), the creators of the upcoming AV1 video codec will be creating a new image codec called AVIF, likely using AV1 as the backend. It is likely that Google's PIK will be incorporated into AVIF like how the development version of the VP10 codec was incorporated into AV1.

    Did you know HEIF was ratified a while back and was added in ios11 ~1month ago and offers 50% better compression than jpeg, it uses h265 as the backend.
    1. I'm confident AVIF will use AV1 I-Frame (like WebP uses VP8 I-Frame) and I guess in the end it will only be HEIF with AV1.

    2. HEIF is codec agnostic, h265 is used by Apple, but it also supports h264 or jpeg too, and in the future could support AV1. Generally, I have mixed feeling about the approach of HEIF, it's "just" a over-engineered container for images.

  8. #6
    Member
    Join Date
    Jan 2017
    Location
    uk
    Posts
    8
    Thanks
    0
    Thanked 5 Times in 1 Post
    Quote Originally Posted by quikee View Post
    1. I'm confident AVIF will use AV1 I-Frame (like WebP uses VP8 I-Frame) and I guess in the end it will only be HEIF with AV1.
    AVIF will likely have 65% better compression than jpeg vs 50% for heif though especially as PIK is aiming for 65% before they end development. AV1 has better compression than h265 even for no-motion video so AVIF should end up better although HEIF even if PIK isn't incorporated into AVIF. Websites are not going to adopt HEIF due to patents so AVIF will become popular on the internet, HEIF will just be used by some phones, tablets and cameras.

  9. #7
    Member
    Join Date
    Nov 2011
    Location
    france
    Posts
    38
    Thanks
    2
    Thanked 26 Times in 18 Posts
    Quote Originally Posted by quikee View Post
    1. I'm confident AVIF will use AV1 I-Frame (like WebP uses VP8 I-Frame) and I guess in the end it will only be HEIF with AV1.
    Here are some reasons why it's not as simple as that to re-use a video codec for image coding, as learnt from VP8->WebP experience.

    First there's two typical use-cases for image compression: capture/storage and delivery (esp. on low bandwidth condition).
    WebP addresses the latter, HEIF is likely addressing the former. I'll focus on the latter (efficient delivery for the web).
    For the former, manufacturers will likely use any HW available and wrap it into some container like HEIF.
    So, here are some the typical problems around using a video format for efficient image format delivery over the wire:


    • small header. By small, i mean ~20bytes. HEIF (or ISO-BMFF) are quite verbose and sub-optimal. And, coming from the video world, they often have useless information unrelated to image coding.
    • incremental decoding: image decoding needs to be easily interruptible, since it's rare to have the whole bytes available for one frame. Video codec often assumes you have a least 1 frame buffered and rather targets frame-level parallelism instead. To be easily interruptible, you need to save the decoding context often, 'just in case' you run out of bytes before the current row is finished decoding. If the context is too large (for instance, because you use adaptive probabilities), you can expect a huge overhead saving/restoring it. WebP-lossless ran into this problem with its local colormap, which can be quite large.
    • memory consumption: typical video frame resolutions are usually smaller than image (4k vs 16k e.g.). You can't really take a video codec implementation and expect it to work well 'as is' for video, despite all profiles and levels. Try loading at once ~20 <video> tags in an html page for instance!
    • Conversely, libwebp is optimized for memory consumption, and only consumes O(width) memory, not O(width * height * DPB-size). Also, to save memory, post-processing (like: film-grain or deringing) can be made in-loop instead of out-of-loop since there's only 1 frame to decode.
    • video is often tuned for YUV420, not necessarily YUV444 (or a mix thereof).
    • lossless coding is often block-based in video codecs, instead of window-based similar to LZ77. The latter proved more efficient so far for image coding (if WebP-lossless is any reference). Efficiently mixing LZ77-like lossless coding with block transforms is a challenge.
    • Alpha information needs to be interleaved with YUV information at block level, to allow row-by-row decoding and display of images. In video world, the alpha information is often compressed as a separate plane. In WebP, we had to put the ALPH chunk with alpha bytres first in order to allow proper incremental display. This meant a huge size penalty, because of this large chunk being upfront in the file layout. For capture, alpha is a non-problem, since phone cameras don't capture transparency. Alpha channel comes mostly from edition / post-prod.
    • Animation format is quite different from temporally-predicted video format. Memory consumption and random access points are a priority for animated image (this is where they are different from a short video-sequence)
    • Hardware is pretty much useless for image decoding: video hardware decoding is often serial, means you can't decode dozens of smallish images in parallel. This is often needed for web pages. HW chips don't like resolution change moreover. And speaking of HW, it's so unreliable (at least at driver-level) that you *need* to run them sandboxed, with some penalties. Secure software decoding (like libwebp) has to traverse way less software layers, running non-sandboxed.
    • Finally, IMHO, image coding can often spend more CPU at compressing better, since the number of frames is usually lower. It's hard to spend 300ms compressing each frame when you're running at 60fps!


    just my 2c
    skal/
    [*] this tweet thread is enlightening, coming from an engineer at tumblr, the typical lot-of-gifs-on-the-same-HTML-page example!

  10. The Following 2 Users Say Thank You to skal For This Useful Post:

    Bulat Ziganshin (12th January 2018),Jyrki Alakuijala (1st December 2017)

  11. #8
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by spaceship9876 View Post
    PIK is aiming for 65% before they end development.
    We are at around 65 % with our reference quality (butteraugli 1.0).

    PIK is very simple in comparison to AV1, think 30 kloc vs 300 kloc -- somewhat different design goals and philosophies. PIK is aiming at high-quality fast-decoding photography compression, and even some older solutions like VP8-based WebP lossy can still beat PIK in low-quality (low bit rate) cartoon image compression. We are thinking how to make PIK more competitive in a larger range of use cases.

    PIK is optimizing currently for the 1.5 bpp photographs, at around ~150 MB/s decompression speed. We are working hard to push down the competitive bpp rates to somewhere around 0.5 bpp without adding a lot of code, decoding slow down, encoding computation, memory requirements, etc.

  12. The Following User Says Thank You to Jyrki Alakuijala For This Useful Post:

    boxerab (1st December 2017)

  13. #9
    Member
    Join Date
    Jan 2017
    Location
    uk
    Posts
    8
    Thanks
    0
    Thanked 5 Times in 1 Post
    Quote Originally Posted by Jyrki Alakuijala View Post
    We are at around 65 % with our reference quality (butteraugli 1.0).

    PIK is very simple in comparison to AV1, think 30 kloc vs 300 kloc -- somewhat different design goals and philosophies. PIK is aiming at high-quality fast-decoding photography compression, and even some older solutions like VP8-based WebP lossy can still beat PIK in low-quality (low bit rate) cartoon image compression. We are thinking how to make PIK more competitive in a larger range of use cases.

    PIK is optimizing currently for the 1.5 bpp photographs, at around ~150 MB/s decompression speed. We are working hard to push down the competitive bpp rates to somewhere around 0.5 bpp without adding a lot of code, decoding slow down, encoding computation, memory requirements, etc.
    are you going to work with AOM to create a single image format instead of having PIK and AVIF?

  14. #10
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by spaceship9876 View Post
    are you going to work with AOM to create a single image format instead of having PIK and AVIF?
    I don't know. It can be that AV1's complexity is too much for us to handle.

  15. #11
    Member
    Join Date
    Nov 2015
    Location
    boot ROM
    Posts
    83
    Thanks
    25
    Thanked 15 Times in 13 Posts
    I gave AV1 a try few days ago and I can admit right now...
    1) Compression is slow. Damn, it is REALLY slow. Its FUCKING slow. I barely made it to encode me 1500 frames of 480p Big Buck Bunny in like 2 hours using 2-pass encoding. Somehow, aomenc tool utterly ignored all my attempts to hint "deadline". Maybe "deadline" isnt supposed to work for 2 pass or something, but that's where both vpxenc and aomenc are underdocumented or so. Actually, I've failed to get above like 40 FPM. Yea, Frames Per Minute. Not Frames Per Second at all. Somehow VP9-like "deadlines" faced some cutoff. So now fastest AOM is like slowest VP9 deadlines or so. At least in 2-pass mode. Ouch.
    2) It always puzzled me why I could specify 8 threads as long as i want to, but aomenc/vpxenc never uses more like 2-3 cpu cores anyway, even if I have 8 cores. I do not get why this tool can't be happy with just numbers of threads = number of cpus to utilize. This setting is misleading/underdocumented, too.
    3) Decompresion speed seems to be slow as well. My fairly powerful desktop barely decodes 500kbps 480p BBB in something close to real time. Seriously? I bet I had yasm and so on so it should be "optimized" version.On good side, compression efficiency is quite stunning. I've never seen anything looking so good at aggressive bitrates.

    p.s. but as far as I understand not even bitstream is finalized yet. Still in AV1 team shoes I would take care to speed it up hell a lot to improve adoption. Haven't VP10 been meant to use ANS to be faster than VP9, etc? Is it used in AOM?

  16. #12
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    this pretty much looks like they aren't YET optimized for speed. or just a reference implementation, in expectation that commercial/hardware ones will fill the optimization bill

  17. #13
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    this pretty much looks like they aren't YET optimized for speed. or just a reference implementation, in expectation that commercial/hardware ones will fill the optimization bill
    I don't know what they are planning, but sometimes such optimizations come with the expense of compression density or image quality -- or both. PIK has a slow mode that is somewhere in the same slow speed bucket with guetzli and AV1, and a fast encoding mode that is faster than libjpeg encoding (see Khavish's thread about pik fast mode for details). The fast mode gives 8 % more bytes and the maximum error in the image is 30 % more. The average error suffers less, the fast mode is just less able to have guetzli-like constant quality in the image.

    Same kind of density/quality degradation can be found in hardware jpeg encoders in photo cameras. Hardware jpeg encoders are typically 10-30 % worse than software jpeg encoders, and simple lossless jpeg re-encoding can give significant savings.
    Last edited by Jyrki Alakuijala; 12th January 2018 at 15:52.

  18. #14
    Member
    Join Date
    Nov 2015
    Location
    boot ROM
    Posts
    83
    Thanks
    25
    Thanked 15 Times in 13 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    this pretty much looks like they aren't YET optimized for speed.
    Possbile. Yet I could remember early versions were considerably faster. Sure, they lacked some advances features, but were surely beating VP9 in terms of bitrate to quality tradeoff...
    or just a reference implementation, in expectation that commercial/hardware ones will fill the optimization bill
    I doubt it. HW implementations tend to be inclined on real-time performance while keeping gate count (IC die area) low. So HW is fairly limited in tricks it could do and levels of processing it would afford. Ever seen 2-pass mode? Or hardcore motion compensation, etc? Most of time you can re-compress movie from camera to like 1/3 of its size without being able to spot differences. Just use reasonable codec settings, 2-pass mode, etc.

    Commercial implementations? I think it is unlikely AOM memebers are interested. Developing specs and codec just to pay someone else? Sounds like really weird idea. I guess they would go as far as they can. I could remember VP10 has been set to use ANS, which should be even faster than VP9's entropy coding. Have they abandoned using it or what? Or there're many other things slowing decode down? And I'm curious like a hell which commandlines google ppl are using to encode, so it does not takes forever and half, or how the hell they benchmark its coding performance, etc?

  19. #15
    Member
    Join Date
    Feb 2018
    Location
    US
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Smile

    Quote Originally Posted by spaceship9876 View Post

    You are probably aware that WebP is a decent image codec that was released a few years ago. Did you know that some google employees are creating PIK, which offers 55% performance improvement currently and will be finalised when it reaches 65%?
    Did you know that the Jpeg Alliance has announced they are creating JPEG-XL a patent-free successor (base profile only, other profiles paid) to jpeg that will offer 60%+ better compression and will be ratified in 2020/2021?
    Did you know that the Alliance for Open Media (AOM), the creators of the upcoming AV1 video codec will be creating a new image codec called AVIF, likely using AV1 as the backend. It is likely that Google's PIK will be incorporated into AVIF like how the development version of the VP10 codec was incorporated into AV1.
    Did you know HEIF was ratified a while back and was added in ios11 ~1month ago and offers 50% better compression than jpeg, it uses h265 as the backend.
    So glad to know these new image formats and I will wait for further info about them. Hope you can update it at that time. Frankly, I have already got much info about HEIF, which has been released for nearly half a year. As you said, it only takes up 50% file size than JPG. Attracting as HEIF is, it comes with limited compatibility. This is also the reason why my friends cannot open my HEIC images on their Andriod phones directly. Though there are diverse heic to jpg converter for android in market, getting local support from all operating systems is also necessary. However, we can do nothing but to take the aid of these converting tools at this moment.

  20. #16
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    What a pity DLI is not used by anybody. I guess being Open Source is the only option in some cases. Years ago I transcoded my JPG images to DLI. I think 25% of space was enough to store them, with identical quality...

Similar Threads

  1. Psychovisual analysis on modern lossy image codecs
    By khavish in forum Data Compression
    Replies: 94
    Last Post: 17th August 2018, 07:17
  2. List - Image/ Video corpus for compression
    By khavish in forum Data Compression
    Replies: 20
    Last Post: 26th November 2017, 23:02
  3. Psychovisual measurements on modern image codecs
    By khavish in forum Data Compression
    Replies: 28
    Last Post: 28th August 2017, 02:41
  4. Optimized compression codecs on Intel's Clear Linux
    By SolidComp in forum Data Compression
    Replies: 4
    Last Post: 11th March 2017, 02:58
  5. Adding new codecs to 7z.dll
    By Shelwien in forum Data Compression
    Replies: 1
    Last Post: 12th June 2016, 00:51

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •