Page 1 of 2 12 LastLast
Results 1 to 30 of 35

Thread: exists an recompressor for h264 streams and audio streams?

  1. #1
    Member
    Join Date
    Jun 2008
    Location
    G
    Posts
    372
    Thanks
    26
    Thanked 22 Times in 15 Posts

    exists an recompressor for h264 streams and audio streams?

    Hi,

    exists an lossless recompressor for an h264 + acc/mp3 stream included in an mp4/mkv container?

    if not will it be possible to create one which recompressed it to 70% of orginalsize with ca. 10mb/s on of i7 with 4 cores? Or is this mostlikely not possible?


    best regards

    TT

  2. #2
    Member zody's Avatar
    Join Date
    Aug 2009
    Location
    Germany
    Posts
    90
    Thanks
    0
    Thanked 1 Time in 1 Post
    h264 and aac/mp3 are already lossy compressed formats. It is usually impossible to compress them further without loosing quality (creating a new stream with lower bitrate)

  3. #3
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    I doubt something like this exists, because the closest thing would be Ocarina's (now Dell's) MPEG recompressor (http://encode.ru/threads/1111-Ocarin...deo-compressor) - this was two years ago and I haven't heard of something similar meanwhile, so it's pretty much state of the art.

    Is it possible? I think so, but you should split "70% of size" and "10 mb/s" - 70% is possible for mp3 and MPEG-1/2, but h264 has better compression, so I'd expect worse ratios for it. Dell's MPEG recompressor states "15-25% compression at about 200 kbps", so speed is much worse than 10 mb/s, even if running at 4 cores simultaneously.

    Note that creating such a recompressor is complicated, much work and lossy compression is much easier and common for video data, so doing this is more a proof of concept - you'll be able to lossy convert most video files to 50% of their size with existing tools and without much quality loss - and because of YouTube and similar streaming services, people tend to favor small size instead of high quality nowadays, at least for web videos, so most won't care about losslessness. On the other end of the spectrum, there are home cinema users that want very high quality, 6 or more channel sound and won't really care about the video being 4 or 8 GB in size. And, although it's getting more uncommon, there are still cases in which you'll have a fixed medium size (like 25 GB for a bluray disc) and need compatible, fast decompression with good quality instead of a filesize as low as possible.

    Creating a new video/audio codec that gives 50% of h264/aac size with similar quality is another option. Recompression of existing videos is nice, but avoiding large new video files is closer to the problem's roots. CPU/GPU speed is evolving quicker than most of the bottlenecks (bandwidth, disk space), so codecs should evolve with it.
    Last edited by schnaader; 22nd January 2012 at 19:14.
    http://schnaader.info
    Damn kids. They're all alike.

  4. #4
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    Quote Originally Posted by thometal View Post
    if not will it be possible to create one which recompressed it to 70% of orginalsize with ca. 10mb/s on of i7 with 4 cores? Or is this mostlikely not possible?
    if it would be possible, it would be a part of h264 itself

  5. #5
    Member
    Join Date
    Jun 2008
    Location
    G
    Posts
    372
    Thanks
    26
    Thanked 22 Times in 15 Posts
    hm but the recompressed data stream dont need to be searchable etc. so maybe it can compress redundancies which is not possible for a video codec?

  6. #6
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Bulat, h264 is a 2003 codec. Its successor is being prepared for quite some time already. Which doesn't change the fact that theometal's dreams are as beautiful as unrealistic.

  7. #7
    Member
    Join Date
    Oct 2010
    Location
    Germany
    Posts
    275
    Thanks
    6
    Thanked 23 Times in 16 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    if it would be possible, it would be a part of h264 itself
    There are much better image compression systems than H.264, but they are computionally expensive which rule them out to be used in a video-codec.

  8. #8
    Member
    Join Date
    Jun 2008
    Location
    G
    Posts
    372
    Thanks
    26
    Thanked 22 Times in 15 Posts
    Other question how much is the data stream compressed by calvc or cabac against if i would use neither? I know it would not be a compartible h264 stream. Exists any tools which can produce such a stream/file?

  9. #9
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    even h264 itself can't handle 10mb/s on modern cpus, so slower codecs are out of question

  10. #10
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    48
    Thanks
    25
    Thanked 10 Times in 7 Posts
    Hello there!

    I have a question. Is there a program that can compress AAC in the way packmp3 compress MP3? packAAC?

    Remove Huffman compression from AAC files and replace it with some better compression?

  11. #11
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    There kinda is: http://nishi.dreamhosters.com/u/aaczip_v0.rar
    But its very alpha and only supports AAC-LC.

  12. The Following 4 Users Say Thank You to Shelwien For This Useful Post:

    Gonzalo (10th January 2017),RamiroCruzo (9th January 2017),WinnieW (9th January 2017),xinix (9th January 2017)

  13. #12
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    48
    Thanks
    25
    Thanked 10 Times in 7 Posts
    Thank you for your help, Shelwien.
    Unfortunaly I can't download the file. The antivirus program on my PC blocks the file, claims there is malware inside that archive.
    Last edited by WinnieW; 9th January 2017 at 01:49. Reason: typing error

  14. #13
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Sorry, its not my problem - looks like somebody reported /u to Dr.Web, or something.
    I didn't find a way to check what it doesn't like on my site, or report false positive.

  15. The Following User Says Thank You to Shelwien For This Useful Post:

    xinix (9th January 2017)

  16. #14
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    48
    Thanks
    25
    Thanked 10 Times in 7 Posts
    I know you are not responsible for it an I can't blame you.
    The question is is it a false positive or real malware? Who can tell?

  17. #15
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Well, I guess IntelC's anti-AMD check can be considered a malware.
    Otherwise I compiled that .exe just before posting.
    In any case, you can always download the archive, unpack it, and _then_ check it with AV.

  18. #16
    Member
    Join Date
    Nov 2015
    Location
    France
    Posts
    7
    Thanks
    2
    Thanked 0 Times in 0 Posts
    AAC uses Huffman-like coding, and probably not a very good one:
    - The example file, 1.aac, compresses from 1552687 bytes to 1424502 (zip) and 1078431 (swp? don't know how to decode)
    - Another file, ct_faac-adts.aac, goes from 299647 to 290817 or 265753

    Much like avrecode or Pied Piper or some of the JPEG recompressors, they probably work mostly because they know what elements are VLC and they move them to arithmetic coding. Selecting/defining the proper context is of course a win, and as H.264/AAC defined theirs 15+ years ago, there were likely easy improvements since.

  19. #17
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > swp? don't know how to decode

    See test1.bat, aaczip also supports decoding.

    > they probably work mostly because they know what elements are VLC and they move them to arithmetic coding

    I am the developer actually, though its a discarded project from 2005.
    Its an aac dumper/restorer made from faad + mp3zip (aka soundslimmer) model

    > there were likely easy improvements since.

    video/audio format developers just usually optimize for different things.
    Mainly they just want an efficient hardware implementation and don't care much if there's 5% redundancy.

  20. The Following User Says Thank You to Shelwien For This Useful Post:

    WinnieW (9th January 2017)

  21. #18
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    48
    Thanks
    25
    Thanked 10 Times in 7 Posts
    Quote Originally Posted by Shelwien View Post
    Otherwise I compiled that .exe just before posting.
    You wrote and compiled the program aaczip? I did not know that.

    Quote Originally Posted by Shelwien View Post
    In any case, you can always download the archive, unpack it, and _then_ check it with AV.
    I'll try it. Thank you.

  22. #19
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    48
    Thanks
    25
    Thanked 10 Times in 7 Posts
    Quote Originally Posted by Shelwien View Post
    I am the developer actually, though its a discarded project from 2005.
    Its an aac dumper/restorer made from faad + mp3zip (aka soundslimmer) model
    A buried treasure of program code, released to the public.

  23. #20
    Member
    Join Date
    Nov 2015
    Location
    France
    Posts
    7
    Thanks
    2
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Shelwien View Post
    > swp? don't know how to decode

    See test1.bat, aaczip also supports decoding.

    > they probably work mostly because they know what elements are VLC and they move them to arithmetic coding

    I am the developer actually, though its a discarded project from 2005.
    Its an aac dumper/restorer made from faad + mp3zip (aka soundslimmer) model
    Ah OK. I don't well know batch syntax, and just tried aaczip -d/-dmv 1.swp

    I imagine then you dump the raw (mdct) coefficients and get them through an improved entropy coder.

  24. #21
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > I imagine then you dump the raw (mdct) coefficients and get them through an improved entropy coder.

    Kinda, its something similar to this - http://encode.ru/threads/271-mp3dump
    But there's also a fairly advanced context model, not just entropy coder.
    Its not _that_ easy to write a recompressor - just try compressing mp3dump output with paq8 or something.

  25. #22
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    Any chance of release the code? Please?

  26. #23
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts

  27. The Following 3 Users Say Thank You to Shelwien For This Useful Post:

    Bulat Ziganshin (10th January 2017),Gonzalo (10th January 2017),xinix (10th January 2017)

  28. #24
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    48
    Thanks
    25
    Thanked 10 Times in 7 Posts
    Quote Originally Posted by Shelwien View Post
    No chance, because of NDA.
    Ok. Can you describe the processing steps aaczip performs in words? I guess that should not break NDA.

  29. #25
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    You can look at the mp3dump thread, aaczip does the same for aac format.
    As to processing steps...
    1. Remove original entropy code and convert a frame into a fixed-size static data structure
    2. Use a good modern context model + entropy coder to compress/decompress the structures
    3. Convert decompressed structures to original aac entropy coding and bitfields

  30. The Following 3 Users Say Thank You to Shelwien For This Useful Post:

    RamiroCruzo (10th January 2017),WinnieW (10th January 2017),xinix (10th January 2017)

  31. #26
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    48
    Thanks
    25
    Thanked 10 Times in 7 Posts
    I can confirm my antivirus software reported your program as a false positive. It contains no malware.

    A few days ago I tried to compress a few randomly selcted aac files from my music archive using 7zip, both LZMA and PPMd, and I got a result of 98% of original file size on average. PPMd achieved a little better result.
    The example file – harpsichord solo – is better compressible than aac files usualy are.

  32. #27
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Yes, its just a "feel good" sample - the average aaczip gain with normal files encoded by faac was ~15% afair.

  33. #28
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    48
    Thanks
    25
    Thanked 10 Times in 7 Posts
    Unfortualy to my own experience your statement the programm is alpha is true.
    I got one file mismatch decodeced file vs. original one and aaczip crashed at start of decoding a .swp file.

  34. #29
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Well, even for packjpg dropbox had to implement verifying and anti-crash / anti-exploit wrappers, same applies here i guess.

  35. #30
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    48
    Thanks
    25
    Thanked 10 Times in 7 Posts
    Are you willing to release a simple aacdumper? I'd write one for myself but there are more than 10 years gone the last time I wrote a piece of software, the time I attended university for applied science.
    I guess my programming skills now are even worse than my english.

    I had two ideas in my mind to test:
    1. replacing identical frame headers in aac files with some shorter symbol (What about aac files that contains CRC in every frame?)
    2. testing if there is a benefit to compression by sorting the MDCT coefficients with e.g. BWT

Page 1 of 2 12 LastLast

Similar Threads

  1. reflate - a new universal deflate recompressor
    By Shelwien in forum Data Compression
    Replies: 119
    Last Post: 28th March 2018, 22:52
  2. Audio Compression
    By osmanturan in forum Data Compression
    Replies: 129
    Last Post: 2nd November 2017, 12:51
  3. lzma recompressor
    By Shelwien in forum Data Compression
    Replies: 33
    Last Post: 25th February 2016, 23:40
  4. Concealing data in deflate streams
    By Shelwien in forum Data Compression
    Replies: 4
    Last Post: 10th January 2013, 16:02
  5. Wondering whether to release my deflate recompressor
    By m^2 in forum Data Compression
    Replies: 15
    Last Post: 4th November 2011, 19:31

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •