Results 1 to 28 of 28

Thread: Ocarina's MPEG1 and MPEG2 video compressor

  1. #1
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts

    Ocarina's MPEG1 and MPEG2 video compressor

    ------------------------------------------------------------------------------
    OCA_MPEG 0.6 (August 5, 2010) (C) 2010, Ocarina Networks
    Written by Przemyslaw Skibinski, Portions by Matt Mahoney
    ------------------------------------------------------------------------------

    OCA_MPEG is a symmetric command-line MPEG1 and MPEG2 file compressor.
    It accepts MPEG files in MPEG-ES, MPEG-PS, and MPEG-TS containers.
    OCA_MPEG gives usually 15-25% compression at about 200 kbps.

    *** THIS IS A TEST VERSION *** MAY CONTAIN BUGS *** USE AT YOUR OWN RISK ***
    *** DO NOT REVERSE ENGINEER *** NOT FOR COMMERCIAL USE ***

    Please help us in testing. Please report bugs or send feedback to: inikep@gmail.com
    Attached Files Attached Files
    Last edited by inikep; 11th August 2010 at 10:55.

  2. #2
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by inikep View Post
    [...]
    Please help us in testing.[...]
    Quick test 1:
    Code:
    I:\Tmp\ocarina-mpeg_06>timer oca_mpeg 6 2 test 01.mpg
    Timer 9.01 : Igor Pavlov : Public domain : 2009-05-31
    Optimizing file [01.mpg] (1 MB).
    INFO: Detected MPEG-PS (Program Stream)
    INFO: MPEG-PS (Program Stream) 368x208 chroma=184x104 mcusize=6, level=6
    INFO: Chunk at 0 (1967521->1529377 bytes)
    
    Kernel Time  =     0.109 =    0%
    User Time    =    33.859 =   77%
    Process Time =    33.968 =   78%
    Global Time  =    43.486 =  100%
    
    I:\Tmp\ocarina-mpeg_06>
    Reverted:
    Code:
    I:\Tmp\ocarina-mpeg_06>timer oca_mpeg d test 01d.mpg
    Timer 9.01 : Igor Pavlov : Public domain : 2009-05-31
    Restoring file [01d.mpg] (1 MB).
    INFO: Chunk at 0 (1967521->1529377 bytes)
    INFO: MPEG-PS (Program Stream) 368x208 chroma=184x104 mcusize=6, level=6
    
    Kernel Time  =     0.250 =    0%
    User Time    =    33.718 =   84%
    Process Time =    33.968 =   84%
    Global Time  =    40.057 =  100%
    Quick test 2:
    Code:
    I:\Tmp\ocarina-mpeg_06>timer oca_mpeg 6 10 test2 02.mpeg
    Timer 9.01 : Igor Pavlov : Public domain : 2009-05-31
    Optimizing file [02.mpeg] (44 MB).
    INFO: Detected MPEG-PS (Program Stream)
    INFO: MPEG-PS (Program Stream) 320x256 chroma=160x128 mcusize=6, level=6
    INFO: Chunk at 0 (16695296->13679864 bytes)
    INFO: MPEG-PS (Program Stream) 320x256 chroma=160x128 mcusize=6, level=6
    INFO: Chunk at 16695296 (29671424->24318536 bytes)
    
    Kernel Time  =     1.187 =    0%
    User Time    =   797.453 =   80%
    Process Time =   798.640 =   80%
    Global Time  =   991.445 =  100%
    
    I:\Tmp\ocarina-mpeg_06>
    Maybe I misunderstood the phrase "chunk" but the output-file I found in the folder was just one file with size 37.998.420 Bytes.
    Reverted:
    Code:
    I:\Tmp\ocarina-mpeg_06>timer oca_mpeg d test2 02d.mpg
    Timer 9.01 : Igor Pavlov : Public domain : 2009-05-31
    Restoring file [02d.mpg] (44 MB).
    INFO: Chunk at 16695296 (29671424->24318536 bytes)
    INFO: MPEG-PS (Program Stream) 320x256 chroma=160x128 mcusize=6, level=6
    INFO: Chunk at 0 (16695296->13679864 bytes)
    INFO: MPEG-PS (Program Stream) 320x256 chroma=160x128 mcusize=6, level=6
    
    Kernel Time  =     3.109 =    0%
    User Time    =   802.000 =   87%
    Process Time =   805.109 =   87%
    Global Time  =   915.437 =  100%
    
    I:\Tmp\ocarina-mpeg_06>
    Decompressed mpegs seem to be bit-identical

    Edit: These tests were made on AMD Athlon XP 1700+ with 1,25 GB RAM running Win XP SP3

    Best regards!
    Last edited by Vacon; 12th August 2010 at 00:08.

  3. #3
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    Vacon, thanks for testing. Chunking in OCA_MPEG means that an input file can be divided into parts (if it's possible) and each part/chunk can be decompressed separately.

  4. #4
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by inikep View Post
    Vacon, thanks for testing. Chunking in OCA_MPEG means that an input file can be divided into parts (if it's possible) and each part/chunk can be decompressed separately.
    That is similar to what I expected. To be honest, I expected to find the file split to chunks in the folder. Now, if I'm not totally lost, how to decompress these chunks separetly? There is no switch I could find:
    Code:
    To compress: oca_mpeg level chunk archive file
      where:
      "level" is compression level 1..12
      "chunk" is a minimal chunk size in MB
    To decompress: oca_mpeg d archive file
    How can one use this feature? Or is it designed for multi-core systems?
    Having said that, I have to apologize for not giving my system's description for the tests I made. I will edit my last posting.

    Best regards!

  5. #5
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    Quote Originally Posted by Vacon View Post
    How can one use this feature?
    In this version it's not possible to use this feature.

  6. #6
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    2,965
    Thanks
    153
    Thanked 802 Times in 399 Posts
    Nice codec, thanks for posting it.
    I was trying to compress a 3G dvdiso file, so couldn't reply earlier :)

    Anyway, I have some questions:

    1. "oca_mpeg" doesn't seem to be able to compress mp3s (it does
    say something about detecting a ES stream though). Does Ocarina
    have a mp3 codec, what's its performance (vs soundslimmer), is
    it possible to integrate with this video codec?

    2. Did you use any parser generators in development (eg. flavor)?

    3. Does it work like paq (tracking the decoder state for each source bit) or is there
    a separate huffman decoder and a compressor?

    4. Are there intermediate DCT coefs in contexts (i mean,
    like in paq8 jpeg - partial iDCT), or maybe actual pixels?

    5. Is it possible to play the video from oca_mpeg stream,
    or mpeg stream has to be reconstructed first and decoded again?

    6. On decoding, chunks are printed in reverse order, starting from EOF.
    And it seems the file is really decoded from end to start - isn't that weird?

    Also, my results:

    3,324,870,656 1.iso
    3,102,321,624 1.iso.7z
    2,519,841,523 1.oca // oca_mpeg 12 2
    2,519,928,084 1.oca.7z
    Compression timing is lost, decompression took 22108s, restored
    file matched.
    Despite "1.oca.7z" result, there're quite a few zero runs in the
    stream - I'd say better zero run processing could be useful.
    Before finding the actual video stream, codec printed a few
    obvious misdetections (eg. 944x3088 chroma=552x24).

    Also I think that for single-file codecs the "input_file output_file"
    commandline syntax is more convenient. Anyway, "oca_mpeg" destroys
    the output file (if it exists) even before checking the input file,
    and its diagnostics can be interpreted as input file having wrong format.
    Thanks to this feature I was able to free some space on my hdd :)

    Still, even as is, this program already can be useful for dvd-video compression.
    Unfortunately, its speed is not quite reasonable for that (and I did test other "levels" too).

  7. #7
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    Shelwien, thanks for testing and sorry for your overwritten files.

    Sorry, but I'm not allowed to talk about technical details of OCA_MPEG. Therefore I will answer only the questions I think I'm allowed:

    ad 1. OCA_MPEG will compress MP3 or MPEG4 (because the containers are similar to MPEG1/2), but without any significant compression ratio.

    ad 6. You are right. It meant to be like that as it proves that chunking works well.

  8. #8
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    2,965
    Thanks
    153
    Thanked 802 Times in 399 Posts
    As to "talking about technical details", its unfortunate, but imho mostly for you
    Btw, flavor is that - http://flavor.sourceforge.net/samples.htm

    (1) I know, the question is whether you considered the possibility and your engine supports that (ie a 3rd-party mp3 codec) without rewriting it from scratch.
    (6) That's an interesting idea, but hope that you checked and it works with straight order too

    I'm going to test another file - TV capture .ts this time, currently downloading it.

  9. #9
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    Quote Originally Posted by Shelwien View Post
    the question is whether you considered the possibility and your engine supports that (ie a 3rd-party mp3 codec) without rewriting it from scratch.
    A combination of MPEG2 video and MP3 audio can be found in container formats e.g. MKV or AVI. So it mainly depends on a higher level layer, which would call MPEG2 and MP3 compressors. I don't expect any bigger problems with this.

  10. #10
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    2,965
    Thanks
    153
    Thanked 802 Times in 399 Posts
    Uh, I was talking about detection and APIs and block sizes.
    I mean, there you already have some code for stream detection and encoding of "garbage" data (not in stream or broken streams).
    So I understand about specific format parsers, but what about support for interleaved video/audio in unknown containers?
    Its just that it seemed a bit strange, but it looks like you have different types of "garbage" there - in-stream, controlled by
    that "minimal chunk size", and in-between (guessing from the difference of chunk end and start offsets).
    So I just wondered whether there won't be conflicts if you'd try to add mp3 support to your current framework.

  11. #11
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    Quote Originally Posted by Shelwien View Post
    guessing from the difference of chunk end and start offsets
    I see some misunderstanding here. If you look at this:
    INFO: Chunk at 0 (16695296->13679864 bytes)
    INFO: Chunk at 16695296 (29671424->24318536 bytes)
    The first number in parentheses is an original chunk size and the second is a compressed chunk size.
    Last edited by inikep; 12th August 2010 at 18:08.

  12. #12
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    2,965
    Thanks
    153
    Thanked 802 Times in 399 Posts
    I got that, i was talking about "Chunk at 16695296" offsets - it seemed like there were gaps when it processed that .iso file.

  13. #13
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    The gaps depend on a real chunk size (position of restart markers) in MPEG data.

  14. #14
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    2,965
    Thanks
    153
    Thanked 802 Times in 399 Posts
    The promised results with TV capture files, Q9450 @ 2.67Ghz

    345,477,120 // shiki_01_1920x1080.m2ts
    327,519,135 // 1.oca; c.time=2226.610s, d.time=2250.250s
    332,235,632 // 1.7z

    3,141,177,132 // shiki_05_1440x1088.ts
    2,163,401,843 // 2.oca; c.time=20054.016s, d.time=20701.328s
    The first file likely contains h264 video, so overall results are pretty good and useful.
    But at least 10x speed improvement seems necessary to make it practical for dvdiso/ts handling.

  15. #15
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Did quick tests, only on low quality videos because it took so long...
    Each file had just one stream, either audio or video.
    I checked correctness only in one case, again the reason is speed.
    CPU: Pentium D 2.66, dual core. Some things were running in background, everythign tested once, so timing is a bit off.

    Results:
    Code:
    I Zimbra, AC3
    #############
    
    TODO\tmp\**\Video>oca_mpeg.exe 1 256 a.oca a.mpg
    Optimizing file [a.mpg] (6 MB).
    INFO: Detected MPEG-PS (Program Stream)
    INFO: Chunk at 0 (6924288->6678518 bytes)
    6924288 -> 6678538 in 33.05 sec. 7.7161 bpc @ 3.55% @ 210 KB/s
    
    E:\TODO\tmp\**\Video>oca_mpeg.exe 12 256 a12.oca a.mpg
    Optimizing file [a.mpg] (6 MB).
    INFO: Detected MPEG-PS (Program Stream)
    INFO: Chunk at 0 (6924288->6678518 bytes)
    6924288 -> 6678538 in 33.52 sec. 7.7161 bpc @ 3.55% @ 207 KB/s
    
    E:\TODO\tmp\**\Video>oca_mpeg.exe d a.oca ad.mpg
    Restoring file [ad.mpg] (6 MB).
    INFO: Chunk at 0 (6924288->6678518 bytes)
    6678538 -> 6924288 in 34.61 sec. 7.7161 bpc @ 3.55% @ 200 KB/s
    
    O Dziewiątce, MP3
    #############
    
    E:\TODO\tmp\**\Video>oca_mpeg.exe 1 256 a.oca a.mpg
    Optimizing file [a.mpg] (4 MB).
    INFO: Detected MPEG-PS (Program Stream)
    INFO: Chunk at 0 (4755456->4615620 bytes)
    4755456 -> 4615640 in 24.42 sec. 7.7648 bpc @ 2.94% @ 195 KB/s
    
    
    Ready To Rock, MP2
    #############
    
    E:\TODO\tmp\**\Video>oca_mpeg.exe 1 256 a.oca a.mpg
    Optimizing file [a.mpg] (7 MB).
    INFO: Detected MPEG-PS (Program Stream)
    INFO: Chunk at 0 (7563264->7077647 bytes)
    7563264 -> 7077667 in 38.25 sec. 7.4864 bpc @ 6.42% @ 198 KB/s
    
    E:\TODO\tmp\**\Video>oca_mpeg.exe 12 256 a12.oca a.mpg
    Optimizing file [a.mpg] (7 MB).
    INFO: Detected MPEG-PS (Program Stream)
    INFO: Chunk at 0 (7563264->7077647 bytes)
    7563264 -> 7077667 in 37.69 sec. 7.4864 bpc @ 6.42% @ 201 KB/s
    
    I Zimbra, MPEG2
    #############
    
    E:\TODO\tmp\**\Video>oca_mpeg.exe 1 256 v.oca v.mpg
    Optimizing file [v.mpg] (146 MB).
    INFO: Detected MPEG-PS (Program Stream)
    INFO: MPEG-PS (Program Stream) 720x480 chroma=360x240 mcusize=6, level=1
    INFO: Chunk at 0 (153903104->140527605 bytes)
    153903104 -> 140527625 in 1167.91 sec. 7.3047 bpc @ 8.69% @ 132 KB/s
    
    E:\TODO\tmp\**\Video>oca_mpeg.exe 12 256 v12.oca v.mpg
    Optimizing file [v.mpg] (146 MB).
    INFO: Detected MPEG-PS (Program Stream)
    INFO: MPEG-PS (Program Stream) 720x480 chroma=360x240 mcusize=6, level=12
    INFO: Chunk at 0 (153903104->124032541 bytes)
    153903104 -> 124032561 in 2710.30 sec. 6.4473 bpc @ 19.41% @ 57 KB/s
    
    Ready To Rock, MPEG2
    #############
    
    E:\TODO\tmp\**\Video>oca_mpeg.exe 1 256 v.oca v.mpg
    Optimizing file [v.mpg] (48 MB).
    INFO: Detected MPEG-PS (Program Stream)
    INFO: MPEG-PS (Program Stream) 320x240 chroma=160x120 mcusize=6, level=1
    INFO: Chunk at 0 (50397184->43978562 bytes)
    50397184 -> 43978582 in 412.76 sec. 6.9811 bpc @ 12.74% @ 122 KB/s
    
    E:\TODO\tmp\**\Video>oca_mpeg.exe 12 256 v12.oca v.mpg
    Optimizing file [v.mpg] (48 MB).
    INFO: Detected MPEG-PS (Program Stream)
    INFO: MPEG-PS (Program Stream) 320x240 chroma=160x120 mcusize=6, level=12
    INFO: Chunk at 0 (50397184->40110814 bytes)
    50397184 -> 40110834 in 968.80 sec. 6.3672 bpc @ 20.41% @ 52 KB/s
    Observations:
    1. Strength is a bit lower than advertised, but the test set is in no way representative.
    2. Speed is much lower than expected. What generation is the mighty "Xeon 1.6"?
    3. On audio, mode 1 == mode 12.
    4. It seems to focus on video, yet it managed to squeez some from every audio track tested, nice.
    5. CPU usage was usually around 52-56%, one core at ~100% the other 0-25. I'd think too much for an IO thread, yet it doesn't look like regular multithreading.

    2 inikep:
    -You're Przemyslaw Skibinski, right?
    -What are memory requirements?
    -Any info about Ocarina's plans regarding the codec? I'm curious about the reasons why you posted it here, up to now Ocarina stood alone. With the current licensing we can't do much except for monkey testing, which you can do yourself even better. Do you want to release the codec as a standalone tool? Discuss some internals that you'll disclose later? Grab attention?
    ADDED:
    -Any Dell related comments?
    Last edited by m^2; 13th August 2010 at 22:29.

  16. #16
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    307
    Thanks
    68
    Thanked 165 Times in 63 Posts
    m^2 , thanks for testing.
    Quote Originally Posted by m^2 View Post
    1. Strength is a bit lower than advertised, but the test set is in no way representative.
    2. Speed is much lower than expected. What generation is the mighty "Xeon 1.6"?
    The speed and effectiveness are average measured on about 100 MPEG files. I don't know what kind of Xeon is it.

    Quote Originally Posted by m^2 View Post
    3. On audio, mode 1 == mode 12.
    4. It seems to focus on video, yet it managed to squeez some from every audio track tested, nice.
    Sure, it's MPEG1 and MPEG2 video compressor.

    Quote Originally Posted by m^2 View Post
    5. CPU usage was usually around 52-56%, one core at ~100% the other 0-25. I'd think too much for an IO thread, yet it doesn't look like regular multithreading.
    The current version of OCA_MPEG doesn't support multithreading.

    Quote Originally Posted by m^2 View Post
    -You're Przemyslaw Skibinski, right?
    True

    Quote Originally Posted by m^2 View Post
    -What are memory requirements?
    About 120 MB

    Quote Originally Posted by m^2 View Post
    -Any info about Ocarina's plans regarding the codec? I'm curious about the reasons why you posted it here, up to now Ocarina stood alone. With the current licensing we can't do much except for monkey testing, which you can do yourself even better. Do you want to release the codec as a standalone tool? Discuss some internals that you'll disclose later? Grab attention?
    -Any Dell related comments?
    We did everything in our power to test OCA_MPEG, but it's a compilcated compressor and external help is very valuable. I don't know Ocarina's and Dell's plans regarding the codec.

  17. #17
    Member
    Join Date
    Oct 2009
    Location
    usa
    Posts
    54
    Thanks
    1
    Thanked 8 Times in 5 Posts

    Excellent 'proof of concept'

    Congratulations on your release of this compressor! I've known something like this concept is possible for years now, but the implementation had yet to come.

    All of the .mpeg2 Es and other .MPEG streams I compressed were bit-identical on decompression. Speed was very slow, as others have noted, and this only averaged around 270 kbytes/sec on level 1 and ~ 210 on level 3, on an overclocked Core 2 Duo system (E7200 @ 3.5 GHz). I didn't try other levels because of the slow speed.

    Please consider releasing a version which has SMP multithreaded support, also unless we can get a player which supports real-time playing of the compressed streams, I think this is an incredibly interesting idea, but it will only be useful for 'proof of concept'

    There must be some more speed optimization possible.

    Thanks very much for letting us try your compressor!

  18. #18
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,254
    Thanks
    305
    Thanked 774 Times in 484 Posts
    I guess I can say a little about the MPEG compressor. Ocarina (now owned by Dell) sells a compression appliance that deduplicates and compresses files in the background. When the user wants to read or update a file, a separate reader decompresses the file. This all happens transparently, sort of like when you set Windows to compress files on disk, except that we use much better (but maybe slower) compression algorithms. For example, we can deduplicate and compress files inside of zip archives. To compensate for speed, only files that haven't been accessed for awhile are compressed.

    We also develop custom compression algorithms for single customers based on what kind of data they have. For example, we have developed custom algorithms for seismic data, DNA sequencing machine output, medical images, and some weird video formats. It's worth it when they have terabytes of data.

    The MPEG compressor parses the stream into frames which are compressed similar to JPEG. Our JPEG compressor is similar to the one in PAQ, but with some additional tuning for better speed and compression. The mode (1 through 12) sets the compression level by selecting how many models to use in the context mixing stage. The models are ordered from most to least effective. So a higher mode will run slower and there are diminishing returns with respect to compression. A mode of around 4 is probably a good compromise.

  19. #19
    Member
    Join Date
    Sep 2011
    Location
    Germany
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    The MPEG compressor parses the stream into frames which are compressed similar to JPEG.
    Does that mean, that the recompression is an intra-frame only process? Isn't there some inter-frame recompression like some kind of prediction of the motion vectors or some prediction of the pixels depending on the past frames?

    I'm asking because i'm really interested in this kind of compression (non-transform-based multimedia recompression; but i'm more or less new to data compression in general) and are kind of impressed by the results of your program and wanted to check if there is still potential of higher compression gains.

  20. #20
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    2,965
    Thanks
    153
    Thanked 802 Times in 399 Posts
    Its a recompressor similar to packjpg and such.
    You can read about that in http://mattmahoney.net/dc/dce.html#Section_6164
    So afaik it preserves the motion vectors and stuff from original format, just improves entropy coding.

    As to further compression gains, it might be possible to squeeze out another few %, but direct recompression is obviously limited.
    Its better to just design a whole new codec with modern entropy coding.

  21. #21
    Member
    Join Date
    Sep 2011
    Location
    Germany
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thanks for your answer Shelwien.
    I'm new to all this stuff, so i keep asking...

    I already took a look at Matts Data Compression Text and i think i even took a look at Matthias Stirners paper about PackJPG too.

    I do understand, that most of this recompression software is based on entropy recoding, but sometimes the "how" is different.

    First of all i want to mention, that i'm interested in this recompression stuff in terms of archival (e.g. DVD archival). This means, that the development of a new "general-purpose" codec isn't a goal, simply because of the necessary transformation from the already given lossy data to the new codec which will lose information (e.g DVDs MPEG2 -> H264) OR will lose any kind of compression because of the lossless nature in the new codec (e.g. DVDs MPEG2 -> FFV1 / Huffyuv).

    I was asking about the intra- vs. inter-frame process because of the nature of MPEG2. PackJPG and others can efficiently recompress frames, but these frames sure are only one part of the video codec (i-frames only). So the techniques of PackJPG help with spatial redundancy, but not temporal redundancy.
    So do i understand you right, that you think, that this intra-frame recompression is the only part which is working with the help of some context-models and all the other stuff is only "unpacked and repacked" with arithmetic coding instead of MPEG2s not-quite efficient VLC-tables?
    I'm sure i'm not really precise with my formulation, but can't describe it better in the moment.

    And would it be possible to use some kind of temporal prediction in this recompression? I think a few lossless video compressors (most of them seem to be intra-frame only) use motion compensation techniques internally.
    Would it help with compression if we "decode" the motion vectors for predicting the next picture in our contex-model and use this information along with the spatial one?
    The one thing which is unpredictable for myself when thinking about all these information prediction stuff, is the DCT transformation and the effects on the data. If we would ignore the DCT and Quantization, i think i'm almost trying to use some ideas from the lossless video compression area but don't know if these ideas will help.
    Another example: i don't have any idea how to compare PAQs/PackJPGs recompression and FFV1 intra-frame compression (ffmpeg-based lossless video compression codec; very efficient especially for noisy data; quite simple and intra-only; http://www1.mplayerhq.hu/~michael/ffv1.html). Maybe someone else can give draw the line between these two areas of compression.

    The other thing is about the "how" of the intra-frame recompression. To use the terms of Matts Text, PackJPG seems to be a transform-based recompressor, while the PAQ-JPEG model seems to be without a transformation step.
    I suppose that the second kind of processing does have some advantages regarding "bad-data/unknown-data" and stuff like that. If i did understand something about context-model-based compression, then a parsing-error will result in bad predictions and less compression, but the decompression should still be bitwise-identical. I'm mentioning that, because even PackJPG fails for some JPGs produced by my camera which is sold in the range of millions (meaning that the JPGs shoudn't be all bad). Maybe a transform-based approach is not really practical for large-data like video recompression? What do you think about these two (different?) approaches?

    Thanks for all your input.
    Last edited by sascha; 23rd September 2011 at 15:31. Reason: added ffv1 link

  22. #22
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    2,965
    Thanks
    153
    Thanked 802 Times in 399 Posts
    > because of the necessary transformation from the already given lossy
    > data to the new codec which will lose information

    Its not so simple actually.
    There's also an option of "pixel-perfect" compression, where you preserve
    all the original content, but discard the unnecessary metainfo such as frame headers.
    Pure lossless recompression (aka "bit-perfect") is only easier to test and demonstrate.

    For example, here're the stats for mp3 recompression:
    Code:
    4282121 barbershop.mp3
    3775712 mp3zip.exe barbershop.mp3 1 (bit perfect)
    3741518 mp3zip.exe -c barbershop.mp3 2 (sample perfect)
    
    (1-3775712/4282121)*100 = 11.82%
    (1-3741518/4282121)*100 = 12.62%
    Then, its also possible to add further quantization - a controlled loss of precision
    would be still better than "random" loss due to change of quantization method.

    > So the techniques of PackJPG help with spatial redundancy, but not temporal redundancy.

    If the motion vectors are encoded using the same huffman stuff, then obviously
    better entropy coding would also improve compression of motion vectors.

    > So do i understand you right, that you think, that this intra-frame
    > recompression is the only part which is working with the help of
    > some context-models

    Not sure why you think that context models only can be used with pixel contexts.

    Anyway, such recompression methods work by removing the original entropy coding,
    and then replacing it with something better.
    And obviously "something better" is not limited with replacing a huffman coder
    by a rangecoder, but its quite possible to build custom statistical models for
    all types of data - be it DCT coefs or motion vectors.

    > And would it be possible to use some kind of temporal prediction in
    > this recompression? I think a few lossless video compressors (most
    > of them seem to be intra-frame only) use motion compensation
    > techniques internally.

    I think the use of contextual statistics can be seen as "temporal prediction"
    in this case.
    I also think that this specific coder (oca_mpeg) just recompresses the
    mpeg structures without changing anything - as I said, it doesn't mean
    that some smart context models couldn't be applied.

    Of course there're stronger methods too. For example, instead of encoding
    some huffman table id with a best possible model, its possible to compute
    it from the other data (like original encoder does).
    Or another example: packjpg just directly encodes the DCT coefs, while
    paq8 applies 1D iDCT to already processed rows/columns and uses the results
    as context.

    Probably something similar can be applied to motion vectors as well
    (maybe fully decode the previous frame and sort the possible locations
    by available pixel context).

    However, stronger compression is usually also slower, so there might
    be no places for anything too smart in video recompression
    (unless you're ok with unpacking a dvdiso in 100 hours).

    > i don't have any idea how to compare PAQs/PackJPGs recompression and FFV1 intra-frame compression

    Well, C.Bloom did some comparisons like that a while ago in his blog.
    That's probably the only method - compress an image both ways
    (jpeg+packjpg vs i-frame coder), adjust the settings to get the same
    quality metric values, then compare the size.

    > PackJPG fails for some JPGs produced by my camera which is sold in the range of millions

    Its likely not a design problem, but simply a bug.
    Jpeg is actually a fairly complicated format, so bugs are nothing surprising.

    > What do you think about these two (different?) approaches?

    For me the main difference in approach between paq and packjpg is about these
    iDCT contexts, while whether they store the unpacked DCT coefs of the whole
    picture is not very important.

    Either way, there's no ambiguity in huffman coding, so if you managed to
    decode something, that means that you'd be able to encode it back too
    (even if its actually some random garbage)

    Also packjpg parser is probably better in case of multi-scan jpegs.
    I guess, paq technically could process these too, but that would require
    creating a custom model for each multi-scan profile, and in the end
    anyway compression would be worse.

  23. #23
    Member
    Join Date
    Sep 2011
    Location
    Germany
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thanks for all the input, especially for mentioning Mr. Blooms blog.

    I think, thats enough for now, so that i have a lot of keywords to think about and investigate modern compression techniques further.

    btw / off-topic:

    This is not the first time "mp3zip" is mentioned in this forum. But i didn't find more information about that project. Can you give me some short introduction? Maybe a really short PM because i don't want to hijack this thread with off-topic stuff.
    I know the commercial Soundslimmer project and also read the thread about the mp3 recompression attempts in this forum, where one can find a lot of code from you (dct dump + (i think) image based compression techniques). Thanks in advance.

  24. #24
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    2,965
    Thanks
    153
    Thanked 802 Times in 399 Posts
    soundslimmer=mp3zip, it creates .mpz files if you noticed.

  25. #25
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Just found a file which makes oca_mpeg to crash. The file is: Intro.wmv (Windows Media Video 7 1280x720 25fps 9017kbps)
    I uploaded the file so you can grab it from here: http://slil.ru/34495382/654299fa.518d5e40/intro.7z (Password: encode) or from http://files.fm/down.php?i=jwpiqbz&n=intro.wmv

  26. #26
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    2,965
    Thanks
    153
    Thanked 802 Times in 399 Posts
    Just forced oca_mpeg to compress mp3:
    Code:
    > ffmpeg.exe -i 1.mp3 -acodec copy 1.mpg
    > oca_mpeg.exe 12 10 1 1.mpg
    
    Optimizing file [1.mpg] (4 MB).
    INFO: Detected MPEG-PS (Program Stream)
    INFO: MPEG-PS (Program Stream) 752x560 chroma=376x280 mcusize=6, level=12
    INFO: Chunk at 0 (5117952->4999732 bytes)
    5117952 -> 4999752 in 8.73 sec. 7.8152 bpc @ 2.31% @ 586 KB/s
    
    5,058,769 1.mp3
    5,117,952 1.mpg
    4,999,752 1.oca
    I wonder what it did :)

  27. The Following User Says Thank You to Shelwien For This Useful Post:

    xinix (3rd July 2017)

  28. #27
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    318
    Thanks
    172
    Thanked 51 Times in 37 Posts
    Very interesting Shelwien...

  29. #28
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    515
    Thanks
    182
    Thanked 163 Times in 71 Posts
    Quote Originally Posted by Shelwien View Post
    Code:
    > ffmpeg.exe -i 1.mp3 -acodec copy 1.mpg
    > oca_mpeg.exe 12 10 1 1.mpg
    
    5,058,769 1.mp3
    4,999,752 1.oca
    I wonder what it did
    Interesting idea, but result isn't very impressive.

    As this is far away from the usual 10-30% gain from MP3 compression, my bet would be one of those two:
    • ffmpeg stripped some metadata, e.g. a JPG thumbnail
    • simple non-mp3 aware compression, 98% of original size isn't unusual for MP3 files


    So the following things would be interesting follow-ups:
    • standard compression with something like 7z
    • checking for metadata
    • compare to packMP3 or similar
    http://schnaader.info
    Damn kids. They're all alike.

Similar Threads

  1. Ocarina Compression Challange (Total Prize: $1 Million)
    By osmanturan in forum Data Compression
    Replies: 8
    Last Post: 2nd October 2008, 09:19

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •