Results 1 to 14 of 14

Thread: unpackmp2 - a preprocessor for lossless compression programs

  1. #1
    Member
    Join Date
    Dec 2009
    Location
    Germany
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts

    unpackmp2 - a preprocessor for lossless compression programs

    unpackmp2 is a filter program that performs a lossless transformation of MPEG audio Layer II data (mp2) into an unpacked format and back into mp2 format.

    It is intended to be used as a preprocessor for lossless compression programs in order to achieve a better compression of mp2 files.

    The attached archive contains Windows executables and scripts. Also included is the full source code. It should compile under Linux and other systems as well. This is free software under GNU GPL v3, http://www.gnu.org/copyleft/gpl.html

    Code:
    test results
    ------------
    test1.mp2 (125.8 MiB, 54min 57sec, 320kbps, 48kHz, stereo, source: DVB-S radio)
    
    program            options            compressed size     unpackmp2 gain
    -----------------  -----------------  ------------------  ---------------------
    unpackmp2 | lpaq8  5                  110859854 = 84.0%   12489717 (11.9 MiB)
    unpackmp2 | 7-Zip  7z LZMA ultra      117928435 = 89.4%   8262710  (7.9 MiB)
    unpackmp2 | 7-Zip  7z Bzip2 ultra     119080485 = 90.3%   7493074  (7.1 MiB)
    unpackmp2 | 7-Zip  7z PPMd ultra      120598552 = 91.4%   4933122  (4.7 MiB)
    unpackmp2 | 7-Zip  Zip Deflate ultra  120718098 = 91.5%   6127333  (5.8 MiB)
    lpaq8              5                  123349571 = 93.5%   (n/a)
    7-Zip              7z PPMd ultra      125531674 = 95.2%   (n/a)
    7-Zip              7z LZMA ultra      126191145 = 95.7%   (n/a)
    7-Zip              7z Bzip2 ultra     126573559 = 96.0%   (n/a)
    7-Zip              Zip Deflate ultra  126845431 = 96.2%   (n/a)
    (original mp2)     (n/a)              131912640 = 100.0%  (n/a)
    unpackmp2          (n/a)              202374495 = 153.4%  -70461855 (-67.2 MiB)
    Attached Files Attached Files

  2. #2
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Nice one, thanks.
    Although it'd be better if you could include a small sample for testing.

    Also, we had a thread about similar mp3 tool before:
    http://encode.dreamhosters.com/showp...0&postcount=19

  3. #3
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Thank you smack!
    But, oo-oof, I'm still dreaming about BIK preprocessor

  4. #4
    Member
    Join Date
    Dec 2009
    Location
    Germany
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Shelwien View Post
    Although it'd be better if you could include a small sample for testing.
    Code:
    mum.mp2 (19.77 seconds, 320 kbps, 48 kHz, stereo, source: DVB-S radio)
    
    program               compressed size  
    -------------------   -----------------
    unpackmp2 | lpaq8 5   676317  = 85.5%
    lpaq8 5               764512  = 96.6%
    (original mp2)        791040  = 100.0%
    unpackmp2             1288770 = 163.0%
    Of course, you could always encode mp2 files yourself... (using TwoLAME, Win32 binaries at RareWares.org, for instance)

    Quote Originally Posted by Shelwien View Post
    Also, we had a thread about similar mp3 tool before:
    http://encode.dreamhosters.com/showp...0&postcount=19
    Yeah, I had found that one before. Do you have plans to develop it further or to open its source? I know that an "unpackmp3" program would be much more interesting than unpackmp2 for most people...
    Attached Files Attached Files

  5. #5
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > Of course, you could always encode mp2 files yourself...
    > (using TwoLAME, Win32 binaries at RareWares.org, for
    > instance)

    Yeah, but there's always risk of making a mistake with
    unfamiliar format, so imho its better to have a known sample.
    Also, even within the same format its frequently possible
    to acquire different results with different coding options.
    Actually we discarded the layer-I-II parsing code just
    because we haven't been able to find any "live" samples to test with.
    While being able to artifically produce some files doesn't mean
    that there's any sense in supporting their recompression.

    > Also, we had a thread about similar mp3 tool before:
    > http://encode.dreamhosters.com/showp...0&postcount=19

    > Yeah, I had found that one before.
    > Do you have plans to develop it further or to open its source?

    Its kinda further developed in http://soundslimmer.com

    And as to source, I kinda planned to open it when started that
    thread, but it seems like there's no demand for that
    (at least nobody asked until now), and that source requires
    some cleaning (like replacing the unreasonable built-in
    huffman tree compression with static trees .
    But I can send you a copy if you want.

    > I know that an "unpackmp3" program would be much more
    > interesting than unpackmp2 for most people...

    Who knows. AC3/DTS are required for DVD/BD recompression
    and AAC/ogg for mp4/mkv recompression, and HEAAC for stuff
    like youtube.
    So mp3 might be already becoming a history too.
    Last edited by Shelwien; 9th December 2009 at 04:05.

  6. #6
    Member
    Join Date
    Dec 2009
    Location
    Germany
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Shelwien View Post
    Actually we discarded the layer-I-II parsing code just
    because we haven't been able to find any "live" samples to test with.
    While being able to artifically produce some files doesn't mean
    that there's any sense in supporting their recompression.
    Well, you can call mp2 out-dated because it's 20 years old and a lot of more advanced codecs exist now. But it's not dead, you can find samples easily: it's used in DVD-Video (although many discs use AC3/DTS) and DVB for terrestrial and satellite TV/radio broadcast. The latter is my mp2 data source.

    Quote Originally Posted by Shelwien View Post
    But I can send you a copy if you want.
    Yes please, I would like to have a look at it.
    There is another interesting mp3 transformation tool called MP3packer. It re-arranges mp3 data with optional brute-force Huffman optimization in order to create smaller mp3 files. It's open source but written in OCaml.

    Quote Originally Posted by Shelwien View Post
    Who knows. AC3/DTS are required for DVD/BD recompression
    and AAC/ogg for mp4/mkv recompression, and HEAAC for stuff
    like youtube.
    So mp3 might be already becoming a history too.
    Is there a new SoundSlimmer in the works that will compress those formats?

    Anyway, I'm not interested in "market shares" when I think about what to program in my spare time. At the moment I'm still thinking about improving mp2 recompression. Do you have any ideas how to transform the scale factors and quantized samples to make them more compressible for LPAQ, LZMA etc?

  7. #7
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > Yes please, I would like to have a look at it.

    Sent via PM.

    > There is another interesting mp3 transformation tool
    > called MP3packer. It re-arranges mp3 data with optional
    > brute-force Huffman optimization in order to create
    > smaller mp3 files. It's open source but written in OCaml.

    Afair there was also a version written in perl.
    Well, my parser can do something similar too.

    > Is there a new SoundSlimmer in the works that will
    > compress those formats?

    Probably not soundslimmer as I don't have any relation
    to its current copyright holders.
    But I'm working on a project where such recompression
    stuff would be eventually required.

    > At the moment I'm still thinking about improving mp2
    > recompression.

    I'd suggest to think about making it completely lossless
    first. Otherwise it gets very hard to prove that modified
    version is equivalent to original, and its completely
    incompatible with stuff like torrents.

    > Do you have any ideas how to transform the scale factors
    > and quantized samples to make them more compressible for
    > LPAQ, LZMA etc?

    Forget LZMA as you can't depend on frequently appearing strings
    for compression of such data.
    Also, the data in question consists of a few 2D tables,
    so the applicable existing models would be mostly the ones used
    for image compression, like BMF or paq8 bmp model
    (in fact it was demonstrated in the mp3dump thread).

    And as to usual CM/PPM coders, these are not very good for
    this case, because they never do anything to the data which
    they use as a context. The same data is encoded and then
    used as context, and only sequentially.
    So while I can't say that reaching better compression by
    preprocessing is impossible, but to acquire good results
    you'd have to use some kind of statistical ranking model,
    so its imho much more reasonable to just write a specialized
    coder right away.

    In short, you can't depend on preprocessing too much -
    btw that's why paq has different submodels for different
    data types.

  8. #8
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    Well, I can suggest something (though maybe you already do it).
    1) It should be better to store the coefs by columns instead of rows.
    2) Element width matters a lot - its usually better to use a bytewise
    code, but it might be different if you're willing to use a wav model or something.
    3) Starting with simple delta (storing coef differences instead of original coefs),
    its usually possible to gain something by predicting the next value and
    subtracting the prediction before coding.

  9. #9
    Member
    Join Date
    Dec 2009
    Location
    California
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Shelwien View Post
    Well, I can suggest something (though maybe you already do it).
    1) It should be better to store the coefs by columns instead of rows.
    Not columns - zigzag, as they're natively stored in most formats. Don't know about MP2. Statistical analysis can show you the optimal zigzag; JPEG uses a very basic diagonal one, whereas H.264's jumps all over the place. As you evolve your prediction model you'll probably have to run the analysis a few more times. There might be an easy way to signal an End Of Block, for low bitrates, I don't know enough about MPEG-1-L-II to say. Either give each block a length or define a magic byte.

    Quote Originally Posted by Shelwien View Post
    3) Starting with simple delta (storing coef differences instead of original coefs),
    its usually possible to gain something by predicting the next value and
    subtracting the prediction before coding.
    It's much better to extend this inter-frame. Start by storing every DC component from the entire block next to each other. You should be able to predict better than simple delta by using lossless audio techniques: Predict it by fitting a curve to the last x samples. Then store the AC components progressively, each band grouped together. Now you can try to predict it from the preceding coeffs, I suppose, but I'm sure you'll need to go through a lot of models before you find a good predictor. Near the end some will be shorter because the blocks were terminated early in the first step; generally this seems to save more than leaving the zeros in.

    A context mixer might not be a bad choice at this stage. It might even point to further exploitable patterns. The packjpg paper and the h.264 standards both have a lot of very good information on treating dct-quantized data, they're worth an overview. (I'm far more familiar with those formats and what works than MP2.)

    Note that this method prevents any kind of real-time playback. The larger the blocks, the more data has to be decompressed before playback.

  10. #10
    Member
    Join Date
    Dec 2009
    Location
    Germany
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Shelwien View Post
    I'd suggest to think about making it completely lossless
    first. Otherwise it gets very hard to prove that modified
    version is equivalent to original, and its completely
    incompatible with stuff like torrents.
    It's lossless when you compare the decoded audio data of the original and the unpacked/repacked mp2 files, using foobar2000's "Bit-compare tracks" feature or by simply "fc"-ing the decoded wav's.

    The program currently skips all non-audio data in the mp2 files - that's a feature not a bug. But I recognise your point that for most uses it would be better to preserve the original mp2 with all its non-audio data, such as ID3 tags at head or tail of file and the "filler" data inside of the mp2 frames. I'm going to add that to the next version.

    Quote Originally Posted by Shelwien
    1) It should be better to store the coefs by columns instead of rows.
    I think unpackmp2 already does this: it orders the scale factors and samples by subband and groups many consecutive frames in order to create longer runs of similar bytes. This is a kind of transforming the "2D tables" of data into "1D streams", isn't it?

    Quote Originally Posted by Shelwien
    2) Element width matters a lot - its usually better to use a bytewise
    code, but it might be different if you're willing to use a wav model or something.
    Yes, I noticed this at the very beginning of my experiments. Unpackmp2 "inflates" nearly all data elements of the mp2 stream to full bytes. This is the reason why the unpacked files are so much larger than the mp2 files. It seems to help compressors quite a lot.

    Quote Originally Posted by Shelwien
    3) Starting with simple delta (storing coef differences instead of original coefs),
    its usually possible to gain something by predicting the next value and
    subtracting the prediction before coding.
    I had done some tests with simple prediction recently and wasn't successful with that, but probably that test code had some bugs or something. Will explore this further.

    Quote Originally Posted by Shelwien
    So while I can't say that reaching better compression by
    preprocessing is impossible, but to acquire good results
    you'd have to use some kind of statistical ranking model,
    so its imho much more reasonable to just write a specialized
    coder right away.
    In fact, I had thought about this quite a lot. My motivation to create unpackmp2 in the first place is to use it for myself and to learn something while writing, testing and improving the program. I can't spend a lot of time on this (usually just a few hours during one or two nights per week) and therefore opted for the most simple and straightforward approach, which in my opinion is a small stand-alone preprocessor program. The actual compression part is slightly out of my scope at the moment.

    I really appreciate the input I get from you and others here. This the main reason why I posted the program in this forum.

  11. #11
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > But I recognise your point that for most uses it would be
    > better to preserve the original mp2 with all its non-audio
    > data, such as ID3 tags at head or tail of file and the
    > "filler" data inside of the mp2 frames. I'm going to add
    > that to the next version.

    Unfortunately that's not all.
    A practical recompressor also has to be able to losslessly
    transform _any_ data, not only valid data in its format.
    Otherwise even a mostly valid file, with a transmission
    error in one byte, won't be losslessly recompressed
    (or might even cause a crash).
    And btw, one good reason for lossless recompression
    is container format integrity. For example, suppose
    we used another preprocessor to extract a mp2 audio
    track from some movie, and that preprocessor is also
    able to losslessly reconstruct the video container,
    but only if its output is not modified.

    >> It should be better to store the coefs by columns
    > I think unpackmp2 already does this: it orders the scale
    > factors and samples by subband and groups many consecutive
    > frames in order to create longer runs of similar bytes.
    > This is a kind of transforming the "2D tables" of data
    > into "1D streams", isn't it?

    Sure, but I have doubts about such subband serializing,
    as well as zigzags suggested in the previous post.
    Supposing that
    ...aaaabbbbbcccccc...
    ...AAAABBBBBCCCCCC...
    turns into
    ...aaaaAAAAbbbbbBBBBBccccccCCCCCC...
    (if that's what you meant)
    the compressor would try to encode
    "A" in context of last "a" coef in subband.
    Or we can use zigzag here, but that
    would only preserve the context for one
    coef in subband.
    Anyway, I tried various permutations and same
    coef from previous row seemed like the best context.
    Of course, my model uses other coefs as well,
    but imho that can't be simulated with preprocessing+
    compression with prefix contexts.

    > Unpackmp2 "inflates" nearly all data elements of the mp2
    > stream to full bytes. [...] It seems to help compressors
    > quite a lot.

    Then there's also an idea to separate the alphabets for
    different data types, so their statistics won't mix.
    For example, if we need to preprocess 32-bit values for
    compression with a bytewise compressor, it can work
    like this:
    12 34 56 78 -> 71 62 53 44 35 26 17 08
    This commonly increases compression despite apparent
    redundancy, because otherwise the compressor would still
    predict higher probability of 12 in 2nd byte of the
    number, even if its a long table of numbers which all look
    like 1234xxxx.

    > The actual compression part is slightly out of my scope at
    > the moment.

    Actually context model + arithmetic coding may be simpler
    than many preprocessing tricks, while providing better
    compression.
    I can patch up a simple demo for you, if you want.
    Last edited by Shelwien; 12th December 2009 at 01:22.

  12. #12
    Member
    Join Date
    Dec 2009
    Location
    California
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Shelwien View Post
    Unfortunately that's not all.
    A practical recompressor also has to be able to losslessly
    transform _any_ data, not only valid data in its format.
    Otherwise even a mostly valid file, with a transmission
    error in one byte, won't be losslessly recompressed
    (or might even cause a crash).
    And btw, one good reason for lossless recompression
    is container format integrity. For example, suppose
    we used another preprocessor to extract a mp2 audio
    track from some movie, and that preprocessor is also
    able to losslessly reconstruct the video container,
    but only if its output is not modified.
    I disagree. If the stream's broken, it's not generally going to be recompressible without breaking it, in the same way it's no longer correctly decodable. The only thing you could possibly do in that situation is have a marker that would indicate an uncompressed block coming up, then attempt to scan for the next sync point and start again. (Fortunately, MPEG standards have a lot of resiliency built in, even in raw streams.)

    The recompressor shouldn't concern itself with meta information, sync markers, and encapsulation. If that's desired, and I agree that it is, implement it in another layer. By giving up some of the compression, you could actually stick the raw data back into MPEG frames, but it would make more sense to parse the frames and put them elsewhere (beginning or end, perhaps).

    Quote Originally Posted by Shelwien View Post
    Actually context model + arithmetic coding may be simpler
    than many preprocessing tricks, while providing better
    compression.
    I can patch up a simple demo for you, if you want.
    Agreed, I don't think there's ever going to be a good way to compress AC components better than arithmetic. They are essentially somewhat random components no matter how you cut and slice them, though DC components are strongly correlated.

  13. #13
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,134
    Thanks
    179
    Thanked 921 Times in 469 Posts
    > I disagree. If the stream's broken, it's not generally
    > going to be recompressible without breaking it, in the
    > same way it's no longer correctly decodable. The only
    > thing you could possibly do in that situation is have a
    > marker that would indicate an uncompressed block coming
    > up, then attempt to scan for the next sync point and start
    > again. (Fortunately, MPEG standards have a lot of
    > resiliency built in, even in raw streams.)

    As I said, there're use cases where such preprocessor
    is applied along with other filters - like precomp or
    avi parser. And in such a case data transformations not being
    completely lossless won't allow to reconstruct the
    original data at all.
    Of course, some patching framework can be added to
    ensure that data is always losslessy reconstructed,
    but that would be 2-3x slower (forward pass, backward pass,
    patch generation), and not streamable.

    Anyway, as an example, we have a mp3 preprocessor here,
    which _does_ it like I described - for any broken or
    unknown data.
    http://encode.dreamhosters.com/showp...0&postcount=19

    > The recompressor shouldn't concern itself with meta
    > information, sync markers, and encapsulation. If that's
    > desired, and I agree that it is, implement it in another
    > layer.

    Layers or anything... it would become much slower and awkward
    if you'd implement it as a multipass algorithm, and its much
    more natural to include the necessary validation into the
    parser code.

  14. #14
    Member
    Join Date
    Dec 2009
    Location
    Germany
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    v1.2 (2010-04-02)
    preserve non-audio data before first and after last mp2 frame (e.g. ID3 tags).
    preserve non-audio data inside of mp2 frames.
    (format of unpacked files is not compatible with previous version!)


    Previous versions of the program unpacked the audio data only, while dropping all non-audio stuff from the mp2 files. That transformation could be called "lossless" only when the decoded audio content of the original and the transformed mp2 files was compared.

    This new version of the program also stores non-audio data from the mp2 files. This means that original and transformed mp2 files are identical bit for bit (except for bugs...), which can be verified using tools like fc or diff.

    There is no improvement in compression this time. Some tests were done to re-order the unpacked data or to employ linear prediction, but to no avail, LPAQ8 or LZMA compression could not be improved this way.
    Attached Files Attached Files

Similar Threads

  1. Comparison of lossless PNG compression tools
    By Surfer in forum Data Compression
    Replies: 54
    Last Post: 19th September 2011, 22:58
  2. GraLIC - new lossless image compressor
    By Alexander Rhatushnyak in forum Data Compression
    Replies: 17
    Last Post: 29th November 2010, 21:27
  3. BMF is not binary lossless NOR pictore lossy
    By SvenBent in forum Data Compression
    Replies: 4
    Last Post: 23rd August 2009, 12:54
  4. Lossless Audio Codec
    By encode in forum Forum Archive
    Replies: 8
    Last Post: 1st August 2007, 18:36
  5. TTA - very promising lossless WAV packer
    By Bulat Ziganshin in forum Forum Archive
    Replies: 12
    Last Post: 27th March 2007, 13:12

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •