Results 1 to 24 of 24

Thread: dump uncompressed PNG residuals?

  1. #1
    Member
    Join Date
    Oct 2013
    Location
    Filling a much-needed gap in the literature
    Posts
    350
    Thanks
    177
    Thanked 49 Times in 35 Posts

    dump uncompressed PNG residuals?

    Is there an easy way to dump the residuals from a PNG file, to try compressing them with various algorithms?

    Right now I'm just using pnmtopng with -compression 0 to generate an uncompressed png file, and running the whole file through various compressors, but I'd like to be able to use png optimizers and recover the best residuals.

    (I'd also be interested in newer/better front ends to try this with, if it makes sense. and isn't difficult.)

  2. #2
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by Paul W. View Post
    Is there an easy way to dump the residuals from a PNG file, to try compressing them with various algorithms?

    Right now I'm just using pnmtopng with -compression 0 to generate an uncompressed png file, and running the whole file through various compressors, but I'd like to be able to use png optimizers and recover the best residuals.

    (I'd also be interested in newer/better front ends to try this with, if it makes sense. and isn't difficult.)
    Why PNG? If you make something new, you can use the webp lossless filtering layer. The paeth equivalent there I designed with one branch vs. 3-4 for truecolor png, and it compresses more, too.

    One possibility would be to send an uncompressed PNG and use brotli for content encoding, that would be better than PNG but still much worse than webp lossless.

    Also, if you want to reuse the residuals in png you should also know the filter bytes that the optimiser was using.

  3. #3
    Member
    Join Date
    Oct 2013
    Location
    Filling a much-needed gap in the literature
    Posts
    350
    Thanks
    177
    Thanked 49 Times in 35 Posts
    Jyrki,

    I'm not married to PNG, and am interested in WebP Lossless, too, especially if it's easy to dump residuals in some easy-to-process, relatively compressor-neutral form.

    Right now I'm just doing some very basic experiments, largely to get some idea how GLZA rates relative to PPMD, LZMA, and gzip for this sort of thing, and what would improve it.

    (So far, really not bad on uncompressed PNGs. Not as good as PPMD, but distinctly better than LZMA, and of course lots better than gzip. I have no idea how it would compare to your 2D LZ thing.)

    Is there a way to generate an uncompressed WebP file? (I looked at the cwebp man page and didn't see one. Is there a utility that does things like that?)

  4. #4
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by Paul W. View Post
    Is there a way to generate an uncompressed WebP file? (I looked at the cwebp man page and didn't see one. Is there a utility that does things like that?)
    You can store the filtered image only by modifying the code in libwebp.

    https://groups.google.com/a/webmproj...ss/ic00NLoJH24

    Last summer Mislav Bradac studied the use of different LZ77 engines after WebP filtering. His results showed that for a 1000 (at least 2 % surface area transparent) images from the internet WebP filtering works more efficiently.

    https://docs.google.com/document/d/1...J3uyLizQ/edit#

    Pngout (with png filtering) compressed this corpus to 25942082 bytes. WebP filtering with Zopfli gives 22506013 bytes (-13.2 %), xz -9 gives 20852520 bytes (-19.6 %), brotli 21601337 (-16.7 %), and WebP LZ77 gives 19169032 (-26.1 %). We tried hacking brotli's context modeling to fit this problem better, that gave about 4 % improvement with relatively simple changes. Color channel separation (after all WebP filtering) gives 19702002 bytes with the vanilla brotli.

  5. The Following User Says Thank You to Jyrki Alakuijala For This Useful Post:

    Paul W. (24th June 2016)

  6. #5
    Member
    Join Date
    Oct 2013
    Location
    Filling a much-needed gap in the literature
    Posts
    350
    Thanks
    177
    Thanked 49 Times in 35 Posts
    Thanks, Jyrki. That's very interesting.

    It looks like unmodified Brotli compresses uncompressed WebP files within about 2.8 percent of normal WebP, if the color planes are separated, and is about 5.1 percent better than LZMA. (And normal WebP is about 8 percent better than uncompressed WebP + LZMA.)

    For uncompressed PNGs of 24-bit RGB photographs (photos only, and not crushed) from Kodak and Waterloo, we're seeing GLZA beat LZMA by about 8 percent. But cwebp does *much* better, I'd guess mostly because the front end filtering is so much better. And it's very much better on the "artistic" (digital painting) images from Waterloo. I'm guessing that's largely due to the color cache, but don't know.

    (BTW in some very simple experiments with very simple filters we also found separating the color planes to be crucial for good compression with gzip/LZMA/GLZA.)

    It would be interesting to see results for WebP filtering + LZMA/Brotli on the standard RGB photo sets... did you try that?

  7. #6
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by Paul W. View Post
    I'm guessing that's largely due to the color cache, but don't know.
    Yes, color cache works in an amazing way for those very rare images that have a small palette of colors used locally. If I remember correctly, there is one such image in the waterloo set where this works very well (like -10 % in size).

    Quote Originally Posted by Paul W. View Post
    (BTW in some very simple experiments with very simple filters we also found separating the color planes to be crucial for good compression with gzip/LZMA/GLZA.)
    WebP lossless solves this by having different entropy codes for R, G, B, A and distances. Lengths (and the color cache) are combined with G, the same way as deflate combines lengths with literals.

    Quote Originally Posted by Paul W. View Post
    It would be interesting to see results for WebP filtering + LZMA/Brotli on the standard RGB photo sets... did you try that?
    I think we tried that with 2-3 images only, and results were to the same direction as for the 1000 png test. WebP lossless spatial filtering is not ideal for photographs. LZ77 rarely works with photos -- this is why the encoder tries an advanced rle, too. The cross-color transform can turn out very useful for low noise photos. WebP lossless is rather sensitive for noise in the image because of the small spatial prediction filter size (4 closest pixels). During the initial development I had larger prediction filters that helped a lot with the most grainy/noisy photos, but in later simplification steps I threw them out for better decoding speed.

    If you want to target lossless RGB photo compression, you'd probably benefit for adding a larger averaging (or adaptive) filtering into the format.

  8. The Following User Says Thank You to Jyrki Alakuijala For This Useful Post:

    Paul W. (25th June 2016)

  9. #7
    Member
    Join Date
    Oct 2013
    Location
    Filling a much-needed gap in the literature
    Posts
    350
    Thanks
    177
    Thanked 49 Times in 35 Posts
    Jyrki, can you explain your predictors and why SELECT works better than Paeth?

    As i understand it, it it's similar to Paeth in that it's first doing a local 2D gradient estimation based on the pixels immediately to the NorthWest, North, and West, where the horizontal gradient Gh is N - NW and the vertical gradient Gv is W - NW. Those deltas are added to the NW value to estimate the current value P.

    At that point, Paeth then picks among the observed values of NW, N, and W, rather than actually using P, but your SELECT only chooses between N and W. Why? One case where it's better is where you've encountered a diagonal edge running NE to SW, but there are lots of cases and I'm not sure what motivates this.

    I would guess that it's partly because if you really think it's just a gradient, you'll tend to be using an entirely different predictor for that image or tile, maybe ClampAddSubtractFull.

    As I understand it, ClampAddSubtractFull is just the 2D gradient prediction, but clamping negative values at 0 and positive values beyond 255 at 255, so that you don't predict that gradients continue past black/white and wrap around into white/black. That won't be right if your blacks aren't quite black or your whites aren't quite white, but it will be less wrong.

    (I guess a generalization of that would be to clamp at the minimum and maximum observed value for the particular color channel for the particular image or tile in question. For reasonable-sized tiles, that might matter.)

    ---

    One thing I've been thinking about is that what works depends a whole lot on quantization. For example, if you have a simple shallow gradient fill with no noise, but quantized to 8 bits, you'll tend to get a stairstep function like 31 31 31 30 30 30 29 29 29...
    So ClampAddSubtractFull will overshoot at each step---once you see a step down, you won't see another for another for two or more pixels. But maybe that case doesn't matter much, because the errors will be small anyhow. The small errors will inhibit string matching, but entropy coding the literals will work reasonably well.

    ---

    An annoying version of the quantization issue is where contrast and color adjustments have stretched or squeezed the tonal range. That will convert simple gradients from relatively regular step functions to irregular ones, with some steps disappearing (where the range has been squeezed) and others being doubled (where it's been stretched), and which steps disappear or are doubled will depend on the exact gradient, contrast adjustment curve, and quantization. Ew.

    I am very unclear on what cases really matter, and how all these things interact with color decorrelation, etc.

    I'd be interested in your thoughts on these things.

  10. #8
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by Paul W. View Post
    Jyrki, can you explain your predictors and why SELECT works better than Paeth?
    The main problem of Paeth is that it can take the red component from the L pixels and the green from T. For RGBA, this creates a possibility of 8x more prediction pixels than Select. For example, if you only have red and green pixels in the image, it is possible that Paeth predicts yellow or black. This creates different groupings of prediction errors than Select. These different groupings reduce the efficiency of the color cache, LZ77 and cross color transform to further reduce entropy.

    In an even call between T and L pixels, I used to return L (it is a better candidate because of gravity), but later optimizations done by the webp-team changed it accidentally to T. When it was noticed, they decided to change the bit stream specification rather than return the code to its original state. https://developers.google.com/speed/...ctor_transform -- search for 'AMENDED'

    If I remember correctly it is around 0.01 % in compression density, and definitely no more than 0.15 %.

    ... and never returning TL is again less branches, I don't really remember how much it was in compression density, and if it was positive or negative -- I think it was probably negligible (< 0.01 %).

    I tried quite a few things when designing the Select predictor, and its design was driven only by simplicity, decoding speed and compression density. I particularly didn't want to just replicate Paeth because I disliked its aesthetics, it creating a pixel out of two pixel somewhat randomly, and the many many branches it needs.

    Quote Originally Posted by Paul W. View Post
    One thing I've been thinking about is that what works depends a whole lot on quantization.
    Me, too. New video formats and even the old JPEG solve some quantization issues efficiently by having an intermediate representation of more than 8 bits of dynamics. If I would have to redo WebP lossless from scratch, I'd probably base it on 12-15 bit dynamics (not 16 so that much of arithmetic could still be done with 16 bit SIMD vectoring, one bit for prediction residual sign, possibly a few for averaging -- not quite sure about that), and expect 5-10 % compression gains in certain use cases.

  11. The Following User Says Thank You to Jyrki Alakuijala For This Useful Post:

    Paul W. (27th June 2016)

  12. #9
    Member
    Join Date
    Sep 2010
    Location
    US
    Posts
    126
    Thanks
    4
    Thanked 69 Times in 29 Posts
    Very flat gradients like 00001111122222 can only be predicted well with high intermediate precision *and* very wide filter neighborhoods.

    But those are pretty rare in real images. In most real images that have shallow gradients like that, you also have noise or dithering, which is even *worse* for localized predictors, because they see the noise as real slopes to predict. You need quite large filter regions indeed to be able to remove the dithering from the prediction.

    In the end there's a compromise for the efficiency of using only the 1-ring neighborhood.

    The big thing about Select is that it's done on pixels whereas Paeth is done on channels. It seems like a common thread in general in webp-ll to try to work on whole pixels instead of individual channels, which I think is a pretty strong principle.
    Last edited by cbloom; 3rd August 2016 at 19:06.

  13. The Following User Says Thank You to cbloom For This Useful Post:

    Paul W. (27th June 2016)

  14. #10
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by cbloom View Post
    Very flat gradients like 00001111122222 can only be predicted well with high intermediate precision *and* very wide filter neighborhoods.
    No need for wide filters actually for flat gradients. This example can be generated by a simple small linear extrapolator, first you emit 0.17, then 0.38, and extrapolate the rest linearly:

    >>> [int(0.17 + 0.21 * i) for i in range(14)]
    [0, 0, 0, 0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2]



    Quote Originally Posted by cbloom View Post
    It seems like a common thread in general in webp-ll to try to work on whole pixels instead of individual channels, which I think is a pretty strong principle.
    The basic inspiration for thinking in pixels came from my earlier experience of writing a fast PNG encoder. There, I noticed that for a relatively fast speed encoding I got better results when I aligned LZ77 matches and lengths at pixel boundaries even when up to four bits would be wasted per backward reference, and even when the interleaved filter bytes spoil part of the entropy reduction gained for the shortest lengths and distances.

  15. #11
    Member
    Join Date
    Sep 2010
    Location
    US
    Posts
    126
    Thanks
    4
    Thanked 69 Times in 29 Posts
    Jyrki, is the lz77 part of webp-ll done on pixels before filtering? eg. on original pixels, not residuals?

    Did you ever look at doing delta-matches? (send imperfect match offset then residual)

    In something like PNG where the lz77 is done on pixels after filtering, it's sort of a weird thing. On noisy continuous tone images, the residuals are pretty close to memoryless laplacians, so the lz77 winds up being just a funny way to do entropy coding. (this is why cranking min-match-len way up on the PNG lz can help a lot). On some graphic images, the lz of PNG residuals can still find patterns, so it's doing more of a shape-match, but it's still not ideal.
    Last edited by cbloom; 3rd August 2016 at 19:06.

  16. #12
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by cbloom View Post
    Jyrki, is the lz77 part of webp-ll done on pixels before filtering? eg. on original pixels, not residuals?
    LZ77 is the last phase, i.e., it can work on pixels only if no filtering was used. There are up to four possible filtering stages (8-bit palette, subtract green, cross color, and spatial prediction). The encoder can decide about the order, but only use one of each.

    Quote Originally Posted by cbloom View Post
    Did you ever look at doing delta-matches? (send imperfect match offset then residual)
    I thought about it when designing WebP lossless. I didn't do experiments with it, but just thinking about it I couldn't find a way that was reasonably fast to encode and that would give savings for a large class of images. Also, when you have a lot of zeros in a residual, you'd need arithmetic coding, and one of the design goals I had set was to be able to keep it in more simple prefix coding.

    Quote Originally Posted by cbloom View Post
    In something like PNG where the lz77 is done on pixels after filtering, it's sort of a weird thing. On noisy continuous tone images, the residuals are pretty close to memoryless laplacians, so the lz77 winds up being just a funny way to do entropy coding.
    WebP lossless LZ77-phase has an "RLE"-only trial for this purpose. The "RLE" includes two kinds of RLE, the real RLE for copying a single pixel, and a second LZ77 try of copying aligned pixels (or residuals) from the previous row.

    My understanding of the residuals is that sometimes there are interesting correlations, at least when the image has been upsampled or constructed by a camera from the sensor to lossless RGB (or even possibly with high quality mode to JPEG). Some pixel level interpolation patterns can be found from these images. I believe PCIF/BCIF is good at compressing such patterns, and possibly loses some of its edge when the image is downsampled.

  17. The Following User Says Thank You to Jyrki Alakuijala For This Useful Post:

    Paul W. (30th June 2016)

  18. #13
    Member
    Join Date
    Oct 2013
    Location
    Filling a much-needed gap in the literature
    Posts
    350
    Thanks
    177
    Thanked 49 Times in 35 Posts
    Compressing uncompressed PNGs of the Waterloo RGB images, we found that PPMd works well, but the best -o (maximum order) can be anywhere from 2 to 6 or so. 4 seems good on average. We tried an adaptive order 0 model but that did not work nearly as well, which we interpreted as meaning that there's significant sequential information in there, not just residuals needing to be entropy coded.

    Presumably that would be less true with better prediction up front, but my impression is that unless you have really very good prediction, it'll be hit or miss---a more subtle predictor will generate smaller residuals, but also tend to obscure higher-order regularities and inhibit string matching by the subsequent compressor. For LZ77, you may usually be better off just giving up on string matching because the typical strings are short, but PPM can do a good job at low orders that aren't 0. GLZA is better than LZMA in this regime, and more closely approximates PPM while still being fast to decode.

    For some images, you can get better results by quantizing out the lowest bit of each byte, and just storing those bits, because it's the lowest bit that's noisiest and inhibits string-matching the most. I'd think that would work much better and more consistently if you only did it to problematic tiles of an image, rather than a whole image, and often only for the hoisiest color plane. Where you have a region with a dominant color (reddish, say), and that color has a dominant color component (red, say), the other components are darker and effectively more coarsely quantized already, so you get longer string matches and should leave those alone.

  19. The Following User Says Thank You to Paul W. For This Useful Post:

    Jyrki Alakuijala (1st July 2016)

  20. #14
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    This is very interesting. Would you be willing to share more detailed results of the experiments, even down to how many bytes per image for a given attempt you made?

    How did PPMd compare with just WebP lossless?

    Quote Originally Posted by Paul W. View Post
    For some images, you can get better results by quantizing out the lowest bit of each byte, and just storing those bits, because it's the lowest bit that's noisiest and inhibits string-matching the most. I'd think that would work much better and more consistently if you only did it to problematic tiles of an image, rather than a whole image, and often only for the hoisiest color plane.
    WebP near lossless at setting 80 removes one bit in areas where the missing one bit is very difficult to observe. It keeps smooth areas intact, only changes the noisy areas.

    At setting 60 two lowest bits can be removed, but again this is done in a way that is quite difficult to observe.

    Quote Originally Posted by Paul W. View Post
    Where you have a region with a dominant color (reddish, say), and that color has a dominant color component (red, say), the other components are darker and effectively more coarsely quantized already, so you get longer string matches and should leave those alone.
    Could you elaborate on this?

  21. #15
    Member
    Join Date
    Oct 2013
    Location
    Filling a much-needed gap in the literature
    Posts
    350
    Thanks
    177
    Thanked 49 Times in 35 Posts

    results of compressing PNGs w/LZMA, GLZA, PPMd, vs. WebP lossless

    Jyrki,

    Here are some basic compressiion ratio results for compressing unoptimized Waterloo PNGs, vs. WebP lossless.

    WebP lossless usually wins, which I'd pretty much expect given that these are unoptimized
    PNGs, and PNG is pretty primitive anyhow. WebP kills on 2 of the 3 digital paintings; I wish I knew
    what they're representative of.

    LZMA dominates gzip, of course. On photos, GLZA dominates LZMA, and PPMd dominates GLZA (but GLZA
    is faster).

    Code:
    
    Photographic images from Waterloo:
                         
                              blindly recompressing                compressing
                        |     unoptimized PNG files         |      from scratch
               pngout + |                                   |
    image      pngcrush |  gzip-9   lzma-9  glza0.5  ppmd-4 |  WebPll  flif-N   flif
    -----      -------- | -------   ------  -------  ------ |  ------  ------  ------
                        |                                   |
    lena         473243 |  512773   472670   462729  394441 |  428642  442895  431625
    monarch      613953 |  638287   571898   564104  529833 |  457392  431001  404839
    peppers      423881 |  452698   417627   398446  394151 |  344492  315417  325056
    sail         771089 |  796427   721655   678377  666192 |  562986  524936  506708
    tulips       679152 |  709338   646904   631710  606599 |  531486  495859  458325
                        |                                   |
    PHOTO SUM:  2965041 | 3109523  2830754  2735366 2591216 | 2324998 2210108 2126553
                        |                                   |
    % vs. flif:   +39   |   +46      +33      +29     +22   |     +9     +4      +0
    
    ---
    
    "Artistic" images from Waterloo (synthetic, digital paintings):
    
                               blindly recompressing                compressing
                        |      unoptimized PNG files        |       from scratch
               pngout + |                                   |
    image     pngcrush  |  gzip-9   lzma-9  glza0.5  ppmd-4 |  WebPll  flif-N   flif
    -----      -------- | -------   ------  -------  ------ |  ------  ------  ------
                        |                                   |
    clegg       474298  |  509195   373394   408192  394441 |  379506  338218  716210
    frymire     252423  |  375351   301294   299794  319100 |  174132  213268  273275
    serrano     107167  |  152839   131785   133108  131389 |   76190   98522  116886
                        |                                   |
    SYNTH SUM:  833888  | 1037385   806473   841094  844930 |  629828  650008 1106371
                        |                                   |   
    % vs. WebPll:  +32  |   +65      +28      +33     +34   |     0      +3     +76 
    ---
    
    SUM (ALL)  3798929  | 4146908  3637227  3576460 3436146 | 2954826 2860116 3232924
    Last edited by Paul W.; 3rd July 2016 at 00:14. Reason: improved table

  22. The Following User Says Thank You to Paul W. For This Useful Post:

    Jyrki Alakuijala (2nd July 2016)

  23. #16
    Member
    Join Date
    Oct 2013
    Location
    Filling a much-needed gap in the literature
    Posts
    350
    Thanks
    177
    Thanked 49 Times in 35 Posts
    The quantization thing I'm talking about is basically to selectively quantize parts of the image if that makes them a lot more compressible, and store the differences so that the image can be reconstructed losslessly. (Pretty much what I take Charles to be talking about.)

    (Any of the following could be wrong... I only did some simple unsystematic sloppy tests and might have made a conceptual error.)

    If you're going to get much benefit from using a string-matching compressor like LZ77 or PPMd, rather than just entropy coding residuals, you have to match strings exactly; they have no notion of an approximate or fuzzy match. You can quantize the data, though, and more strings will match. For patterns that are almost compressible, the biggest cause of mismatches is random or mispredicted variation in the lowest bit or two.

    For the brightest color component---say, red, if we're looking at something reddish---the lowest bit varies more often than for dimmer components. If there's a slight random brightness variation but the color is constant, most of that variation will be in the red channel, so its low bit will change more often.

    Two examples:

    1) Suppose you have a slight random variation of about 1 bit of luma in a fixed reddish color. The brighter red component is responsible for most of the luma, so it's likely that its low bit will change. But the dimmer blue and green channels can vary by the same percentage, without usually affecting the 8-bit quantized value: the *very* slight variation usually either won't be picked up by the camera sensor, or will be lost at some point in conversion to 8-bit RGB.

    2) Suppose you have a descending gradient of a fixed reddish color. If you separate color planes, you'll see more total steps from the lightest value to the darkest value in the red channel than in the other two channels---it starts out higher, and has to step down more often than the other two channels as it descends toward black, where they're all zero. In the different color planes, you'll see different stairstep functions, with the red stairs steeper than the green and blue stairs.

    (That reasoning is going to be wrong where the image is underexposed and you're actually getting sensor noise in the lowest bits, affecting the dim channels as much as the bright one.)

    In this case, it would probably pay to quantize away the lowest bit of the red channel, and just store 1/8 of the bits so that the remaining 7-bit values can be string-matched. The red and blue channels may compress better as they are, because they vary less often, and you don't take a 12.5 % ratio hit for storing the low bits of each byte uncompressed.

    It seems to me that it should be useful to have a quantizing-split transform, which you'd often apply to only the brightest color plane in certain areas of the picture. (That should be a very fast transform, with SIMD instructions or just bit-slice SIMD-in-scalar parallelism.)

    It may work better to do smart color transforms and get the same effect by quantizing the luma slightly in some cases, but that seems much trickier; at any rate, I haven't figured out what happens to patterns when you take values that are already simply quantized and start combining them.
    Last edited by Paul W.; 1st July 2016 at 18:59.

  24. #17
    Member
    Join Date
    Sep 2010
    Location
    US
    Posts
    126
    Thanks
    4
    Thanked 69 Times in 29 Posts
    With LZMA you need to boost the image to 32-bit XRGB and use 2 pos bits. That gives each component it's own entropy coding.

    Also with the PNG filters you should be using the optional MNG LOCO color space transform.

    Comparing to your results on monarch :

    image gzip-9 lzma-9 glza0.5 ppmd-4 WebPll
    ----- ------- ------ ------- ------ ------
    monarch 638287 571898 564104 529833 457392


    Here's monarch boosted to 32-bit, filtered with my RRZ filter (clamped gradient + LOCO colorspace) and then compressed with lots of compressors :

    monarch.tga.rrz.ppmonstr,479244
    monarch.tga.rrz.ppmd,503476
    monarch.tga.rrz.zl8.LZNA,507548
    monarch.tga.rrz.co.nz,507828
    monarch.tga.rrz.cOO.nz,510104
    monarch.tga.rrz.zl6.LZNA,514054
    monarch.tga.rrz.d28lc0lp2pb2.7z,517702
    monarch.tga.rrz.d24.7z,532354
    monarch.tga.rrz.brotli11,540613
    monarch.tga.rrz.zl7.Kraken,554215
    monarch.tga.rrz.rar,568999
    monarch.tga.rrz.lzx21,572006
    monarch.tga.rrz.zstd060,576995
    monarch.tga.rrz.mc-.rar,608967
    monarch.tga.rrz.brotli9,624428
    monarch.tga.rrz.z9.zip,640171
    monarch.tga.rrz.raw,1572919
    Last edited by cbloom; 3rd August 2016 at 19:06.

  25. The Following 2 Users Say Thank You to cbloom For This Useful Post:

    Jyrki Alakuijala (2nd July 2016),Paul W. (1st July 2016)

  26. #18
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Appended with flif and flif -N, and calculated the total size of these eight images.

    Disclaimer: this benchmark is rather exotic, and probably not representative of your actual payload.

    Code:
    image      gzip-9   lzma-9  glza0.5  ppmd-4  WebPll  flif    flif -N
    -----     -------   ------  -------  ------  ------  ------  -------
    
    lena3      512773   472670   462729  394441  428642  431625   442895
    monarch    638287   571898   564104  529833  457392  404839   431001
    peppers3   452698   417627   398446  394151  344492  325056   315417
    sail       796427   721655   678377  666192  562986  506708   524936
    tulips     709338   646904   631710  606599  531486  458325   495859
    clegg      509195   373394   408192  394441  379506  716210   338218
    frymire    375351   301294   299794  319100  174132  273275   213268
    serrano    152839   131785   133108  131389   76190  116886    98522
    
    sum       4146908  3637227  3576460 3436146 2954826 3232924  2860116
    Last edited by Jyrki Alakuijala; 2nd July 2016 at 00:49.

  27. The Following User Says Thank You to Jyrki Alakuijala For This Useful Post:

    Paul W. (1st July 2016)

  28. #19
    Member
    Join Date
    Oct 2013
    Location
    Filling a much-needed gap in the literature
    Posts
    350
    Thanks
    177
    Thanked 49 Times in 35 Posts
    Thanks, Charles.

    Uh-oh. I've been under the impression that PNG deinterleaves 24-bit RGB images into R, G, and B image planes, so that the low offset bits thing wouldn't help LZMA. (But maybe I was misinterpreting something that was really about *prediction* within planes.) If not, I need to re-think several things. (Including the fact that it works fairly well to just deinterleave the planes, DPCM, and compress. I thought PNG was doing basically that, but with 2D prediction rather than just horizontal. If it's not, I should try that, too.)

    Concerns about unfairness to LZMA partly motivated the original post---I was thinking that pnmtopng splitting the image data into a bunch of IDAT chunks with checksums would throw off LZMA's repeat match offset thing somewhat. I wouldn't think it would matter much for this, but am not sure.

    If the low offset bits thing matters, that's more important, and there's another problem---if more than one predictor is used, each raster has an extra byte at the beginning to say which predictor was used for it, which will rotate the offsets through 0, 1, and 2 in a cycle at raster boundaries too. (If I knew how to dump just the residuals, I could easily strip those out.)

  29. #20
    Member
    Join Date
    Oct 2013
    Location
    Filling a much-needed gap in the literature
    Posts
    350
    Thanks
    177
    Thanked 49 Times in 35 Posts
    In some experiments with just splitting color planes and DCPMing (i.e., LEFT (horizontal) predicton only), I found that doing a simple color transform first helped sometimes but often hurt, too. (I think the transform is equivalent to MNG LOCO or SUBTRACT_GREEN, which I take to be identical: (R-G) G (B-G).)

    In that scenario (separate color planes), I guessed that the problem was that the planes were often independently compressible, and that mixing information from different channels was bad. That's what got me thinking about the quantization thing and string matching, and thinking that the more predictable/string-matchable channels should be left alone.

    All other things being equal, I'd expect that prior stuff in a given color plane is often as good a predictor as any for that color plane, and the decisions whether to do a MNG LOCO transform and/or deinterleave is pretty much a crapshoot. But I'm a noob, and just don't know.
    Last edited by Paul W.; 2nd July 2016 at 02:44. Reason: typo: SELECT_GREEN -> SUBTRACT GREEN

  30. #21
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    In WebP lossless I estimate the entropy of a five combos of transforms, and decide the final transform combo on minimizing the entropy. It is not a replacement for crunching through all options for the absolutely best results, but works relatively well. At that stage there is also a simple heuristic that tries to simulate the impact of possible LZ77 on the entropy, too (it just copes with rle and previous line copy).

    https://github.com/webmproject/libwe...nc/vp8l.c#L139 (five combos)

    https://github.com/webmproject/libwe...nc/vp8l.c#L197 (simple heuristic to reduce LZ77/rle encoding discrepancy with entropy estimation)

  31. The Following User Says Thank You to Jyrki Alakuijala For This Useful Post:

    Paul W. (2nd July 2016)

  32. #22
    Member
    Join Date
    Oct 2013
    Location
    Filling a much-needed gap in the literature
    Posts
    350
    Thanks
    177
    Thanked 49 Times in 35 Posts
    I added the FLIF numbers Jyrki reported to the table in post #15 above, and numbers for pngout + pngcrush.

    Pretty striking that FLIF is the best on the photos, and easily the worst on the synthetic images---even worse than just gzipping an unoptimized, uncompressed PNG.

  33. #23
    Member
    Join Date
    Oct 2013
    Location
    Filling a much-needed gap in the literature
    Posts
    350
    Thanks
    177
    Thanked 49 Times in 35 Posts
    Jyrki,

    In the Bradac experiment, was the palette cache disabled for the "other" compressors (besides the normal 2D-LZ77), or were hits to it encoded into the front-end output somehow?

  34. #24
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by Paul W. View Post
    Jyrki,

    In the Bradac experiment, was the palette cache disabled for the "other" compressors (besides the normal 2D-LZ77), or were hits to it encoded into the front-end output somehow?
    The color cache is relatively deeply integrated with the entropy coding, so it was disabled for all other compressors.

  35. The Following User Says Thank You to Jyrki Alakuijala For This Useful Post:

    Paul W. (10th July 2016)

Similar Threads

  1. Replies: 3
    Last Post: 16th May 2016, 00:02
  2. PNG in .ICO file optimizer ?
    By SvenBent in forum Data Compression
    Replies: 9
    Last Post: 21st April 2016, 17:30
  3. Changing PNG compression
    By przemoc in forum Data Compression
    Replies: 5
    Last Post: 15th August 2012, 21:29
  4. Easy way for BMF to png
    By SvenBent in forum Data Compression
    Replies: 5
    Last Post: 13th November 2008, 09:13
  5. unsupported PNG verient for Precomp
    By maadjordan in forum Data Compression
    Replies: 5
    Last Post: 22nd May 2008, 16:22

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •