Results 1 to 20 of 20

Thread: Combining compression algorithms (image + archive)

  1. #1
    Member
    Join Date
    May 2016
    Location
    Earth
    Posts
    8
    Thanks
    0
    Thanked 1 Time in 1 Post

    Combining compression algorithms (image + archive)

    Hello,

    I'd like to ask the experts' opinion.

    I want to distribute ~3500 bitmaps, so I'm looking to minimize the size of this bunch of files.
    I used a sample of 140 bitmaps of various sizes, and ran a few experiments.

    First, I just put the uncompressed bitmaps in a tar, and compressed with xz.
    Then I thought I could compress the bitmaps beforehand (I tried PCX and PNG).

    74M bmp
    67M pcx
    35M png

    20M bmp.txz
    24M pcx.txz
    34M png.txz

    As you can see, the uncompressed bitmaps weigh 74 MB.
    PCX saves ~10%
    PNG saves ~50%

    But then compressing these folders brings disappointing results.
    The BMP archive compresses best, while the PNG archive doesn't compress at all.
    As a result, the compressed BMP archive is much smaller than the PNG archive.

    Is this expected?

    Is there a way to distribute a small archive, containing compressed images?
    Maybe I should try other formats than PNG and xz?
    What do you think?

    Regards.

  2. #2
    Member Samantha's Avatar
    Join Date
    Apr 2016
    Location
    italy
    Posts
    38
    Thanks
    31
    Thanked 7 Times in 4 Posts
    You can use "bpgenc & bpgdec" compressor, in my opinion one of the best compressors for images...after converting a series of images you can to archive as you see fit.
    If you need to explanation or help ask.

    Code:
    FOR /r "Temp\" %%F IN (*.jpg) DO (
    WorkBook\bpgenc.exe "%%F" -q 30 -f 420 -c ycbcr -b 10 -o"Temp\%%~nF.bpg" "%%F"
    ECHO Files Processed [%%~nF.jpg] To [%%~nF.bpg]
    )
    pause
    bpg-0.9.6-win64.zip

  3. #3
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    You could also give FLIF a try - it's specialized on images and could perform even better than BPG (and also, much better than PNG).
    http://schnaader.info
    Damn kids. They're all alike.

  4. #4
    Member
    Join Date
    May 2016
    Location
    Earth
    Posts
    8
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Samantha View Post
    You can use "bpgenc & bpgdec" compressor, in my opinion one of the best compressors for images... after converting a series of images you can to archive as you see fit. If you need to explanation or help ask.
    Hello,

    Thanks for the suggestion. Does BPG have a lossless mode?

    However, I think my problem statement was not very clear. The images are meant to be distributed and used as graphics resources for a game. (I'll have to check, but I don't think e.g. SDL supports BPG.)

    My main question was: why do a bunch of BMP images compress to a much smaller size than the same PNG-compressed images. I think it might be because, with BMP, lots of colors repeat, but with PNG, the compression has "mixed" the bytes in a way that xz finds much less repetitions?

    Regards.

  5. #5
    Member
    Join Date
    May 2016
    Location
    Earth
    Posts
    8
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by schnaader View Post
    You could also give FLIF a try - it's specialized on images and could perform even better than BPG (and also, much better than PNG).
    Thanks for the suggestion. I will give it a try, to get a feel for the improvement over PNG.

    I suppose a bunch of images compressed with BPG or FLIF do not compress at all with xz? (Same as with PNG).

    Until now, the best compression I got was to tar the BMPs and use xz on that.
    (But I don't like that solution because it takes up too much space on the target.)

    Regards.

  6. #6
    Member
    Join Date
    May 2016
    Location
    Earth
    Posts
    8
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Noob View Post
    I'll have to check, but I don't think e.g. SDL supports BPG.
    https://www.libsdl.org/projects/SDL_image/

    "supports the following formats: BMP, GIF, JPEG, LBM, PCX, PNG, PNM, TGA, TIFF, WEBP, XCF, XPM, XV"

    I suppose WEBP is the most recent/best algorithm in the list?

    Regards.

  7. #7
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    If you want lossless i think FLIF/PNG are your best bets overall. FLIF is not well supported yet though, there is a plugin posted here for xnview though, sadly not the awesome IrfanView yet PNGs will just work for your target audience though without having to dig up special tools they've never heard of.

    TIFF with LZW compression is a good choice, and will on certain images compress better than PNG.

    PCX is ancient, so avoid that.

    WebP can do lossless but last i tried it wasn't that impressed but worth checking out as it was a few versions ago.

    BPG can do lossless too i believe, but zero support pretty much apart from old test versions of IrfanView which he pulled due to potential copyright issues he didn't wanna risk, he said i could keep my copy for testing/personal purposes only. BPG is still based on earlier x265 code as well, not current. Good quality at low bitrates, but i would honestly still use JPEG/WEBP at higher levels anyday. 0.6.7 was released a few days ago but it just patched JS issues.

  8. #8
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    To answer the original question, you have mutual information in the BMP files that goes undetected if you compress them individually.

  9. #9
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    I don't think changing to flif is going to get you near just compressing BMP's as whole. as Matt is saying by using per file compression you are only finding redundant data within a small section, by keeping them uncompressed and compressing them in a big archive you are finding redundant data through all of them.
    basically it like using a small dictionary vs a big dictionary compression.

    What you might try to do is use png with optimal delta filters but no compression and see if it helps. basically you want some kind of preprossessing that is not a full compression of the data.

  10. #10
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by Noob View Post
    74M bmp
    67M pcx
    35M png
    Guesswork numbers:

    Assuming the png is a result of a ZopfliPNG run:

    If you compress with WebP lossless (or even better if with -near_lossless 60), you will get around 25.9 MB, and decoding time is unchanged.

    If you compress with FLIF, you will get around 25.1 MB, but decoding gets 15x slower than with PNG and WebP lossless.

    Anything else is going to be either too slow or worse in compression. If you didn't do PNG optimization yet, you will see slightly better numbers for WebP lossless and FLIF.

    If you combine the images into larger images consisting of self-similar images, you will see a further improvement in compression ratio, better than your txz numbers. An image compressor can do this more efficiently than a normal compressor, particularly both WebP lossless and FLIF are more efficient than xz in compressing images. CSS sprite generation is one example application of this. See http://graphicdesign.stackexchange.c...parent-iconset

  11. #11
    Member
    Join Date
    May 2016
    Location
    Earth
    Posts
    8
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Jyrki Alakuijala View Post
    Guesswork numbers:

    Assuming the png is a result of a ZopfliPNG run:

    If you compress with WebP lossless (or even better if with -near_lossless 60), you will get around 25.9 MB, and decoding time is unchanged.
    (I used imagemagick to convert from BMP to PNG.)

    I had to compile cwebp 0.5 from source because my distro packages only version 0.4 (which doesn't support near_lossless option)

    I used -q 100 (for max compression) for the cwebp runs.

    Here are the results:

    76486418 bmp
    20764348 bmp.txz
    36281860 png
    35994088 png.txz
    29079746 webp_lossless
    29123488 webp_lossless.txz
    18277494 webp_near_lossless_60
    18314088 webp_near_lossless_60.txz

    So thanks for pointing out the near_lossless option.

    Regards.

  12. The Following User Says Thank You to Noob For This Useful Post:

    Jyrki Alakuijala (27th June 2016)

  13. #12
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    That looks great! WebP near lossless was recently improved significantly based on butteraugli experiments, roughly so that level 40 become as good as 60 in quality, 60 as good as 80 etc. If you do a quality comparison try to get your version from the head instead of using the versioned 0.5.0.

    WebP quality settings are not that simple: lossless and near lossless coding requires two additional parameters for maximum compression: -m 6 and -q 100 (or according to the help a new flag -z 9 would be sufficient, too -- but I never tried that one yet).

    ... and -alpha_q does not impact WebP lossless, it is only for lossy + alpha.

  14. #13
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    The problem with accepting lossy is that you can always get it smaller. i would re think if you really want to throw away quality/accuracy for the compression gains.
    it all depends on what userbase it is. if its just a one time thing or a repeated compress/decompress situation. In the later you risk continual destruction of quality over time.

    Just my 2 cents.

  15. #14
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by SvenBent View Post
    The problem with accepting lossy is that you can always get it smaller.
    There are many aspects to it. If the loss is not-that-bad, i.e., +-1 in the most noisy areas of the images, it will be near impossible to find it, even in a delicate flip test. Still, you can get 15 % less bytes, and a 15 % speedup in decoding. cwebp -near_lossless 80 does that, with -60, it allows +-2, still near impossible to see in the noisy areas. Smoother areas are not touched.

    Another cool compression option for releasing dense and fast-to-decode graphics is still the palette encoding. A lot less entropy coding can be done, and the decoding can be twice as fast as for real true color. Of course the image will suffer more than in simple -near_lossless 60 encoding.

    If you are releasing a game for example, light near lossless (and palette when it works) seems to me almost always favorable to true lossless. Give us more resolution, more different textures rather than every leaf on a tree down to their correct rgb in 24 bits

    Try out -near_lossless 60 and -near_lossless 80, they are not the kind of loss that you are used to with JPEG or WebP lossy.

  16. #15
    Member
    Join Date
    May 2016
    Location
    Earth
    Posts
    8
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Jyrki Alakuijala View Post
    WebP near lossless was recently improved significantly based on butteraugli experiments, roughly so that level 40 become as good as 60 in quality, 60 as good as 80 etc. If you do a quality comparison try to get your version from the head instead of using the versioned 0.5.0.
    I had actually started by cloning the repo. But it seems the project only bundles the configure script with actual releases?

    How do I perform a quality comparison? By visually comparing the two versions of the image? Or did you have objective measuring tools in mind?

    Regards.

  17. #16
    Member
    Join Date
    May 2016
    Location
    Earth
    Posts
    8
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Jyrki Alakuijala View Post
    Try out -near_lossless 60 and -near_lossless 80, they are not the kind of loss that you are used to with JPEG or WebP lossy.
    (These images are indeed game resources.)
    It seems that, even though -near_lossless takes a 0..100 value, many values are equivalent?
    (Apparently, only multiples of 20 are useful for now?)
    Regards

  18. #17
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by Noob View Post
    (These images are indeed game resources.)
    It seems that, even though -near_lossless takes a 0..100 value, many values are equivalent?
    (Apparently, only multiples of 20 are useful for now?)
    Regards
    Yes, only multiples of 20 -- and most likely 0 and 20 are useless for all reasonable purposes.

  19. #18
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by Noob View Post
    By visually comparing the two versions of the image? Or did you have objective measuring tools in mind?
    Yes, check your results visually. -near_lossless 60 should be a safe bet, and 40 is getting to a riskier zone (but likely still ok).

  20. #19
    Member
    Join Date
    May 2016
    Location
    Earth
    Posts
    8
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Jyrki Alakuijala View Post
    If you do a quality comparison try to get your version from the head instead of using the versioned 0.5.0.
    Hello Jyrki,

    Are you the dev in charge of the lossless codec in WebP?

    I noticed that James Zern announced libwebp 0.5.1
    https://groups.google.com/a/webmproj...ss/vdwAxnmzsbw
    (God, I hate Google Groups)

    Does this new version include the near_lossless improvements you had in mind?

    The release notes mention:

    * lossless encoding performance improvements
    * memory reduction in both lossless encoding and decoding

    Regards.

  21. #20
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by Noob View Post
    Are you the dev in charge of the lossless codec in WebP?
    No. I contribute sometimes to WebP development, but I don't make decisions about any part of WebP. Particularly, I never participated in WebP lossy.

    I designed the WebP lossless format in 2011, participated in the initial production-level implementation with the WebP team in early 2012, and moved away from the project in August 2012 to work on an research compressor that seeded the development of brotli.

    Every now and then I improve the old WebP lossless algorithms for density or speed -- the team maintaining WebP has been focusing on WebP lossy, at least until very recently.

    Near-lossless was already in my initial version in 2011, but the WebP team didn't want it in the inital production-level code. The way I implemented near-lossless is very different from the typical near-lossless coding. I don't maximize savings and quantize everywhere, but I first evaluate where the loss is invisible and try to apply it only there.

    I found the near-lossless mode very useful, so I kept insisting on it, and in 2015 the WebP team was ready to add it to the production version. The most recent near-lossless improvements are a result of the development of butteraugli. I had run butteraugli on 1000 pngs for WebP near-lossless and analyzed what happened with the worst few images, and came up with the simple bankers rounding algorithm that removed the slight intensity shift that the previous version had for certain kinds of images.

    Quote Originally Posted by Noob View Post
    Does this new version include the near_lossless improvements you had in mind?
    Yes. There will be some more density/quality improvements to near-lossless, but the first big step is in 0.5.1-rc5 (and it is missing in 0.5.0).

Similar Threads

  1. fastest open source integer compression algorithms
    By mitra in forum Data Compression
    Replies: 2
    Last Post: 5th September 2015, 18:21
  2. Papers - Summary of Data Compression Algorithms
    By Gonzalo in forum The Off-Topic Lounge
    Replies: 12
    Last Post: 10th February 2015, 08:24
  3. Compressed Archive: Identifying compression method and Decompressing?
    By UnknownToaster in forum Data Compression
    Replies: 34
    Last Post: 24th September 2014, 21:42
  4. Replies: 1
    Last Post: 21st June 2011, 11:48
  5. Replies: 2
    Last Post: 18th April 2011, 04:13

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •