Results 1 to 6 of 6

Thread: Multiple image compression problem - looking for solution

  1. #1
    Member
    Join Date
    Sep 2017
    Location
    Norway
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Multiple image compression problem - looking for solution

    Hello,

    I hope this is the right forum for my question, and if not please advise me to remove the post.

    I have an interesting problem. At least I think it's interesting. And I think that you might be able to help me with considering your extensive knowledge of data compression.

    We have a series of images produced during the working day, around 500 000 to 800 000 of them, that needs to compressed from a BPM format, and uploaded to a storage service in the cloud (AWS is our choice, but it can be anything). The images are around 3MB uncompressed. The compression has to be lossless.

    The images are all of salmon filets, so look almost identical.

    I've looked at some compression technology and found the FLIF format which looks very promising for the images individually, but am wondering. If we are to send these 500 000 images to the cloud, would it be advisory to try to compress them altogether into an archive after compressing them with FLIF? Or maybe not use FLIF at all, but there are other technologies that suit this challenge better?

    Any advice would be greatly appreciated!

  2. #2
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    Due to the similarity of the images, I'd try to take a bunch of them (10 - 100 images), encode them as a FLIF animation (e.g. on the command line, just enter multiple input images and adjust --max-frame-lookback as you need) and see if it's better than encoding them seperately.
    http://schnaader.info
    Damn kids. They're all alike.

  3. The Following 2 Users Say Thank You to schnaader For This Useful Post:

    Gonzalo (26th September 2017),Shelwien (26th September 2017)

  4. #3
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    464
    Thanks
    202
    Thanked 81 Times in 61 Posts
    I tried a few different approaches to the same problem a while ago. I haven't tried Christian's tough.
    But for me, the best result was using something like this:
    Code:
    shar as - *.bmp | fazip rep+mm+grzip
    //or
    shar as - *.bmp | fazip rep+lzma
    Plain and simple. That if the images are really alike. But given the amount of files you have, their sizes, and how similar they are, I would favour any solid lempel-ziv method with a large dictionary over an image-specific one. Maybe 7z, rar or even the new razor.

    Also, zpaq is really great at storing similar files because it is designed to de-duplicate everything in the first place, and it can use the very first file you added to find similarities with any other you might add later. I'm a little out of my element here, so I don't know if it were to be practical to use with the cloud.

    Maybe if you could provide a sample of any random day for us to try something else...


    BTW:
    We have a series of images produced during the working day, around 500 000 to 800 000 of them ... The images are all of salmon filets, so look almost identical.
    What the fish?

    Please excuse my ignorance, but would you care to elaborate on what kind of job do you have exactly? Perhaps some quality control department? Government health agency? Thank you! Just curious...

  5. #4
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    437
    Thanks
    1
    Thanked 96 Times in 57 Posts
    Quote Originally Posted by baretomas View Post
    We have a series of images produced during the working day, around 500 000 to 800 000 of them, that needs to compressed from a BPM format, and uploaded to a storage service in the cloud (AWS is our choice, but it can be anything). The images are around 3MB uncompressed. The compression has to be lossless. The images are all of salmon filets, so look almost identical.
    Frankly, if it has to be lossless, you'll get 2:1 if you try something simple. If you try very hard, you get 2:1. Maybe a little bit better. But you will not get away with much less than 1.5MB. If "visually lossless" is an option, then we're back in business, but otherwise, you will not make your images considerably smaller than that.

  6. #5
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by baretomas View Post
    I've looked at some compression technology and found the FLIF format which looks very promising for the images individually, but am wondering. If we are to send these 500 000 images to the cloud, would it be advisory to try to compress them altogether into an archive after compressing them with FLIF? Or maybe not use FLIF at all, but there are other technologies that suit this challenge better?
    FLIF is great!

    Psychovisually lossless: PIK's fast mode would likely be the best solution, but requires you to have AVX2 and linux. WebP -near_lossless 40 or 60 would very likely work out of the box for your needs, and is likely 20-30 % smaller than FLIF lossless. FLIF has similar near lossless abilities that are psychovisually ok.

    Faster decoding -- if decoding speed with FLIF becomes a problem, you can encode/decode to WebP lossless, and get 5x faster decoding with ~2 % more bytes. For the internet lossless use size ranges (5-250 kB) WebP lossless tends to be smaller in size, too.

    BCIF/PCIF seemed (benchmarking around 2011 done by others) to have wonderful lossless perfomance edge for non-resampled images, i.e., images that would include camera reconstruction artefacts, but lost that ~10 % edge if the camera resolution image was slightly downsampled (like often is in internet use).

    FLIC might be great, too, but is not open source.

  7. #6
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    232
    Thanks
    38
    Thanked 80 Times in 43 Posts
    Quote Originally Posted by thorfdbg View Post
    if it has to be lossless, you'll get 2:1 if you try something simple. If you try very hard, you get 2:1. Maybe a little bit better.
    Well, sometimes it's 2 : 1 with something as simple as bzip2 , but 3.5 : 1 with something not too complex, see http://qlic.altervista.org
    Note the numbers in the leftmost column are clickable, and as you'll see there are actually six test sets.

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

Similar Threads

  1. Replies: 8
    Last Post: 23rd September 2016, 13:41
  2. Replies: 3
    Last Post: 6th April 2014, 20:17
  3. Linux memory manager discussion, same problem as compression
    By nburns in forum The Off-Topic Lounge
    Replies: 10
    Last Post: 11th December 2013, 22:44
  4. Compression Problem
    By Nquisitive in forum Data Compression
    Replies: 19
    Last Post: 13th October 2012, 23:42
  5. Srep with multiple files support ?
    By SvenBent in forum Data Compression
    Replies: 3
    Last Post: 30th September 2010, 19:41

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •