Page 1 of 2 12 LastLast
Results 1 to 30 of 36

Thread: jpegultrascan, an exhaustive JPEG scan optimizer

  1. #1
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    47
    Thanks
    5
    Thanked 22 Times in 14 Posts

    jpegultrascan, an exhaustive JPEG scan optimizer

    I've just published jpegultrascan, a JPEG lossless recompressor that exhaustively searches all scans to find the smallest combination. Due to its exhaustive nature, it's slower than jpegrescan, which only searches a predetermined set of scan combinations, but not as slow as many PNG recompressors. With my files, I've seen an additional 5-10% file size savings over jpegrescan. It can generate files with a large number of scans, but I haven't run into compatibility problems thus far. It also supports certain obscure JPEG types such as CMYK.

    https://github.com/MegaByte/jpegultrascan

    As mentioned in the README, I recommend using it with both the IJG version and mozjpeg version of jpegtran (and -t enabled) to get the best compression the fastest.

    This version is a Perl script. I have part of a C implementation, but the libjpeg API is a bit of a mess, so I haven't finished it. In addition, in crunching JPEGs in this way, I found several bugs in mozjpeg that I'd rather have resolved before further development.

  2. The Following 6 Users Say Thank You to MegaByte For This Useful Post:

    Euph0ria (14th May 2016),fhanau (26th April 2016),Jaff (26th April 2016),lorents17 (26th April 2016),nikkho (26th April 2016),Stephan Busch (26th April 2016)

  3. #2
    Member
    Join Date
    Jun 2013
    Location
    USA
    Posts
    98
    Thanks
    4
    Thanked 14 Times in 12 Posts
    Wow you weren't kidding when you said this was slow. Seems to work though.

    Isn't mozjpeg an abandoned project?

    Does -i create files with the first scan being black and white? fun software bugs with that one...

  4. #3
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    (deleted)

  5. #4
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    47
    Thanks
    5
    Thanked 22 Times in 14 Posts
    Quote Originally Posted by Mangix View Post
    Does -i create files with the first scan being black and white? fun software bugs with that one...
    The first scan could be monochrome (luminance) or it could be full color depending on whichever combination is smaller. What -i does is allow for DC scans that include more than one but less than the full number of channels. Conversely, -i -1 disallows single channel DC scans, so monochrome wouldn't be possible in that case.

  6. #5
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    del
    Last edited by lorents17; 2nd May 2016 at 21:06.

  7. #6
    Member
    Join Date
    Feb 2016
    Location
    USA
    Posts
    41
    Thanks
    27
    Thanked 63 Times in 24 Posts
    Quote Originally Posted by Mangix View Post
    Wow you weren't kidding when you said this was slow. Seems to work though.

    Isn't mozjpeg an abandoned project?

    Does -i create files with the first scan being black and white? fun software bugs with that one...
    https://github.com/pornel/mozjpeg/tree/libjpeg-turbo
    Here is version of mozjpeg brought up to date with libjpeg-turbo. Not quite dead yet

    edit: moved to https://github.com/mozilla/mozjpeg/tree/libjpeg-turbo
    Last edited by Malloc Voidstar; 14th May 2016 at 13:19.

  8. #7
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by MegaByte View Post
    I've just published jpegultrascan, a JPEG lossless recompressor that exhaustively searches all scans to find the smallest combination.
    Would it be easy to try a lossy, but psychovisually lossless approach based on butteraugli within your recompressor? Butteraugli gives a spatial map indicating where the error exceeds the border of noticeably, so it would be perhaps possible to iterate with it.

  9. #8
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    47
    Thanks
    5
    Thanked 22 Times in 14 Posts
    Good question. The spatial map doesn't lend itself directly to the lossy approach since you can't define different plane splits for spatial sub-sections of the image. However, what you could do is estimate the value of each entire scan to see if you could omit it entirely and still preserve psychovisual integrity. To be fast, you would ideally want a way of performing psychovisual calculation on the scans rather than the entire image, though I'm not sure if that's possible. When I was first testing it out, I wasn't sure I'd be able to iterate through all combinations in a reasonable amount of time, so I was looking into ways to estimate the best combinations by looking at the information content of adjacent bits. I think you'd need to keep track of more than just the best combination though if you were interested in maximizing how many bits you could drop. So all said, yes, I think you could integrate lossiness using a similar approach, but if you wanted the optimal combination for a given quality target, it probably wouldn't come from using the exact algorithm in jpegultrascan. Of course, I imagine you'd get a much better result if you were starting with a lossless image to begin with rather than an already-compressed JPEG.

  10. #9
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    At start of a script:

    Code:
    perl jpegultrascan.pl 1.jpg 2.jpg
    Gives an error message:

    Couldn't read file at jpegultrascan.pl line 350.
    I start a script by means of ActivePerl 5.22.1 for Windows (x86). Prompt, please, what I do incorrectly?
    Last edited by lorents17; 13th May 2016 at 01:17.

  11. #10
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    47
    Thanks
    5
    Thanked 22 Times in 14 Posts
    Do you have jpegtran installed and in your path?

  12. The Following User Says Thank You to MegaByte For This Useful Post:

    lorents17 (14th May 2016)

  13. #11
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    It has turned out to solve a problem. But there was other problem, the project doesn't optimize the image (the attached file)?
    Attached Files Attached Files

  14. #12
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    47
    Thanks
    5
    Thanked 22 Times in 14 Posts
    That error message is coming from jpegtran. It says your JPEG is corrupted.

  15. The Following User Says Thank You to MegaByte For This Useful Post:

    lorents17 (14th May 2016)

  16. #13
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    Prompt whether it is possible to expect the version jpegultrascan integrated into jpegtran?

  17. #14
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    MegaByte
    Tell me, how I can save all versions of a file scan?

  18. #15
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    47
    Thanks
    5
    Thanked 22 Times in 14 Posts
    What do you mean all versions? Each scan or each scan combination? What are you trying to do?

  19. #16
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    What is JPEG progressive? Actually it is a number of images where quality of each following image is higher previous the image. JPEG Scan Killer project.

    We will take this image

    Click image for larger version. 

Name:	content.jpg 
Views:	206 
Size:	469.5 KB 
ID:	4436

    We receive 16 images.

    Click image for larger version. 

Name:	Снимок.PNG 
Views:	295 
Size:	5.9 KB 
ID:	4437

    How it is possible to create with the help-scans technology of optimization of JPEG lossy?

    JPEG consists of blocks of 8х8 pixels. Perhaps, there is a sense to segment the image and to optimize each part separately. Later to connect parts of the image in the uniform optimized image. Principle of work of the betterjpeg program.

    I wanted to learn, your opinion on the matter.
    Attached Files Attached Files

  20. #17
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    47
    Thanks
    5
    Thanked 22 Times in 14 Posts
    That description isn't quite accurate. The *combination* of previous scans ("images") along with the next scan is what makes each image iteration quality higher. Each scan by itself has some arbitrary amount of information depending on its parameters.
    As far as I understand the format, you can't choose scans for each block separately; it is a global encoding. However, if you're interested in lossy compression, you could potentially drop information in individual blocks to varying degrees. This is a different problem from optimizing scan choice, though both methods could be combined for even more size gains. It would be most straightforward to perform block-wise optimization and then see which (lossless) scan combination is best suited for that file. In principle, I suppose it's possible that a different combination of block optimization and scan optimization could yield an even smaller file, but searching that space may be very complex. A statistical approach might be possible.

  21. The Following User Says Thank You to MegaByte For This Useful Post:

    lorents17 (31st May 2016)

  22. #18
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    I thank for your answer!
    I wanted to ask you to make the program for optimization with losses with above the described functionality. I think much it will be interesting to test. Unfortunately, my knowledge in given area is very small, and for me this task isn't feasible.

  23. #19
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    164
    Thanks
    115
    Thanked 10 Times in 9 Posts
    C:\Perl\bin>perl jpegultrascan.pl -o . 1.jpg 2.jpg
    Use of uninitialized value $_[0] in join or string at jpegultrascan.pl line 92.
    Couldn't read file at jpegultrascan.pl line 350.

    jpegtran in the same dir
    activeperl 5.22.1.2201
    is it really possinle to run it under win10?
    tried perl jpegultrascan.pl -p . 1.jpg 2.jpg
    tried perl jpegultrascan.pl -o . 1.jpg 2.jpg
    tried perl jpegultrascan.pl -p jpegtran.exe 1.jpg 2.jpg
    tried perl jpegultrascan.pl -o jpegtran.exe 1.jpg 2.jpg

  24. #20
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    144
    Thanks
    10
    Thanked 151 Times in 80 Posts
    I just wrote a patch for mozjpeg that further optimizes the huffman encoding. The size difference is really tiny though (about 0.0013%). Just replace the original "jchuff.c" file with this one.
    Attached Files Attached Files

  25. #21
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    181
    Thanks
    74
    Thanked 10 Times in 10 Posts
    is it possible to create portable version of this tool, with everything needed inside, to make life for us noobs easier

  26. #22
    Member
    Join Date
    Jun 2016
    Location
    Earth, I think
    Posts
    13
    Thanks
    9
    Thanked 1 Time in 1 Post
    deleted
    Last edited by pico; 15th June 2016 at 09:38.

  27. #23
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    47
    Thanks
    5
    Thanked 22 Times in 14 Posts
    Okay, I updated the program to work under Windows. It turned out that there were a few problems (which also would have made jpegrescan not work).
    - The binary distribution of jpegtran for Windows doesn't have the same command line.
    - Perl pipe open doesn't work under Windows.
    - File::Temp has poor random logic when using fork.
    Anyway, these were all addressed and it is now working for me (Windows 10, Strawberry Perl 5.24)

  28. The Following User Says Thank You to MegaByte For This Useful Post:

    pico (27th June 2016)

  29. #24
    Member
    Join Date
    Jun 2016
    Location
    Earth, I think
    Posts
    13
    Thanks
    9
    Thanked 1 Time in 1 Post
    It seems to work fine now, but for me it fails when trying to use -t saying:
    Use of uninitialized value in addition (+) at jpegultrascan.pl line 293.

  30. #25
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    47
    Thanks
    5
    Thanked 22 Times in 14 Posts
    Going to need more info than that...

  31. #26
    Member
    Join Date
    Jun 2016
    Location
    Earth, I think
    Posts
    13
    Thanks
    9
    Thanked 1 Time in 1 Post
    Windows 7 64bit
    Portable Strawberry Perl (64-bit) 5.24.0.1-64bit

    This command line works:
    >perl jpegultrascan.pl 1.jpg a1.jpg

    This one does not:
    >perl jpegultrascan.pl -t 4 1.jpg a1.jpg

    The coutput is:


    Calculating sizes of AC scans, plane 0
    '-' is not recognized as an internal or external command,
    operable program or batch file.
    '-' is not recognized as an internal or external command,
    operable program or batch file.
    '-' is not recognized as an internal or external command,
    operable program or batch file.
    '-' is not recognized as an internal or external command,
    operable program or batch file.


    Searching for best AC scan, plane 0
    Use of uninitialized value in addition (+) at jpegultrascan.pl line 293.
    Use of uninitialized value in addition (+) at jpegultrascan.pl line 293.
    Use of uninitialized value in addition (+) at jpegultrascan.pl line 293.
    ...
    ...
    ...

  32. #27
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    47
    Thanks
    5
    Thanked 22 Times in 14 Posts
    It turns out a couple lines were somehow missed in the last commit. Sorry about that; please try again.

  33. The Following User Says Thank You to MegaByte For This Useful Post:

    pico (3rd July 2016)

  34. #28
    Member
    Join Date
    Jun 2016
    Location
    Earth, I think
    Posts
    13
    Thanks
    9
    Thanked 1 Time in 1 Post
    Works fine now, Thanks!

  35. #29
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    47
    Thanks
    5
    Thanked 22 Times in 14 Posts
    I've created a new script, jpegdropscan, which iteratively shaves off bits and coefficients that contribute least to image fidelity. This initial version uses butteraugli as its quality evaluator. Its logic is not very optimized, so there is potential for future speed up.

    As mentioned before, since this is operating on scans, it is whole image-based, so if you wanted to be more block-based, something like guetzli is likely better. The advantage of jpegdropscan is that it does not do any recoding, so you don't lose fidelity to recoded JPEG artifacts.

    I've also updated jpegultrascan as I realized certain combinations were missed -- some files now gain an extra 0.05-0.5% improvement. Using jpegultrascan before and after jpegdropscan yields the most size savings. Both can be found in the same repo.

  36. The Following 2 Users Say Thank You to MegaByte For This Useful Post:

    Jyrki Alakuijala (28th November 2016),lorents17 (28th November 2016)

  37. #30
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    181
    Thanks
    74
    Thanked 10 Times in 10 Posts
    i would like to play with it, are there any Win7 binaries?

Page 1 of 2 12 LastLast

Similar Threads

  1. PNG in .ICO file optimizer ?
    By SvenBent in forum Data Compression
    Replies: 9
    Last Post: 21st April 2016, 17:30
  2. JPEG XT Demo software available on jpeg.org
    By thorfdbg in forum Data Compression
    Replies: 40
    Last Post: 16th September 2015, 15:30
  3. Replies: 9
    Last Post: 11th June 2015, 23:28
  4. Replies: 0
    Last Post: 6th February 2015, 06:57
  5. help : Psd optimizer
    By Yuri Grille. in forum Data Compression
    Replies: 25
    Last Post: 25th April 2009, 09:29

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •