Results 1 to 3 of 3

Thread: Observations on packJPG and the developers package

  1. #1
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts

    Observations on packJPG and the developers package

    I did some experiments using packJPG and the JPEG Developers Package from Matthias Stirner's new GitHub repository. At first, I had a look at uncmpJPG because I wanted to see if packJPG could be replaced with it in Precomp, because just uncompressing without the arithmetic coding suits the "Precomp way" better. Then, I compressed its output and also had a look at lpaqJPGtest.

    But first, let's have a look at the bare results. Test file is a typical camera JPG including a thumbnail from a Precomp thread (direct link to the image at MediaFire). Test PC was a Intel i5 (M 520, 2.4 GHz, 2 physical cores).

    Code:
    Original: 4,307,553 bytes
      packjpg 2.5j: 5.12 s, 3,415,122 bytes, decomp 5.23 s
      Precomp v0.4.4: 4,196,559 bytes, 2/2 JPG
      lpaqjpgtest: 117 s, 3,822,763 bytes
      paq8p -3: 134 s, 3,113,452 bytes (finds 2 JPEGs: 4608x3456, 1440x1080)
      paq8p -4: 134 s, 3,098,874 bytes (finds 2 JPEGs: 4608x3456, 1440x1080)
      Precomp v0.4.4 -cn -i13486: 4,198,382 bytes, 1/1 JPG (only process the second JPG to "hide" it from paq8p)
        paq8p -4: 3,117,069 bytes (finds the first JPEG: 4608x3456)
      uncmpJPG : 0.9 s, 64,224,983 bytes, recompression: 0.9 s
        Precomp v0.4.4: 14.43 s, 4,368,889 bytes, 2/2 JPG
        7-Zip Ultra LZMA2: 21 s, 4,346,381 bytes
        zpaq v7.05 -method 4: 16.13 s, 3,962,237 bytes
        zpaq v7.05 -method 5: 127.6 s, 3,753,728 bytes
        paq8p -3: 736.66 s, 3,717,779 bytes
        paq8p -4: 4075.35 s, 3,709,691 bytes
        SREP 2.991: 21,913,027 bytes
          7-Zip Ultra LZMA2: 20 s, 4,368,115 bytes
          zpaq v7.05 -method 5: 75.65 s, 3,774,036 bytes
          paq8p -3: 251.77 s, 3,727,443
    Some conclusions follow. I assumed that unpackJPG is close to "packJPG without the arithmetic coding part", but haven't verified it in depth, so don't trust me here

    Unspecific conclusions:
    • packJPG does a very good job at giving context to the uncompressed data while retaining a decent speed. Without knowing the context, even paq8p -4 and zpaq -method 5 don't come close to the 3.4 MB that packJPG gives and they take much more time, even after reducing the data with SREP.
    • uncmpJPG is around 5 times faster than packJPG, so the arithmetic coding takes (at least) 80% of the time.
    • lpaqJPGtest is interesting for developers, but not for users.
    • The JPG model in PAQ is still better giving the smallest result here (3.1 MB), but has other disadvantages (speed, no support for progressive JPGs).
    • Using uncmpJPG in Precomp won't pay off without further modifications to support following compression.
    • uncmpJPG could help when processing multiple JPGs with similarities across them, but this has to be tested.


    File specific conclusions:
    • Precomp doesn't detect the length of the first JPG correct, but succeeds in passing the second one to packJPG.
    • Both packJPG and uncmpJPG treat the second JPG as "garbage following EOI".
    • "Precomp -i | paq8p -4" shows that paq8p doesn't have a big advantage although it detects both streams, although it's very likely that the second JPEG is a smaller version of the first one (could someone check this?).
    • uncmpJPG only wraps a header around the garbage part, so the second JPG can be processed, as seen in the "uncmpJPG | Precomp" result. The header of the first JPG is also detected again.


    The write_ujpg method in uncmpJPG gives a good starting point at what the uncompressed file consists of:

    • "HDR" part (header), 30,044 bytes here
    • 3 "CMP" parts (decompressed DCT coefficients), 63,700,992 bytes here
    • PAD (padding) part, only 4 bytes here
    • GRB (garbage) part, 493,914 bytes here


    The first CMP part consists of 64 runs of ( 4608/8 ) * ( 3456/8 ) = 248,832 16-bit values, the following two are only half the size. Signed 16-bit values are bad for most compressors, so rearranging and preprocessing them could help a lot. The range also decreases with increasing file position, as one would expect from JPG compression.
    Last edited by schnaader; 2nd March 2016 at 20:13.
    http://schnaader.info
    Damn kids. They're all alike.

  2. The Following User Says Thank You to schnaader For This Useful Post:

    Bulat Ziganshin (19th February 2016)

  3. #2
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    I just had a look at the other programs in the JPEG Developers Package binaries and extrJPG was very useful: It generates three JPG files contained in the original file - the first one is the big image, 4608*3456 at offset 0, the second one is a very small 160*120 thumbnail at offset 0x34AE - embedded in the first one - and the third one is a 1440*1080 thumbnail at offset 0x3A3313. So:

    • Precomp detects the second and the third image correctly, but fails with the first one
    • uncmpJPG and packJPG treat the third image as "garbage following EOI", the second one seems to be treated as “header“
    • The nested structure confuses both Precomp and extrJPG - applying extrJPG on the big 4608*3456 image extracts the 160*120 thumbnail version again.
    Last edited by schnaader; 19th February 2016 at 19:33.
    http://schnaader.info
    Damn kids. They're all alike.

  4. #3
    Member
    Join Date
    Feb 2016
    Location
    USA
    Posts
    80
    Thanks
    30
    Thanked 8 Times in 8 Posts
    Very interesting observations, especially the ones about 2nd and 3rd (smaller) images. Do they imply that packJPG can be further enhanced to process all smaller images just as the large one, applying arithmetic coding to them so the resulting size can be reduced even more, maybe beating PAQ8 with still much faster speed?

Similar Threads

  1. I need attention from freearc developers
    By rafikhan in forum Data Compression
    Replies: 6
    Last Post: 1st October 2015, 20:53
  2. Intra-Package Delta Compression
    By comp1 in forum Data Compression
    Replies: 0
    Last Post: 3rd March 2014, 20:37
  3. packJPG v2.5C3
    By packDEV in forum Data Compression
    Replies: 3
    Last Post: 22nd October 2011, 15:23
  4. a small plea for the command line compression developers
    By SvenBent in forum Data Compression
    Replies: 2
    Last Post: 14th June 2008, 02:51
  5. PAQ8o8 threading observations
    By CodeMutant in forum Forum Archive
    Replies: 15
    Last Post: 18th February 2008, 11:02

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •