Page 1 of 3 123 LastLast
Results 1 to 30 of 73

Thread: PIK image format

  1. #1
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts

    PIK image format

    We opensourced today an early version of a new image codec. It is a 8x8 DCT-based codec, i.e., compatible with JPEG, boosted with a more efficient color space, advanced coding, and a plenty of guetzli/butteraugli like tricks. It decodes around 2/3 of jpeg speed, but the encoding is ludicrously slow. We can reach the same butteraugli scores with 54 % less bytes using PIK than what libjpeg gives us. PIK is both faster to decode and compresses quite a lot better than a guetzlified jpegs compressed with PackJPG/Lepton.

    The code is available at https://github.com/google/pik

  2. The Following 8 Users Say Thank You to Jyrki Alakuijala For This Useful Post:

    boxerab (7th September 2017),Bulat Ziganshin (25th July 2017),comp1 (24th July 2017),Cyan (25th July 2017),inikep (24th July 2017),JamesWasil (26th January 2018),khavish (31st January 2018),load (25th July 2017)

  3. #2
    Member jibz's Avatar
    Join Date
    Jan 2015
    Location
    Denmark
    Posts
    114
    Thanks
    91
    Thanked 69 Times in 49 Posts
    Just an FYI, in (at least) Danish, the name is a term for the male sexual organ.

  4. The Following 2 Users Say Thank You to jibz For This Useful Post:

    Stephan Busch (20th June 2018),SvenBent (8th August 2017)

  5. #3
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    174
    Thanked 51 Times in 37 Posts
    Quote Originally Posted by jibz View Post
    Just an FYI, in (at least) Danish, the name is a term for the male sexual organ.
    ... Reminds me of the first reaction on the ZSTD thread...

  6. #4
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    181
    Thanks
    74
    Thanked 10 Times in 10 Posts
    where can i find exe files for windows?

  7. #5
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    539
    Thanks
    192
    Thanked 174 Times in 81 Posts
    In the "Related projects" section, a lossless JPEG repacker named "Brunsli" is mentioned. Never heard of it and couldn't find anything on the web. Was it abandoned or just never/not yet released? Or is it just an insider joke to fulfill Crush's prophecy?
    http://schnaader.info
    Damn kids. They're all alike.

  8. #6
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by schnaader View Post
    In the "Related projects" section, a lossless JPEG repacker named "Brunsli" is mentioned. Never heard of it and couldn't find anything on the web. Was it abandoned or just never/not yet released? Or is it just an insider joke to fulfill Crush's prophecy?
    Yes, brunsli is a not-yet-released project. It is like Lepton, but compresses 2 % worse and compression and decompression speeds are 150 % better. Brunsli is somewhat slower to decode than PIK. Normal jpegs compress about 20-22 % with brunsli, and guetzlified jpegs compress further only 14 %.

  9. #7
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    437
    Thanks
    1
    Thanked 96 Times in 57 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    We opensourced today an early version of a new image codec. It is a 8x8 DCT-based codec, i.e., compatible with JPEG, boosted with a more efficient color space, advanced coding, and a plenty of guetzli/butteraugli like tricks. It decodes around 2/3 of jpeg speed, but the encoding is ludicrously slow. We can reach the same butteraugli scores with 54 % less bytes using PIK than what libjpeg gives us. PIK is both faster to decode and compresses quite a lot better than a guetzlified jpegs compressed with PackJPG/Lepton.
    Question on that: Does "compatible to JPEG" mean "compatible" or, "just using the same DCT?". Given that you seem to use a different component decorrelation transform (the typical YCbCr transform is not exactly a color space transformation), I would guess the latter?

  10. The Following User Says Thank You to thorfdbg For This Useful Post:

    Jyrki Alakuijala (7th August 2017)

  11. #8
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by thorfdbg View Post
    Question on that: Does "compatible to JPEG" mean "compatible" or, "just using the same DCT?"
    Good question. Not really compatible. Compatible in the sense that the main artefacts are roughly the same, i.e., cross-encoding both ways doesn't lead to heavy losses.

  12. #9
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by thorfdbg View Post
    decorrelation transform (the typical YCbCr transform is not exactly a color space transformation)
    The color space in Pik is LMS (with a constant model for spontaneous isomerization), and it is less there for decorrelation and more for making the optimal psychovisual loss. The L and M are modeled however with opponent colors -- there is L + M and L - M axis. The S colors have their own modeling with lower spatial frequency and some decorrelation with the L + M.
    Last edited by Jyrki Alakuijala; 8th August 2017 at 01:09.

  13. #10
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    856
    Thanks
    45
    Thanked 104 Times in 82 Posts
    Quote Originally Posted by jibz View Post
    Just an FYI, in (at least) Danish, the name is a term for the male sexual organ.
    just came in to post the same

  14. #11
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by SvenBent View Post
    just came in to post the same
    How do you deal with Pikatschu's "pika pika", pickpocketing, lock pick, Pic, Water Pik, Picasa, Pick axe, Pik loans, Pictures and spades in German. Constant giggling or you learned to deal with it?

    If we add an alternative spelling of Pic for those languages where Pik is way too funny -- would it solve the issue? (or perhaps bld for bild/billade/... or some other abbreviation?)

  15. #12
    Member jibz's Avatar
    Join Date
    Jan 2015
    Location
    Denmark
    Posts
    114
    Thanks
    91
    Thanked 69 Times in 49 Posts
    I guess it's a matter of context. If English is your second language you wouldn't make any associations automatically.

    How do native English speakers deal with cocktail, cocky, cockney, etc? But if you see the word on its own ...

  16. #13
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by jibz View Post
    I guess it's a matter of context. If English is your second language you wouldn't make any associations automatically.

    How do native English speakers deal with cocktail, cocky, cockney, etc? But if you see the word on its own ...
    A native English speaker first resolves if it is a verb or a noun. If noun, they decide from context if it is a male bird or a slang word for penis. Surely the Danish can learn to do similar context modeling.

  17. #14
    Member jibz's Avatar
    Join Date
    Jan 2015
    Location
    Denmark
    Posts
    114
    Thanks
    91
    Thanked 69 Times in 49 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Surely the Danish can learn to do similar context modeling.
    I apologize, I chose a bad example then - pik has no other meanings in Danish.

    Anyway, I am sorry, but you seem somewhat confrontational about it, so I will stop arguing here.

  18. The Following User Says Thank You to jibz For This Useful Post:

    SvenBent (11th August 2017)

  19. #15
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by jibz View Post
    I apologize, I chose a bad example then - pik has no other meanings in Danish.

    Anyway, I am sorry, but you seem somewhat confrontational about it, so I will stop arguing here.
    We will resolve this problem later with help from the public and experts. Now it is time to first hone the algorithms.

    Did you try it already?

  20. #16
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    202
    Thanks
    165
    Thanked 109 Times in 65 Posts
    dpik.exe always immediately crashes here (appcrash) under win7.

    i compiled it using mingw gcc 6.2 and gcc 5.4, also POSIX, always the same result...

  21. #17
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by load View Post
    dpik.exe always immediately crashes here (appcrash) under win7.

    i compiled it using mingw gcc 6.2 and gcc 5.4, also POSIX, always the same result...
    Thanks for reporting -- this is very useful. We try to fix it early coming week.
    Last edited by Jyrki Alakuijala; 2nd October 2017 at 00:03.

  22. #18
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    202
    Thanks
    165
    Thanked 109 Times in 65 Posts
    hm ok, i just had a closer look at this, we need AVX2 for this. that might be the reason it crashes.
    but a proper error message instead wouldn't hurt.

  23. #19
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by load View Post
    hm ok, i just had a closer look at this, we need AVX2 for this. that might be the reason it crashes.
    but a proper error message instead wouldn't hurt.
    Please add an issue to github about this?

    We will soon (likely this week) release a SSE 4.1 version that fixes the AVX2 compatibility problem.

  24. #20
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    222
    Thanks
    89
    Thanked 46 Times in 30 Posts
    Nice work Jyrki. When you say it takes 54% fewer bytes than libjpeg, what are the input images? RAW files? Existing JPEGs? If they're already JPEGs, what can libjpeg do to them other than lower the quality rating?

    The state-of-the-art benchmark for lossy JPEG compression right now is JPEGmini.com, and possibly TinyJPG.com as well. JPEGmini is patented and proprietary (I think they published a paper on their breakthrough), and they claim 70-80% reduction of large JPEGs (>8MP, that's megapixels, not bytes). They both offer limited free use via their websites, and paid plans for larger scale users.

    Can PIK be an acquisition format? To finally move on past JPEG, we need a format that can serve as an acquisition format, in the camera. That's where JPEGs are born (and JPEG-XR was initially pitched as an end-to-end format that included acquisition and in-camera encoding). I believe that cameras, whether in smartphones or pure cameras, have a specialized chip for encoding photos into JPEGs. I assume the encoding would be much slower if it were assigned to a software encoder on a CPU. So a full replacement for JPEG should probably anticipate having hardware encoders in the camera. I have no idea how that reality would influence the design of the image format, but it's something to think about.

    It would also help to not only "support" HDR, but to make it easier for cameras to implement HDR, perhaps in how HDRs are encoded.

    An efficient binary metadata format would also help, one that enables people to markup regions and people the way they do on Facebook, along with other useful semantics. Some metadata might actually help with the encoding. There are lots of ways I can imagine that. One example: for outdoor shots, marking the exact time and GPS location within a few meters should be able to tell us some things about the light exogenous to image encoding (from weather data). And there might be some ways for a camera or computer to use that GPS, time, and camera orientation/direction info to help make some decisions about the true colors in various parts of the image, or to deal with certain kinds of artifacts, glare, and a few other things...

  25. The Following User Says Thank You to SolidComp For This Useful Post:

    Jyrki Alakuijala (2nd November 2017)

  26. #21
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    202
    Thanks
    165
    Thanked 109 Times in 65 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Please add an issue to github about this?

    We will soon (likely this week) release a SSE 4.1 version that fixes the AVX2 compatibility problem.
    it does work now, it's very slow though, unless i use --fast. it works several minutes on a very small png, but i suppose this is the expected behaviour?

  27. The Following User Says Thank You to load For This Useful Post:

    Jyrki Alakuijala (2nd November 2017)

  28. #22
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by load View Post
    it does work now, it's very slow though, unless i use --fast. it works several minutes on a very small png, but i suppose this is the expected behaviour?
    Yes. It is indeed very slow (inspired by guetzli) in the default mode.

    The results should be worth it at least at butteraugli score < 1.5. At butteraugli 3.0+ it is just slow and bad. We will improve on this during the next half a year.

    Decoding is relatively fast (although we have temporarily slowed things down for some more flexibility in the code) for both the slow and fast mode encoding.

  29. #23
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by SolidComp View Post
    Nice work Jyrki. When you say it takes 54% fewer bytes than libjpeg, what are the input images? RAW files? Existing JPEGs?
    We are currently about 65 % smaller than libjpeg at butteraugli score 1.0. We compare against libjpeg.

    No jpeg optimizer (including guetzli) can get anywhere close to PIK in density/quality -- not even if a format incompatible recompressor such as PackJPG is used.

    PIK's design has been inspired by guetzli and our own attempt at high speed jpeg recompression. We just added adaptive quantization, a better colorspace, some more possibilities to separate decorrelation and color quantization, a frequency lifting prediction system (dc to low freq ac, lower freq ac to higher freq...), and quite a few other smaller details.
    Last edited by Jyrki Alakuijala; 3rd November 2017 at 02:44.

  30. The Following User Says Thank You to Jyrki Alakuijala For This Useful Post:

    boxerab (3rd November 2017)

  31. #24
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by SolidComp View Post
    Can PIK be an acquisition format?
    PIK allows for simpler ways to do encoding, and should be a reasonable fit for HW encoding. The --fast mode encodes about 45 MB/s on a fast CPU and it still does complicated stuff that some HW encoders don't do (dynamic entropy coding, and even including primitive adaptive quantization).

  32. #25
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    48
    Thanks
    25
    Thanked 10 Times in 7 Posts
    Is support of alpha channel and metadata planed in PIK?
    An other nice to have feature: Support of (3D) stereoscopic images.

  33. #26
    Member JamesWasil's Avatar
    Join Date
    Dec 2017
    Location
    Arizona
    Posts
    33
    Thanks
    29
    Thanked 6 Times in 6 Posts
    As a suggestion for possible revised nomenclature, have you considered renaming it to p1K instead of PIK? That way Danish users will see it more as p-one-k rather than a pik. Just a suggestion.

  34. The Following User Says Thank You to JamesWasil For This Useful Post:

    Jyrki Alakuijala (24th January 2018)

  35. #27
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    48
    Thanks
    25
    Thanked 10 Times in 7 Posts
    What about using PKI instead of PIK, shortend form of PIK image?
    PKI is also a file extension which is not in use by any other program.

    btw. Is there a need to limit the file extension to 3 characters nowadays?

  36. The Following User Says Thank You to WinnieW For This Useful Post:

    Jyrki Alakuijala (24th January 2018)

  37. #28
    Member
    Join Date
    Feb 2015
    Location
    United Kingdom
    Posts
    154
    Thanks
    20
    Thanked 66 Times in 37 Posts
    3 characters is the minimum a word can be made out of phonemes in English, like zip or gif. It becomes pretty much impossible to create a unique and distinguishable word using just 2 letters. Unless you want an acronym for a file extension. 3 letters is just the most convenient.

    I think PKI is a decent alternative though.

  38. The Following User Says Thank You to Lucas For This Useful Post:

    Jyrki Alakuijala (24th January 2018)

  39. #29
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    48
    Thanks
    25
    Thanked 10 Times in 7 Posts
    Quote Originally Posted by Lucas View Post
    3 letters is just the most convenient.
    Yes, but there are also a few exceptions, e.g. flac, flif, webm and webp (4 character file extensions in practical use)

    Quote Originally Posted by Lucas View Post
    I think PKI is a decent alternative though.

  40. The Following User Says Thank You to WinnieW For This Useful Post:

    Jyrki Alakuijala (24th January 2018)

  41. #30
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    667
    Thanks
    204
    Thanked 241 Times in 146 Posts
    Quote Originally Posted by WinnieW View Post
    Is support of alpha channel and metadata planed in PIK?
    An other nice to have feature: Support of (3D) stereoscopic images.
    Alpha channel is already supported.

    There is no metadata/3D support yet.

  42. The Following User Says Thank You to Jyrki Alakuijala For This Useful Post:

    WinnieW (24th January 2018)

Page 1 of 3 123 LastLast

Similar Threads

  1. FLIF - Free Lossless Image Format
    By willvarfar in forum Data Compression
    Replies: 173
    Last Post: 28th February 2019, 19:04
  2. The old pack format
    By willvarfar in forum Data Compression
    Replies: 1
    Last Post: 23rd December 2015, 17:41
  3. Replies: 0
    Last Post: 3rd December 2015, 14:44
  4. StuffIt X Format
    By maadjordan in forum Data Compression
    Replies: 19
    Last Post: 9th August 2008, 13:03
  5. New archive format
    By Matt Mahoney in forum Forum Archive
    Replies: 9
    Last Post: 25th December 2007, 12:22

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •