Page 1 of 3 123 LastLast
Results 1 to 30 of 64

Thread: iz: New fast lossless RGB photo compression

  1. #1
    Member cfeck's Avatar
    Join Date
    Jan 2012
    Location
    Germany
    Posts
    31
    Thanks
    0
    Thanked 0 Times in 0 Posts

    iz: New fast lossless RGB photo compression

    Hi all, I finally managed to release a first preview of "iz" (ImageZero). Here some key stats:
    • should be similar in speed as "arc a -mmm:3*8+1xb"
    • compression rate similar to PNG for "good" photos
    • BSD license
    Caveats:
    • only designed for 24 bit RGB natural photos
    • compression of artificial images or very noisy images is bad
    • no grayscale mode
    • currently writes in machine-depended byte order, so files are not portable (they lack a header for now, anyway)
    Some technical details:
    • 3-pixel predictor
    • single context value
    • static Huffman tables (1 ... 6 bits per RGB pixel to encode context delta)
    • branchless design (there is only one branch for the per-pixel loop, and one branch in the bitcoder to flush bitcache to memory)
    More information:
    I am very interested in any kind of feedback, since I am new to this forum. Thanks, Christoph

  2. #2
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,422
    Thanks
    6
    Thanked 26 Times in 20 Posts
    Decompression speed?

  3. #3
    Member cfeck's Avatar
    Join Date
    Jan 2012
    Location
    Germany
    Posts
    31
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by m^2 View Post
    Decompression speed?
    On my system, decompression is 2.2x faster compared to PNG (comparing process time, since the disk speed is a limiting factor here...)

  4. #4
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    459
    Thanks
    19
    Thanked 6 Times in 6 Posts
    Compared to PNG, it compresses 24 bit PPM files about 20 times faster, and decompresses to PPM about 2 times faster, while the compression rate is nearly as good as PNG for natural photos.
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  5. #5
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    459
    Thanks
    19
    Thanked 6 Times in 6 Posts
    <Sorry, doublepost>
    Last edited by Black_Fox; 24th January 2012 at 12:10.
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  6. #6
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    556
    Thanks
    46
    Thanked 37 Times in 22 Posts
    Can anyone please provide a compiled executable?

  7. #7
    Member cfeck's Avatar
    Join Date
    Jan 2012
    Location
    Germany
    Posts
    31
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Stephan Busch View Post
    Can anyone please provide a compiled executable?
    This is the binary I use: http://skulpture.maxiom.de/playground/iz (Linux glibc2.0, x86-32, MMX) Anyone who creates a Linux x86-64 or Windows binary should make sure the compilation options are optimal
    Last edited by cfeck; 24th January 2012 at 15:41. Reason: Fixed URL

  8. #8
    Member cfeck's Avatar
    Join Date
    Jan 2012
    Location
    Germany
    Posts
    31
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Some updates:

    First (thanks to Alex for sending the missing file), I could run the complete LPCB test.

    The "score" for IZ is 1,439,922,512 bytes (=148,14% of gralic111), so it is even slightly better than regular PNG in compression rate, despite the poor results for some PIA* images. In the four camera sets, IZ files are about 5% smaller than PNG; it even reaches JPEG-LS compression rate for the Olympus_xz1 set. Full results at http://skulpture.maxiom.de/playground/list-iz.txt

    LPCB timings (only measuring processor time, since IZ saturates the disk):

    method:encode/decode/bitrate
    IZ: 91,7 s/94,4 s/3.327 bpc
    PNG: 3035,0 s/191,5 s/3.401 bpc
    FLIC: 656,8 s/682,4 s/2.399 bpc

    Additionally, I compiled 32/64 bit RPM packages for Fedora/openSUSE/Mandriva on openSUSE build service, you can download ImageZero from http://download.opensuse.org/repositories/home:/cfeck/

    In other words, if you want to reproduce on your system, we still need someone who can make Windows binaries :-/
    Last edited by cfeck; 26th January 2012 at 04:14. Reason: Removed table crap...

  9. #9
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    459
    Thanks
    19
    Thanked 6 Times in 6 Posts
    Some quick'n'dirty attempt for Win compile:

    Code:
    imagezero-imagezero>g++ -O2 -mmmx -Winline -fomit-frame-pointer -fno-rtti -fno-exceptions main.cpp encode.cpp decode.cpp portableimage.cpp table.cpp -o iz
    portableimage.cpp:3:22: fatal error: sys/mman.h: No such file or directory
    compilation terminated.
    after adding http://code.google.com/p/mman-win32/...e/trunk/mman.h to mingw/include/sys:
    Code:
    Temp\cc0BzDV8.o:portableimage.cpp:(.text+0x1fa): undefined reference to 'munmap'
    Temp\cc0BzDV8.o:portableimage.cpp:(.text+0x250): undefined reference to 'mmap'
    collect2: ld returned 1 exit status
    Not sure what to do, had just a few minutes of time.
    Last edited by Black_Fox; 26th January 2012 at 18:15.
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  10. #10
    Member cfeck's Avatar
    Join Date
    Jan 2012
    Location
    Germany
    Posts
    31
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Black_Fox View Post
    Not sure what to do, had just a few minutes of time.
    Oh, if mingw does not provide "mmap", please change both "#if 1" to "#if 0" in portableimage.cpp. On my system, using "read" instead of "mmap" was slower, though.

  11. #11
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    459
    Thanks
    19
    Thanked 6 Times in 6 Posts
    That helped, thanks. Win32 x86 compile for those interested

    gcc version 4.5.1 (tdm-1)
    Attached Files Attached Files
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  12. #12
    Member Karhunen's Avatar
    Join Date
    Dec 2011
    Location
    USA
    Posts
    86
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Problems using that iz.exe.zip

    I can't seem to ge a PBM/PNM binary encoded I get "can only compress 24 bit binary PPM files" here is an attachment I triedClick image for larger version. 

Name:	r1_sixteenth.png 
Views:	235 
Size:	57.4 KB 
ID:	1828and GBMtools
    says
    #>gbmhdr RGB_OR_1200x1200_082.ppm
    1200x1200 24bpp 4219Kb 100% Pixmap RGB_OR_1200x1200_082.ppm
    imagemagick identify says same thing, and Image Alchemy says (affter I renamed it, its a DOS program) that
    Reading Portable BitMap file R1.PNM
    File Name: R1.PNM
    Width x Height: 1200 x 1200
    Number of Colours: True Colour (24 bits)
    Raw size: 4320000 Actual size: 4320017 (Compression ratio: 1:1)
    Attached Files Attached Files

  13. #13
    Member cfeck's Avatar
    Join Date
    Jan 2012
    Location
    Germany
    Posts
    31
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Karhunen View Post
    I can't seem to ge a PBM/PNM binary encoded I get "can only compress 24 bit binary PPM files"
    Got the same error message when trying the binary with "wine" under Linux, but I blamed wine for this.

    To me, it looks like mingw does not implement the UNIX file operation functions correctly, so they should be ported to Windows native calls. Since I do not have Windows, I can only hope someone else can look into the issue. Using stdio calls could be used as a last resort, but they are so slow that I avoided them.

    Amazing that nobody reported this earlier, when the binary was made available...

    BTW, the image file you attached is fine, it compresses to 2670480 bytes with IZ.
    Last edited by cfeck; 1st February 2012 at 16:59. Reason: Added BTW ...

  14. #14
    Member Karhunen's Avatar
    Join Date
    Dec 2011
    Location
    USA
    Posts
    86
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Thanks Chris, if anyone is successful with Borland, Mars, or VC and could post a binary for it, the semantically challenged me would be grateful

  15. #15
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    556
    Thanks
    46
    Thanked 37 Times in 22 Posts

    Hi there

    I am getting the same error with the win32 build - no matter if the .ppm were created using PictView, XnView nor bmp2ppm.

  16. #16
    Member
    Join Date
    Feb 2010
    Location
    Nordic
    Posts
    180
    Thanks
    3
    Thanked 3 Times in 1 Post
    A good candidate might be: https://gitorious.org/imagezero/imag...ge.cpp#line130

    On Linux, generally, ::read() returns all bytes in one go when accessing local files; on windows, it returns bits at a time; you have to call ::read() in a loop until it returns 0 or -1.

    This is also true on Linux for reading from NFS, for example; so its the right code to have.

  17. #17
    Member cfeck's Avatar
    Join Date
    Jan 2012
    Location
    Germany
    Posts
    31
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thanks willvarfar for the hint.

    The master branch on gitorious has been updated to include some fixes that could affect file reading. Also, the error messages have been improved, so that we can better see which call fails.

    Black_Fox (or anyone else), could you offer an updated binary? If I didn't mess it, it should detect mingw, and automatically switch to using read instead of mmap, so no source changes should be required.

  18. #18
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    459
    Thanks
    19
    Thanked 6 Times in 6 Posts
    Compilation is without problems, can someone check correct functioning on sample data?
    Attached Files Attached Files
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  19. #19
    Member cfeck's Avatar
    Join Date
    Jan 2012
    Location
    Germany
    Posts
    31
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thanks Black_Fox.

    Still does not work under "wine", so let's assume it does not work on Windows either :-/ It says "fixme:msvcrt:MSVCRT__sopen_s : pmode 0x01a4 ignored", then hangs.

  20. #20
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,030
    Thanks
    11
    Thanked 33 Times in 27 Posts
    I suspect there's a problem with Wine. Why do you assume that Wine is more compatible with Windows programs than Windows itself? "fixme:msvcrt" - looks for me like they haven't reimplemented something from msvcrt library.

  21. #21
    Member cfeck's Avatar
    Join Date
    Jan 2012
    Location
    Germany
    Posts
    31
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I assumed mingw cannot correctly convert all POSIX calls (fstat etc.) to the Windows API... but you are right, that error message is actually from wine, so it might work in Windows.

  22. #22
    Member Karhunen's Avatar
    Join Date
    Dec 2011
    Location
    USA
    Posts
    86
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by cfeck View Post
    I assumed mingw cannot correctly convert all POSIX calls (fstat etc.) to the Windows API... but you are right, that error message is actually from wine, so it might work in Windows.
    I couldn't get it to process even a 75x75x24 image, hope this screenshot of running under Windoze 7 helps
    tie down the area of troubleClick image for larger version. 

Name:	prop.png 
Views:	236 
Size:	48.9 KB 
ID:	1837

  23. #23
    Member cfeck's Avatar
    Join Date
    Jan 2012
    Location
    Germany
    Posts
    31
    Thanks
    0
    Thanked 0 Times in 0 Posts
    As expected, so no Windows binary.

    Anyway, the source code is not really intended to be an application, but to be later embedded as a library into image processing or database applications. I think it is safe to assume that a developer is able to add the Windows file reading, if he does not have in-memory data.

    If there is interest, I can post an update when the API is ready for review.

  24. #24
    Member
    Join Date
    Feb 2010
    Location
    Nordic
    Posts
    180
    Thanks
    3
    Thanked 3 Times in 1 Post
    Any thoughts on compressing 3D texture model maps e.g. DXT?

  25. #25
    Member cfeck's Avatar
    Join Date
    Jan 2012
    Location
    Germany
    Posts
    31
    Thanks
    0
    Thanked 0 Times in 0 Posts
    The source code for IZ on gitorious has been changed to use standard C++ file I/O calls instead of POSIX calls, so it should hopefully now work with MinGW. I am still looking for someone contributing Visual Studio and/or Windows file mapping support, though.

    To compile:

    Code:
    g++ -march=corei7 -Werror=inline -fomit-frame-pointer -fno-rtti -fno-exceptions -O3 *.cpp -o iz
    Replace "corei7" with your architecture, see http://gcc.gnu.org/onlinedocs/gcc/i3...4-Options.html


  26. #26
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    3,503
    Thanks
    122
    Thanked 156 Times in 85 Posts

  27. #27
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    556
    Thanks
    46
    Thanked 37 Times in 22 Posts
    can anybody compile it for me please?

    Thanks in advance

  28. #28
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    2,061
    Thanks
    0
    Thanked 5 Times in 4 Posts

  29. #29
    Member cfeck's Avatar
    Join Date
    Jan 2012
    Location
    Germany
    Posts
    31
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thanks Shelwien, but please use the flags I specified above.

    You did not compile with optimization enabled, which makes the binary useless, because the way the code is written function inlining absolutely must be enabled. Also enable at least -mmmx for 32 bit platform.

  30. #30
    Member cfeck's Avatar
    Join Date
    Jan 2012
    Location
    Germany
    Posts
    31
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    Merci, that looks indeed helpful

Page 1 of 3 123 LastLast

Similar Threads

  1. Unknown Moscow - Photo Gallery
    By encode in forum Off-Topic
    Replies: 17
    Last Post: 23rd October 2013, 15:41
  2. FLIC - a new fast lossless image compressor
    By Alexander Rhatushnyak in forum Data Compression
    Replies: 25
    Last Post: 10th January 2013, 20:46
  3. Comparison of lossless PNG compression tools
    By Surfer in forum Data Compression
    Replies: 54
    Last Post: 19th September 2011, 23:58
  4. lossless data compression
    By SLS in forum Data Compression
    Replies: 21
    Last Post: 15th March 2011, 12:35
  5. Replies: 13
    Last Post: 3rd April 2010, 00:46

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •