Results 1 to 14 of 14

Thread: Compressing DNG

  1. #1
    Member
    Join Date
    Jul 2014
    Location
    india
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Compressing DNG

    Hi,
    I there any method to compress Adobe DNG file? I have a Blackmagic cinema camera which records each frame in DNG format, and each file is about 8mb. I would be gathering about 500gb of DNG data daily. Is there any compressor to work with this type of file? I tried zpaq, paq8 and others. But none have succeeded enough.

    Sorry if my english is bad.

    Regards

    mohan

  2. #2
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    872
    Thanks
    457
    Thanked 175 Times in 85 Posts
    There was a camera raw compressor in the past (but nor for DNG) that is discontinued.
    So far there is no special compressor for DNG, but since it uses old JPEG lossless for compression,
    there might be huge gain of up to 50% per file.

  3. #3
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    437
    Thanks
    1
    Thanked 96 Times in 57 Posts
    Quote Originally Posted by mohanohi View Post
    Hi,
    I there any method to compress Adobe DNG file?
    Not exactly. DNG is a rather mixed bag of various "raw" pixel representations (i.e. a container format rather than a pixel representation), so it's fairly obvious that no "generic" method can be recommended. It always depends on how the pixels are ordered. Once that is understood, all the known lossy or lossless compression methods can be applied to the raw samples.

  4. #4
    Member
    Join Date
    Jul 2013
    Location
    Germany
    Posts
    24
    Thanks
    4
    Thanked 4 Times in 4 Posts
    Can you upload a single frame of your video as DNG? Possible size-reduction depends on the current format and on the software you want to use the compressed version with.

    There was a tool for lossless compression of RAW-files, called Rawzor. It is not supported anymore and runs only, if you set your system time back. Right now I cannot open the homepage, maybe it is down. It compresses better than generic compressors and its way of ordering and preprocessing the files, combined with newer compression methods, might be even better. Some RAW-converters for images support this format. I don't know about video-software.

    I would try Adobe Digital Negative Converter first. You can get it for free from Adobe. The big advantage is, that the compressed DNGs can still be opened without decompressing them first by most DNG-compatible software. Lossless compression should be supported by almost any software, capable of importing DNGs. The newer, lossy RAW-compression is not well supported yet, but I would try it, becaus size-reduction can be huge and its disadvantages may be small in video.

    http://www.adobe.com/support/downloa...atform=Windows
    http://www.rawzor.com/
    https://en.wikipedia.org/wiki/Rawzor

    P.S.
    The last time I used DNG-converter to bulk-convert hundrets of RAWs, it used only one of my two processor-cores. If you want to convert 500 GB per day, you definitly want to use as many cores as possible. If DNG-Converter does not process in parallel, you might want to combine ppx2.exe and the command-line-interface of DNG-Converter to use all cores available.

    http://wwwimages.adobe.com/content/d...ommandline.pdf
    http://www.pirosa.co.uk/demo/wxargs/wxargs.html
    http://encode.ru/threads/1978-PngBes...ll=1#post38792
    http://encode.ru/threads/1948-StartM...ll=1#post38024

  5. #5
    Member
    Join Date
    Jul 2014
    Location
    india
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts
    This is the original RAW file from my Blackmagic 4k Camera which records in RAW DNG format.

    https://www.dropbox.com/s/ejilvwlqq711j3a/BMCDNG.dng

  6. #6
    Member
    Join Date
    Jul 2013
    Location
    Germany
    Posts
    24
    Thanks
    4
    Thanked 4 Times in 4 Posts
    Adobe Digital Negative Converter works with your file.

    Looks like your camera applies very good lossless compression, because lossless recompression makes the file slightly bigger.

    Uncompressing the DNG results in a 17.8 MiB DNG-file.

    Lossy DNG compression reduces filesize from 8.8 MiB to 3.4 MiB.

    While I do not like lossy DNG-compression in my still-images, which I tweak a lot to squeeze the tone-values that I want out of the RAW-Files, it should be ok for videos that are exposed correctly and that are not hand-tweaked for dynamic-range and smooth tonal-transition. I would try that, if your video-software can use lossy DNG.
    Last edited by Sebastian W; 3rd July 2014 at 16:56.

  7. #7
    Member
    Join Date
    Jul 2014
    Location
    india
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hi Sebastian,
    Thank you for your reply.
    This camera is used for film making purpose. Hence they need to control every aspect of image manipulation at post. Lossy dng isn't preffered. Then there isn't a way to compress more right?

  8. #8
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    872
    Thanks
    457
    Thanked 175 Times in 85 Posts
    no.. not at present..

  9. #9
    Member Jaff's Avatar
    Join Date
    Oct 2012
    Location
    Dracula's country
    Posts
    100
    Thanks
    112
    Thanked 20 Times in 16 Posts
    Quote Originally Posted by Sebastian W View Post
    Looks like your camera applies very good lossless compression, because lossless recompression makes the file slightly bigger.
    Uncompressing the DNG results in a 17.8 MiB DNG-file.
    IrfanView shows that no compression is used in that DNG file... nothing to be uncompressed.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	Clipboard01.png 
Views:	218 
Size:	6.9 KB 
ID:	3048  

  10. #10
    Member
    Join Date
    Jul 2013
    Location
    Germany
    Posts
    24
    Thanks
    4
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by Jaff View Post
    IrfanView shows that no compression is used in that DNG file... nothing to be uncompressed.
    Better do not trust IrfanView this time. IrfanView says "Compression: none" for any DNG-file, no matter if it is uncompressed, losslessy compressed or lossy compressed.

    The pixel-size is wrong too and the number of unique colors is, by definition, not defined in a bayer-raw-file. What you see is not the information about the Raw-data. It is something different.

    I think, IrfanView is developing the RAW-data in a very simple way (maybe DCRAW half-size) and shows you the information of the developed image. This temporary image is, of cause, uncompressed and half size.

    The DNG is compressed. I checked that in many ways. Feel free to use Adobe Digital Negative Converter to check what I wrote in #6. I'm glad if someone finds the errors that I make.

  11. #11
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    A few test results:

    8,826,880 BMCDNG.dng
    8,780,800 bmcdng.dng.zpaq -method 5
    8,778,554 BMCDNG.dng.paq8pxd11 -8
    39,444,350 bmcdng.png (ImageMagick convert)
    26,514,486 bmcdng.bmp (ImageMagick convert)
    6,167,790 bmcdng.bmp.zpaq -method 5
    5,225,962 bmcdng.bmp.paq8pxd11 -8

    So the DNG file is already compressed. The conversion to BMP is probably lossy because ImageMagick identify says it is 16 bit (4032 x 2192, picture of a sunset). The PNG output is 16 bit.
    Last edited by Matt Mahoney; 9th July 2014 at 17:33.

  12. #12
    Member Jaff's Avatar
    Join Date
    Oct 2012
    Location
    Dracula's country
    Posts
    100
    Thanks
    112
    Thanked 20 Times in 16 Posts
    Click image for larger version. 

Name:	Clipboard01.png 
Views:	148 
Size:	6.9 KB 
ID:	3051

    I had the option "Load half size image (faster, using Format-Plugins)" checked :| sorry

  13. #13
    Member
    Join Date
    Jul 2013
    Location
    Germany
    Posts
    24
    Thanks
    4
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by Matt Mahoney View Post
    A few test results:

    8,826,880 BMCDNG.dng
    8,780,800 bmcdng.dng.zpaq -method 5
    8,778,554 BMCDNG.dng.paq8pxd11 -8
    39,444,350 bmcdng.png (ImageMagick convert)
    26,514,486 bmcdng.bmp (ImageMagick convert)
    6,167,790 bmcdng.bmp.zpaq -method 5
    5,225,962 bmcdng.bmp.paq8pxd11 -8
    17,797,836 BMCDNG_1.dng (uncompressed RAW from Adobe DNG Converter; lossless and fully compatible with RAW-converters)
    8,610,252 BMCDNG_1.dng.xz uncompressed, then 7-ZIP (xz-format, Ultra, LZMA2, WB: 64MB, word:273)
    7,302,484 BMCDNG_1.dng.zpaq uncompressed, then zpaq -method 5

  14. #14
    Member
    Join Date
    Jul 2013
    Location
    Germany
    Posts
    24
    Thanks
    4
    Thanked 4 Times in 4 Posts
    @Matt Mahoney: If you want to play around with uncompressed RAW data from RAW-image-files, you can extract it from almost any RAW-format, including DNG, with the open-source-tool DCRAW. Developing the RAW-file with any kind of RAW-converter, including ImageMagick, makes an image out of the RAW-file, which is something very different, changes the data a lot and throws away much information.

    To extract RAW-data exactly how it is, use dcraw64 -D -6 BMCDNG.dng. This way you get 16-bit per sensor-element of the camera, which is enough even for the most expensive photo-cameras you will meet in wild-life. Most cameras use only the first 14- or 12-bit. The resulting file is slightly smaller than an uncompressed DNG, because the header of the .pgm-format is small and simple.

    17,797,836 BMCDNG_1.dng (uncompressed DNG)
    17,676,307 BMCDNG.pgm (extracted RAW-data with dcraw)
    7,302,484 BMCDNG_1.dng.zpaq uncompressed, then zpaq -method 5
    7,134,214
    BMCDNG.pgm.zpaq extracted, then zpaq -method 5

  15. The Following User Says Thank You to Sebastian W For This Useful Post:

    Matt Mahoney (10th July 2014)

Similar Threads

  1. Compressing pi
    By Matt Mahoney in forum Data Compression
    Replies: 24
    Last Post: 26th September 2016, 19:17
  2. Compressing prime numbers
    By Matt Mahoney in forum Data Compression
    Replies: 14
    Last Post: 18th May 2013, 18:41
  3. Compressing mp3 Files
    By zhuda in forum Data Compression
    Replies: 12
    Last Post: 7th March 2011, 19:26
  4. On compressing series of ones and zeroes.
    By Chuckie in forum Data Compression
    Replies: 4
    Last Post: 3rd March 2011, 12:33
  5. compressing a really small 1k .COM file
    By Rugxulo in forum Data Compression
    Replies: 3
    Last Post: 28th November 2009, 01:32

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •