Adobe offering new reasons to get DNG religion

Photography enthusiasts have seen the light when it comes to shooting raw images, but plenty of them have yet to convert to Adobe Systems' DNG format for storing those images.

But Adobe could bring some new sheep into the Digital Negative fold with abilities arriving in Adobe's Lightroom 4 software and its Photoshop CS6 cousin. Adobe isn't evangelizing heavily, but it is offering new features that could convince people that DNG is a better alternative to the profusion of proprietary raw formats that higher-end cameras produce.

Three significant improvements are coming to DNG, two for speed and one for flexibility:

• "Fast-load data," a miniature raw preview image embedded in the file that makes it faster to switch among images in Lightroom's develop module--eight times as fast, according to Tom Hogarty, principal product manager for Lightroom.

• "Tiled" DNG files divided into parts so multicore processors can read and write them faster.

• An option for "lossy" compression to dramatically reduce file size--though because data from the original is lost irretrievably, this option likely will hold appeal only in some scenarios.

"I do believe the DNG...enhancements will encourage more DNG adoption," Hogarty told CNET News. Adobe has added new raw-photo features to its software before, but this time the features apply only to DNG, meaning that those who don't convert their raw photos won't benefit.

Helping Adobe
More adoption would help Adobe--especially if it encouraged camera makers to embrace DNG natively so Adobe doesn't have to reverse-engineer proprietary formats and so photographers don't have to convert the files manually. But Adobe isn't twisting any arms, Hogarty said.

"DNG won't be heavily promoted or marketed as part of the Lightroom 4 release. I'll be working on DNG awareness in the coming year but format adoption is not conducive to overnight marketing campaigns," Hogarty said. And he won't be paying camera makers to accelerate support: "The value of the format needs to stand on its own and not require fiscal motivation."

One big factor that could help DNG's fortunes is standardization. Today, the format is published publicly and Adobe releases free tools to let software makers support it, but Adobe has been working since 2008 to standardize DNG through the International Standardization Organization (ISO). If that effort succeeds, that could encourage camera makers to embrace DNG because they wouldn't feel they're relying on a single company for a crucial part of their technology. DNG is broad enough to handle unusual image sensors such as Fujifilm X-Pro 1, which uses a 36-pixel pattern with more green pixels than the usual 4-pixel Bayer pattern in most cameras.

Cameras from Pentax, Leica, and Ricoh can take DNG-format raw photos today directly with no need for people to convert them afterward using software. If more camera makers support the format, that would likely be the single biggest driver of DNG adoption. It would resolve concerns that some photographers have about moving away from the original data from the camera, and it would remove what for some is a significant hassle of converting from proprietary raw to DNG.

"I do not convert to DNG. It takes me too long. I'll shoot thousands of photos a day, and the import process for these photos is painful enough as it is now without the conversion, which feels like it provides me little real benefit," said Thomas Hawk, a high-profile photographer.

Better performance in editing catches his attention, though, he said. So perhaps Adobe is onto something when it comes to attracting people to DNG with better performance.

What are raw and DNG?
Raw photos offer more flexibility than JPEGs when it comes to adjusting exposure, white balance, noise reduction, sharpening, and other photo attributes.

Most people rely on their cameras to convert its internal raw image to a JPEG, but raw shooters can use software from companies such as Apple, Corel, DxO Labs, Phase One, Google, and Adobe to perform the conversion themselves. That software must know how to decode the hundreds of proprietary raw file formats from higher-end cameras.

DNG is an attempt to give photographers the benefits of shooting raw without the drawbacks of the proprietary formats. Because it's an openly documented format--the most recently released specification version is DNG 1.3 (PDF)--it offers archival advantages over raw formats that may or may not stand the test of time as camera manufacturers come and go.

In its advocates' eyes, a DNG file is somewhat like a PDF when it comes to longevity. Because it's openly documented, a person doesn't need a particular program to read it--the raw decoding software that came on a CD with your camera, for instance. Twenty years from now, will your computer be able to run that program? Will it have a CD-ROM drive at all? Canon and other players likely will support their own older formats, but camera makers go out of business sometimes, and new word processors often don't support older file formats.

Of course, DNG has archival disadvantages, too. New versions of the specification can mean older software can't read newer files. And people might be leery of putting their future in Adobe's hands. Standardization could help DNG in this area by making it technologically and politically easier for operating system makers, software companies, and camera manufacturers to support.

New DNG features
One of the biggest issues with handling raw images in general is that it takes a lot of computing horsepower. That's because the computer must constantly reconstruct a visually useful image from raw data, for example through "demosaicing" to reconstruct red, green, and blue values for each pixel. (Raw images usually have only a single color value for each pixel.)

So performance boosts are welcome. Multicore tiling of DNGs gives computers "a significant speedup when reading or writing lossless compressed DNGs," said Eric Chan, a Lightroom engineer:

For example, reading a tiled Leica S2 DNG image on a 4-core MacBook Pro with SSD can be done nearly three times as fast. This is because the compression/decompression can be done in parallel across multiple cores. Normally, an untiled DNG (or TIFF) cannot be easily parallelized when compressing or decompressing, because the compression/decompression operation is sequential (you can't process later bytes in parallel because they depend on earlier bytes in the stream).

Fast-load data also helps speed up Lightroom in the develop module where photographers can edit the photos. With fast-load-enabled DNGs, Lightroom 4 essentially gets a miniature version of the raw photo that's easier to process, which means people see something truer to the edited photo than an embedded JPEG preview image might show. Again, Chan:

The difference between the fast-load data and a standard embedded preview is that the fast load data is unrendered. The fast load data is used to make the initial preview display very fast, while we are busy loading the full-res data in the background (which takes longer, naturally). So if you are zipping from image to image, we can get you on-screen display faster by first showing you the fast-load preview image.

Fast-load images are bigger--200KB or perhaps somewhat more, Hogarty said. But for the most part, tiling and fast-load previews offer some benefits without disturbing how photographers work.

Lossy DNG
The lossy DNGs are another matter altogether, though. Some people like raw photos because they record the original image exactly how the camera captured it, with no artifacts resulting from JPEG compression throwing away some data. Lossy DNGs, though, throw away data to produce a file size something like a quarter the size of a regular lossless DNG.

To do that, Adobe demosaics the raw data so that strictly speaking it's not raw anymore. More alarming to some, the original 12-,14-, or 16-bit raw data has been boiled down with the JPEG algorithm to 8 bits, meaning that there aren't as many gradations of color values.

But Adobe didn't just make an ordinary JPEG. First of all, the eight bits of data are carefully distributed according to each image with a "stretched" tone curve, Chan said in a forum post. Second, Adobe "dithers," which means it carefully adds a little noise when moving to 8-bit values, which can actually improve the appearance.

And unlike JPEGs, the lossy DNGs preserve much of the flexibility of raw, such as the ability to change white balance, keep a wide color gamut, and recover overexposed highlights, Chan said.

But while Adobe likes the lossy DNG option and believes some will enjoy its more economical storage demand, it doesn't suffer the illusion that it's for everybody.

"It's not a default nor an option for everyone," Hogarty said. "However, I know that a number of photographers shooting time lapses in a raw format or looking to archive outtakes in a more compact format will appreciate the flexibility." And some could use it as an archival format for second-tier photos.

And Adobe does care about lossy DNG image quality, and it encourages people to test it to see what they think. Here's Adobe's process, he said:

The way we've been testing this is by taking original raw files, making lossy compressed DNG versions of them, and developing the two pictures in Lightroom and then opening the results into Photoshop. We toggle between the two, and try to determine visually which is which. We do this at multiple visual scales (Fit View, 1:1, 2x, 4x, and even higher). Then we do a diff between the two images to scrutinize the visual structure of the differences.

DNG advantages and shortcomings
Another advantage of DNG in general compared to proprietary raw is that, when using Adobe software, the file itself can hold metadata such as titles, captions, camera information, and editing settings. Without DNG, that information is often stored in separate XMP-format "sidecar" files that must be managed carefully in conjunction with the raw files, or in a catalog database separate from the raw files themselves. Another DNG perk is that file sizes can be smaller than some cameras' native raw formats--though still much larger than JPEGs.

The Lightroom 4 beta and the upcoming Photoshop CS6 use a new DNG version, 1.4, with the new performance and compression options.

Adobe plans to publish the DNG 1.4 specification later this year, Hogarty said.

DNG is designed to make shooting raw more approachable, but DNG has plenty of constraints that limit its possibilities when it comes to real-world photography.

Its chief benefit is a way to store the original image data. But through nondestructive editing tools such as Lightroom, editing changes can be stored as just more metadata. That lets the DNG file store not just the original, but the processed version, too. If you edit your photos with Lightroom, though, you'll need Adobe's DNG decoder to see the changes, since those editing instructions aren't part of the specification.

And lots of photo editing takes place after Lightroom, which has imprecise adjustment controls, can't be used to merge multiple images, and doesn't support basic Photoshop features such as layers.

"The raw files are just the foundation image, with maybe 60-90 percent of work being done in Lightroom," said photographer Gary Crabbe. From it, he converts to a 16-bit TIFF image for Photoshop then saves an 8-bit master file. "I almost Never see my raw files again anyway, and even if I lost the sidecar, I usually only spend about 90 seconds working with any given raw file in Lightroom, so it's not to big a deal if I ever have to reprocess the shot."

In other words, neither raw nor DNG is the destination for Crabbe's work. It's a step on a path, not the final product that deserves the archival attention.

But judging from Adobe's changes with Lightroom 4, and Photoshop 6, DNG 1.4 is also just a step on the path. With standardization, more allies, performance improvements, and new features, Adobe has the potential to spread DNG to many more photographers.