noel_carboni's profile

116 Messages

 • 

1.9K Points

Wed, May 18, 2011 11:39 PM

Closed

Lightroom/Camera Raw: Add a Camera Raw Option to Prevent Writing Back into Input Files

Given:

Under some conditions Camera Raw writes data back into (overwrites) its input files. Example: A JPEG file.

Under other conditions Camera Raw maintains this same kind of information in a separate place (e.g., Sidecar XMP or central database). Example: A CR2 or NEF file.

Assuming one uses Camera Raw to open out-of-camera original files as many photographers do, depending on what mode one has captured the images in, Adobe is inconsistent about whether to keep its hands off them or overwrite them... This seems to be because some formats are proprietary and some are well documented. From a programmer's perspective, this makes perfect sense.

Trouble is, from a user's perspective, this behavior cannot be described as anything but inconsistent.

Personally, I do not want my original out-of-camera JPEGs updated/rewritten under any circumstances.

Camera Raw will not touch a proprietary raw file, such as a Canon .CR2 or Nikon .NEF. There's a whole process for remembering settings in a separate database or sidecar XMP files. So far so good.

However, if you open a JPEG, TIFF, or DNG through Camera Raw, data WILL automatically be written back into it to tell another run of Camera Raw in the future what settings you used - without the software ever having warned you it will do so.

It is true that some functions EXPLICITLY rewrite input files. You can ask the software to write new thumbnails back into DNG files, for example. This seems fine - the user has instructed the software to overwrite the file, and the user is in charge, after all.

Overwriting/rewriting an input file without being instructed to do so is NON-INTUITIVE BEHAVIOR for any application. Simply put, I would not expect an input file to be overwritten by Camera Raw.

And we do see that it causes people confusion and surprise from time to time. You may right now be reading this in disbelief. I recommend you go test it for yourself (on a copy of one of your original JPEG files).

The original file being overwritten is a chief reason why I don't configure Photoshop to open my JPEGs through Camera Raw.

Adobe:

Please give those of us who don't want our input files overwritten an option for using the database/XMP sidecar instead in EVERY case.


Thank you.

-Noel

2.3K Messages

 • 

26.4K Points

11 y ago

I believe that because Adobe feels proprietary raws are read only, the alternative is sidecar files. All the other files are read/write, non proprietary files. In that context, the behavior makes some sense. Is there a reason you would not want the metadata data written into the file format itself?

As for saving data without asking, isn’t LR doing this all the time? Well to the database (or the data itself if so set).

116 Messages

 • 

1.9K Points

11 y ago

Hi Andrew,

Thanks for commenting. Yes, that's the reasoning I derived as well.

My point is: Yes, there is a reason I would not want the metadata written back into the file itself: I simply don't want changes to my original out-of-camera files, however trustworthy the application. Yes, I have them backed up several ways.

As it is, I can accomplish this by just not using Camera Raw on any formats that it feels comfortable doing such write-backs to. This makes sense in another way in that Camera Raw was designed for, well, raw files, not JPEG or TIFF or PSD or whatever. The line grays a good bit when we talk about DNGs.

I'd simply like to be able to configure Camera Raw to use the metadata database or XMP file in ALL cases via a configuration option. Then it would be consistent in its operation, and I would not need to worry about whether I'm opening an original JPEG with Camera Raw.

Someone else will have to speak for Lightroom. I don't use it.

-Noel

4.5K Messages

 • 

76.3K Points

11 y ago

Speaking for Lightroom: two thumbs up, way up. And in the case of Lightroom, support for virtual copies would also make sense, big sense...

Thanks for posing this Idea Noel.

-Rob

4.5K Messages

 • 

76.3K Points

10 y ago

To Forum Administrator: please add 'Lightroom" designation to title, so Lightroom users know that this applies to them too - thanks.

To be clear: this idea is about not writing back to original camera files, and handling all photos consistently, by using an XMP sidecar for all file types (and virtual copies), or possibly in a common database if not Lightroom environment.

116 Messages

 • 

1.9K Points

10 y ago

Thank you, Rob.

Yes, I'm envisioning the use of XMP sidecar -or- central database, EXACTLY as is done now for raw files that cannot be written back into.

-Noel

4.5K Messages

 • 

76.3K Points

You bet, Noel.

The common objection to this 'Idea' seems to be the argument that sidecars are a workaround for proprietary raws, but some of us would prefer sidecars regardless, or in your case the central database.

I think the main barrier to adoption is that Adobe is trying to make in-file xmp a selling point for DNG.

Technically, this would be nearly trivial to implement.

I'm hoping but not expecting...,
Rob

9 Messages

 • 

232 Points

10 y ago

Lightroom needs to be patch so that xmp files can be generated for use with dng files.

dng is not a valid raw choice unless I can treat it like any other raw file. I can't.

Our raw files are archived on dvd (read only). the edit data is archived as xmp in another location. I can use acr to force the xmp files after the edit but it would be better if lightroom didn't arbitrarily refuse to do it. (It did it in beta).

I'm not interested in hearing from anyone who wishes to argue that eliminated xmp files is a good thing. It's isn't. They're tiny and they can be used to recreate an edit long after the dngs have been removed from a catalog.

Did I mention that our raw files are archived in read only format before they're edited? They're received from photographers on dvds and those discs become our archive.

This reply was created from a merged topic originally titled
Lightroom needs to generate xmp files from dng..

2.3K Messages

 • 

26.4K Points

10 y ago

>dng is not a valid raw choice unless I can treat it like any other raw file.

Why not?

9 Messages

 • 

232 Points

Interesting. The reason is built in to the sentence that you commented on yet you ask why not anyway.

It can't take the place of what I'm using if I have to treat it differently.

Geez

116 Messages

 • 

1.9K Points

10 y ago

Andrew, I suspect the difference Robert is sensing is the core concept of this thread - that the Adobe software feels comfortable in writing back into the DNG file, while proprietary camera raw files are "hands off", but that's just a guess.

Some people really want their raw files to be untouched. This is not a difficult concept.

-Noel

2.3K Messages

 • 

26.4K Points

10 y ago

Adobe feels comfortable writing back to lots of non proprietary file formats and uncomfortable writing back to proprietary formats they don’t control. That’s not a difficult concept for both legal and other reasons. Since there are oh so many proprietary camera formats that continue to come onto the scene every time a new camera is produced, the idea makes a bit of sense. You’re ‘stuck’ with sidecar files. But none of the above explains why DNG isn’t a valid raw choice. Its a different chose but not valid?

948 Messages

 • 

13.9K Points

He did explain why not. He has read-only DNG files and would like to archive just the metadata in the XMP files, which are a few orders of magnitude smaller than the image files. That would represent a substantial savings in both space and time especially since he has the image data archived already.

2.3K Messages

 • 

26.4K Points

http://forums.adobe.com/message/3245182:

>It would be great to have an option whether to imbed XMP metadata into DNG file (as it is write now), or create XMP sidecar files as Lightroom does for proprietary RAWs.

Answer why its not useful:

Because it's half-baked in backup terms - you can't simply dismiss a swathe of LR work as "image relationships properties" - and it forgets that one valuable aspect of DNG is precisely that it is designed so you can safely write metadata to the format (though you'd be on your own if you do so to your only copy of the image....). Development time spent on sidecars for DNGs (JPEGs too, and why not TIFs?) would be unavailable for more important enhancements.

John Beardy

Uodate: Since you changed your post while I was writing.....

'And by the way, ACR can be forced to act like this. If you mark your DNGs as "read-only" ACR will not touch them, but will create sidecar XMPs.
While Lightroom just removes "read-only" attribute and overwrites original file without shame'

Then use ACR..... LR's ethos is to start from scratch, not slavishly replicate earlier tools.

Author “Color Management for Photographers" & "Photoshop CC Color Management" (pluralsight.com)

9 Messages

 • 

232 Points

Writing the data "safely" has never been a concern of mine. I find the xmp files to be vital since I don't have the option of writing the metadata back to the dng (them being read only)

More than once, I've been able to recreate an edit of an event using the raws and the xmp file which allowed me to pick up where I left off.

If the originals were dng, this would have been impossible. I would have had to re-edit the job from scratch.

Fyi, Lightroom 3 doesn't remove the read only, it returns an error writing metadata.

And I haven't hit on a consistent set of changes to make and change back. Apparently, acr thinks some changes aren't worth saving.

13 Messages

 • 

370 Points

My backup solution does not allow for differential backups... every file that is modified even one bit has to be saved again (which takes about a minute). That is a pretty big issue, I can't just go back and update a couple of tags. Sometimes even regenerating the thumbnails as I moved the catalog location meant I had to reupload a huge portion of my library (taking a few months, which meant had I lost my drive many files would have been lost!). In an ideal world we don't need sidecar files. In reality it is a useful thing for some of us, so why not give us that option?

116 Messages

 • 

1.9K Points

10 y ago

I guess the question I didn't have answered before was "what does the software do when you have a read-only DNG file?"

I just did some experimentation on some proprietary raw files, along with JPG and DNG files, making some of the DNG and JPG files read-only (but the folder still read/write), using Camera Raw 6.4.1, and here's what I found...

When configured to use the central database, Camera Raw did NOT attempt to write back into the DNG files, and instead went to the central database to store/retrieve conversion parameters. Consistently, it used sidecar XMP files in the same folder as the DNG files when configured to use sidecar files instead of the central database.

It emitted an error when Camera Raw tried to write back into the JPG files unconditionally.

Clearly the DNG files are being treated differently than JPEGs. This may answer Robert's point and it does make Andrew's question valid...

Robert, have you actually TRIED using DNG files for the task, given your processes?

-Noel

9 Messages

 • 

232 Points

Yes, despite our requesting actual camera raw files, some photographers have been hoodwinked into adopting dngs on the off chance that some archeologists excavating his studio in a thousand years might want to look at his images.

2.3K Messages

 • 

26.4K Points

Or in less than 10 years, they might actually want to access the data (which today I can’t do with 10 year old PCD and DCS raws).

Author “Color Management for Photographers" & "Photoshop CC Color Management" (pluralsight.com)

116 Messages

 • 

1.9K Points

Can't open a PCD? Why is that? Because Adobe dropped support of the PhotoCD import plug-in from their latest products? Don't you have an older copy of Photoshop still running (or have done as I have and made the old plug-in work with the latest Photoshop)?

So the question becomes, when will Adobe drop support for DNGs (or just stop making new software or go out of business)? Less than 10 years from now? Open standard or no, it's not like it's been embraced by anyone else.

-Noel

2.3K Messages

 • 

26.4K Points

>Because Adobe dropped support of the PhotoCD import plug-in from their latest products?

The Plug-in was Kodak’s, it doesn’t run in Lion. The standalone Kodak software for DCS process is even older and requires OS9. So while its not impossible today, it requires hardware and OS’s that I don’t own any longer.

Even if Adobe did drop support for DNG, its an openly documented format. Anyone who wanted to continue to render that data could. DCS files and probably PCD were not.

Author “Color Management for Photographers" & "Photoshop CC Color Management" (pluralsight.com)

116 Messages

 • 

1.9K Points

Ah yes, I forgot about the discontinuity in the Apple legacy. Thanks for the info.

I suppose it's up to whomever has archival data to occasionally review capabilities and convert to a more modern, still-supported format as needed (e.g., when you still DID have PCD or DCS support, you should have converted all your images to TIFF).

I can see that it probably wouldn't hurt, if sensing an impending discontinuity, to make DNG in addition to keeping your proprietary camera raw files then. But that doesn't mean it has to be right now.

9 Messages

 • 

232 Points

10 y ago

My point has nothing to do with jpgs or tiffs. Only with the fact that lightroom needs to have an option to use xmp files with dng at my discretion, not Adobe's.

2.3K Messages

 • 

26.4K Points

Now all you have to do is convince Adobe why.

Author “Color Management for Photographers" & "Photoshop CC Color Management" (pluralsight.com)

9 Messages

 • 

232 Points

Because I'm the customer and they do things because of me, not themselves.

116 Messages

 • 

1.9K Points

10 y ago

Yes, an option that would FORCE the use of external storage for development settings is exactly what I'm requesting in this thread. And this option DOES need to affect all file types that can be opened by the raw processors (Camera Raw / Lightroom).

For those who seem to want to oppose this request, note that all I am requesting is an OPTION, not a permanent change to the operation of the software. Thus, you get to keep working as you do now, but the products become more useful for people like Robert and me who need to keep it from writing back to original files.

As far as convincing Adobe goes... What part of "applications shouldn't normally write back to input files" don't you understand? What they're doing now is inconsistent and wrong!

-Noel

13 Messages

 • 

370 Points

I don't mind the default behavior, it's just that for some use cases it makes much much more sense to use sidecar files. A here be dragons warning that tells the user what the option is for, and what the risks are behind it should be sufficient. It's like Apple telling iPhone developers that they can not use the volume buttons to take photos, because it confuses the user. Yes, perhaps, but banning a very successful application from the App Store just because there is a hidden website that users of this program can visit, so that they can activate that feature? I assume when they go through such lengths they very well want that feature and know of the consequences! I don't even mind editing a config file hidden somewhere on the hard disk.

9 Messages

 • 

232 Points

10 y ago

Noel Carboni asked:

Robert, have you actually TRIED using DNG files for the task, given your processes?

Yes, some photographers insist on sending us dng even though we've asked for camera raw direct for the camera without any changes.

I can either use the Bridge/ACR method of forcing xmps or burn new dvds using the edited dngs.

I'd rather just save the xmps.

2 Messages

 • 

90 Points

10 y ago

Camera Raw will write sidecar .xmp files for DNGs if the DNG file is readonly. (I use this in my workflow as my main camera is DNG native and I do not want my raw originals getting modified.) I'm not sure if this works in Lightroom, but if not, it could be regarded as a bug and just fixed. This is a small step toward giving users full control over where metadata is stored, but it can help quite a bit.

There is certainly awareness within Adobe of the workflow issues discussed here. I agree that embedded XMP causes pain for some workflows and the lack of user control is regrettable. Unfortunately the easy changes we can make look to introduce more problems than they solve and it is hard to do something winning in just Camera Raw and Lightroom as a strong selling point of XMP is that it interoperates well across a vast range of products from Adobe and other vendors. If the other apps don't also respect the constraints, the metadata ends up with the same field stored in multiple places and becomes ambiguous or inconsistent.

-Z-

4.5K Messages

 • 

76.3K Points

10 y ago

|> I'm not sure if this works in Lightroom, but if not, it could be regarded as a bug and just fixed.

It does not work in Lightroom. I like the idea of regarding it as a bug to be fixed. (there have been several forum discussions about it, but I don't recall any comment from Adobe)

There is a partial solution for Lightroom users:

xEmP

In the case of DNG and RGB, it depends on xmp being 1st written to the source file, therefore you can't make them read-only. xmp will then be extracted and written as sidecar if changed. So, it solves the problem of a smaller (sidecar) file for settings/metadata that is only modified when necessary. It does not solve the problem of writing back to originals, or storing in common database. It also writes xmp-like files for virtual copies so they can be backed up and restored if necessary.