Creating a EXIF-rich backup of all iCloud photos

[teh-hippo]

Hi Community!

Background

For some time I've been trying to implement an optimal approach to backing up all photos from iCloud to another cloud storage provider. Our family uses iCloud Shared Photo Library, allowing my partner to have photos easily seen on iOS on all devices, which is great, and they're automatically pulled down to any platform with iCloud.

My current approach is iCloud on Windows, and mirror that directory through rclone to a cloud provider. It's been great. Windows only because I had a spare PC with a large drive.

I make a lot of edits to photos on the phone. Photos we receive from others get added to iCloud, and then (this is the key bit), I add location and correct the date/time.

I recently discovered that all these edits to location & the photo's date weren't being synced to OneDrive. Working backwards, I eventually came to learn that it's part of how iCloud manages this metadata, rather than amending originals.

Since then, I've been trying to work out how to apply these edits to original photos, and upload those. I'm less concerned about the originals in iCloud, more than if I were to go to another provider, it's not lost.

Finding osxphotos, sounds like I'm joining the club!

Questions

What I really want to do, is have an iCloud library, apply all changes made in iPhotos (either on my phone, or on the mac) onto the originals, so I can then sync them. My ideal scenario, is that the originals are updated on iCloud (honestly, just need the date & location!). I'd be fine with it just being on my back-up.

Looking at the documentation, I think I could use either push-exif or export.

1. The documentation for push-exif says the following. I haven't been able to get metadata to sync back at all. Doesn't look like iPhotos wants to do it. What causes that sync to happen?

"It is also not recommended that you run this command on a library that is managed by iCloud Photos. Changes to files in an iCloud library may not be synced back to iCloud. This is most useful for libraries that are not managed by iCloud Photos or for libraries using referenced files.".

2. If I'm okay with having 2 copies of all files, is there any downside to using export? It sounds like that would sync. Is there a way I could have it clean-up files in the target exported directory that I've deleted on the phone (e.g., no longer in the library)? Apologies if I missed this in the documentation.

[oPromessa]

Welcome @teh-hippo !!! 😃

If I'm okay with having 2 copies of all files, is there any downside to using export? It sounds like that would sync. Is there a way I could have it clean-up files in the target exported directory that I've deleted on the phone (e.g., no longer in the library)? Apologies if I missed this in the documentation.

Yes. Check options --update and --cleanup which do exactly that. Option --touch-file may also be of relevance. The --keep option will prevent osxphotos from deleting files you may want to keep on the export folder.

Look for export on the Command Line Interface (CLI) and check out other iCloud related options like --shared and --download-missing. You must also consider enough space on your local hard disk with the photos library to get all the assets from iCloud.

I'm not an iCloud user myself and @RhetTbull will surely complete these remarks.

[RhetTbull]

You're welcome to ask questions here @teh-hippo. One correction to @oPromessa's excellent comments, there is actually an exiftool sidecar format and I assume that's what you were intending to export? It's also a JSON sidecar but equivalent to using exiftool -j which some 3rd party apps require.

My goal was:
- Only have 1 copy of each picture.
- Said copy needs to be the latest revision (e.g., edited). Hence no suffix being applied.
- Just do normal exports for now, apply `exif` data later, hence a sidecar.
- Any files that aren't in iPhoto should be removed.

You command should do this and in my testing it did.

I used your command (with the addition of the export directory) on a test library and it produced the expected results:

osxphotos export ~/Desktop/export --download-missing --update-errors --skip-original-if-edited --edited-suffix '' --update --sidecar exiftool --alt-copy --touch-file --cleanup

First run exported everything:

Processed: 21 photos, exported: 22, updated: 0, skipped: 0, updated EXIF data: 0, missing: 3, error: 0, touched date: 43

Subsequent runs did not update anything as nothing needed to be updated.

Processed: 21 photos, exported: 0, updated: 0, skipped: 22, updated EXIF data: 0, missing: 3, error: 0, touched date: 0

I touched a file and added a file to verify the expected behavior:

It updated the one file whose signature had changed and deleted the added file.

Processed: 21 photos, exported: 0, updated: 1, skipped: 21, updated EXIF data: 0, missing: 3, error: 0, touched date: 1
Elapsed time: 0:00:06
Cleaning up /Users/rhet/Desktop/export
Deleted: 1 file, 0 directories

I also have a bunch of dot files .IMG_1234.HEIC

OSXPhotos does not create these so not sure where they are coming from.

Please note that your command dumps everything into a single folder. I recommend use of --directory or --export-by-date in addition to your other options.

If you send me the full verbose output using --verbose it might be helpful to figure out what's going on. (osxphotos.py@gmail.com) (You could try exporting only a subset by selecting several photos in Photos then adding --selected so you don't re-export everything)

[teh-hippo]

Regarding the dot files: Turns out it's related to AppleDouble file formats. I'm exporting to a non-mac directory, which is likely why it's sometimes being there. Mystery solved - I think I can just delete it if I want to.

❯ file /mnt/d/photos/._IMG_0076_edited.jpeg
/mnt/d/photos/._IMG_0076_edited.jpeg: AppleDouble encoded Macintosh file

[teh-hippo]

Next, --limit is a godsend. It's been fantastic for testing. --selected is also great.

[teh-hippo]

I had an issue with files that have the same name. I've written up my findings in #134.

[teh-hippo]

I also think I'm having #872. I posted my results there too @RhetTbull.

[RhetTbull]

Welcome @teh-hippo

My current approach is iCloud on Windows, and mirror that directory through rclone to a cloud provider.

Do you have a Mac? I have never tested osxphotos on Windows and doubt it would work. It does work on Linux if you have access to the Photos library from a Linux machine. If you want to backup Photos from iCloud on Windows, I recommend iCloud Photos Downloader.

1. The documentation for push-exif says the following. I haven't been able to get metadata to sync back at all. Doesn't look like iPhotos wants to do it. What causes that sync to happen?

Do not use push-exif with a library synced with iCloud. I do not think it will harm your library (but cannot confirm this) but Photos will not see the changes and sync those to iCloud. Once an "original" photo is imported into the library, the original cannot be changed. Photos only does non-destructive edits meaning any change results in one or both of changing metadata in the Photos database and/or creating an edited image. But the original is never modified and you cannot force iCloud to sync modifications to the original.

2. If I'm okay with having 2 copies of all files, is there any downside to using export? It sounds like that would sync. Is there a way I could have it clean-up files in the target exported directory that I've deleted on the phone (e.g., no longer in the library)? Apologies if I missed this in the documentation.

Using osxphotos export is what I recommend in your case assuming you have a Mac synced to iCloud Photos. If you don't have a Mac, you're likely better off with iCloud Photos Downloader. Please read the osxphotos tutorial and documentation. The tool is very customizable to whatever scenario you want. To get you started, the following would satisfy what you asked for:

osxphotos export /path/to/export --update --cleanup

If you've changed dates and other metadata and want to preserve metadata in the original files, look into --exiftool (which requires installation of exiftool ) and --touch-file. If exporting to a network volume, you should use --ramdb

I would personally use --sidecar xmp as well which exports an XMP sidecar with all the metadata for each image.

is there any downside to using export

No. This is intended for backup and all exports are non-destructive and do not modify the Photos library.

OSXPhotos works best on a library that has "download originals to this Mac" set in Photos > Settings > iCloud. If you are using Optimize Mac Storage then you'll need to use --download-missing as the originals will be missing from the local Mac.

One additional thought: if you are using a Mac and the drive is formatted with APFS (the default in all modern Macs), then if you export to the same volume as your Photos library, OSXPhotos will use "copy-on-write" which means that the copy doesn't use any additional disk space but your backup client sees the files as separate files and can back them up (e.g. TimeMachine) or upload to a cloud provider. However, if you modify the files (e.g. using --exiftool) then a real copy is created which does use up disk space. I use this on my personal Mac so that BackBlaze and TimeMachine create backups of the images separate from the Photos library.

[teh-hippo]

Oh somehow I didn't post my reply - apologies to you both. Frustrating cause I'd typed it all out. Must have closed the tab before posting.

Big thanks to you both @oPromessa and @RhetTbull! In my longer response I broke down what you both said more specifically, but I'll just keep it shorter now.

Given it's been a few days since my original 'reply', my update is that I've been successful in using osxphotos with the suggestions.

• @oPromessa - those flags are perfect and what I'm after thanks. Appreciated. I ran into an issue with one of the commands when running it through WSL - a Python issue. It's convenient to have a server basically pulling the images, updating the library, and then being able to manipulate it on another device. I'll see if I can work out what's wrong later in the week.

• @RhetTbull - I tried iCloud Photos Downloader a while back. At the time it didn't support shared library (or I don't believe it did...), but it certainly didn't have what I needed - the EXIF data being exported. It's progressed - and it looks like dates are now updated. But it doesn't pull the location yet.

I want to play around more with the copy-on-write aspect too - I think that'll prove very useful.

Thanks again to you both!