Filtering duplicates from iCloud Photos Library while migrating from google photos
I recently painstakingly migrated my photo library of 15+ years (over 100gb) from google photos to iCloud. I suspect many more will be doing this in the future as google recently announced they will no longer be doing unlimited storage for photos. The problem here is that the library exports originals as well as an -edited version for those files you have changed in google photos. I've tried multiple automatic tools and had some luck with photosweeper but am sure that i still have a few gb of undetected duplicates hiding in there. In some cases Photosweeper just doesn't seem to be sophisticated enough to find them - For example photos from a scanning service that all have the same date, or edited versions where i have rotated and changed the aspect ratio.
Complicating this is the way Mac OS automatically manages photo storage on the machine, so i'm hesitant to attempt to run any scripts on underlying files that might corrupt the library. In a perfect world I would just do something like
- find all files matching
(.+)-edited.{jpg,png,heic}$- where there is a matching file without the -edited
- select and delete/trash the non-edited version
- in a way that is safe for my photos library
An alternative might be to start over completely and use the google drive app on my desktop to download and copy all pictures that way, but i'm worried i would lose files that way too.
I use PowerPhotos for this and am super happy. Dump everything into one library, clone it and then try out various passes until you’re happy, then do the culling on your main library is my advice.
- https://fatcatsoftware.com/powerphotos/
It lets me make rules once the pre-built ones take a first pass at cleaning the easy and obvious duplicates. You can prefer HEIC and edited or any other sort of metadata for automating which of the duplicates to select to keep and which to trash. It is updated very often, super easy to use and extremely powerful. I’m a decent “shell scripter” but in the end, this tool saved me many days of trying to “yolo” it with scripts. It’s got the goods on parsing photo metadata and wrangling libraries into shape in a safe manner.
Unless you’re willing to eschew Photos and syncing entirely, there aren’t better options. If you could exist entirely on filesystem and folders and a tool like Photo Mechanic to store photo metadata in the filesystem, I would go all in on Photos and iCloud and hire a tool if you can’t script it yourself.