I nearly lost the Lightroom catalog with of all my photos

This is what my photos looked like after I moved them to NAS

With 50MB RAWs, hard drives fill up in no time. In the last five years I collected enough important photos to start thinking about a reliable storage solution that works well with Lightroom. I opted for Synology DS218 with two Seagate IronWolf drives.

When I migrated my RAW files from USB drives to Synology NAS, everything seemed fine at first, but when I browsed through the Lightroom catalog a few days later (tip: always wait until Lightroom rebuilds previews), I noticed that pretty much all my photos had this weird horizontal banding... I hoped it's just an issue with previews in Lightroom, but a quick googling made it clear: files are corrupt and there's nothing that can be done to fix them.

It's fine, I thought. I have backup, I thought.

Haha, nope.

Time Machine Woes

Time Machine used to work flawlessly for me with Apple Time Capsule, but ever since I switched to Synology, backups are created fine for the first few days and then they stop. What I usually do then is recreate the backup from scratch and plug in my external drives so they're backed up once (this data rarely changes). This time I either didn't connect my hard drives or TimeMachine didn't back them up.

Either way, I should have checked.

To make matters worse, I deleted the photos from the USB hard drive, without verifying the new catalog. But at least I didn't touch that drive afterwards, so there was a chance that I'd be able to recover the files.

Recovery

At this point, all I could do was to try to recover the deleted files. I used Stellar Data Recovery app, which cost me $79 and took over 15 hours to scan the hard drive and another 12 hours to recover the files to another drive. Thankfully it did the job.

The only problem was that filenames and directory structure were lost and I was left with thousands of files to sift through...

Stellar found the deleted files, but without the original filenames

With 9214 photos to restore, the process could easily take many days. I had to find a way to automate this.

Although the files on the NAS were corrupt, the directory structure wasn't, so I could use it as a reference. I had the list of files that need to be recovered — I only needed a way to match them with the ones I got from Stellar Data Recovery.

Files were messed up, but I could use the directory tree to recreate the catalog

ExifTool to the rescue. I wrote a script that went through each file in the broken Lightroom catalog, extracted its ModifyDate Exif tag (timestamp of when the photo was taken) and created an index of filename + timestamp. Then it created a reverse index of timestamp + filename from the list of recovered files. That allowed me to match the files and recreate the catalog in a new location, without any manual work.

Recovery script in action

Here's the script, in case you ever need something like that (I hope you don't).

Result

It was a substantial amount of data to process and the script took more than 12 hours to finish, but it worked! I have my photos back. Well, most of them.

Out of 9214 photos, roughly 200 weren't recovered correctly (missing or damaged files, sometimes combining different photos into one). None of them were photos I cared for, so I didn't investigate further.

OK, but What Happened?

I'm still not sure why the files got damaged in the first place. A quick googling indicated a hard drive problem, but I used the exact same NAS to recreate the catalog and this time everything went fine.

Synology doesn't report any issues with the hard drives either. I ran Data Scrubbing too and it didn't show any problems. Same with S.M.A.R.T. tests.

It's likely I messed up by not checking the "Enable data checksum for advanced data integrity" option when I created the shared folder. Still, this option is unchecked by default and it's not really clear that leaving it that way could lead to data corruption.

Another possible cause: files could be damaged when I transferred them over Ethernet due to packet loss/corruption. I use TP Link Powerline adapters to extend my network and it does seem to drop packets sometimes. That said, I used the same setup to transfer the recovered files and had no issues with data integrity, so I don't think that's the case here.

(If you're reading this and have a better explanation for what happened here, please contact me)

Lessons Learned

The whole ordeal was largely of my own making. I never anticipated that the simple act of copying files from one place to another could go so horribly wrong. Still, I should have checked that I had a usable backup in place (Lesson #1). I should have also made sure that the new Lightroom catalog is fine before I deleted the previous version (Lesson #2). But most importantly: I should stop hoarding so many damn photos.

---

PS: If you enjoyed this post, be sure to check out my app:

View Lightroom edits for JPG files | pixelpeeper.io

PixelPeeper.io lets you recover Lightroom edits and camera settings from JPG files. Use it to learn how your favorite photographers achieve their style.

from Hacker News https://ift.tt/3g9MtDa