My photo loss nightmare

As if 2021 wasn’t bad enough, the external hard drive that holds approximately 100,000 of my photographs died. And my backup drive turned out to be no help at all. I’ve detailed the whole catastrophe in case it might help others avoid the mistakes I made. Mistakes were definitely made.

The disaster started in November when I decided to put my entire library online in case my external drives ever failed or were destroyed (say, in a fire). I have a SmugMug account with unlimited storage, so all I had to do was set up a Publish Service with yearly Smart Albums, then hit Publish. This is all very easy to set up and allows one-click syncing of physical and virtual copies.

The only pain was that the process of uploading each album (~5,000 pics each) can take days and ties up Lightroom. So I would just set multiple years up on the Publish task and leave my computer to do its thing. And while Lightroom hummed away I did yard work, cooked, and cleaned the chicken coop. I’d check back every few hours just in case Lightroom stopped uploading, which happened a lot. But that’s normal with Publishing actions so I’d just restart the publish process on the remaining, unpublished photographs in each album and walk away. Days went by like this, and I was slowly getting my collection online, with perhaps 30,000 photographs to go. The remaining photographs were large RAW files (50-60 MB each), so I wasn’t too alarmed by the glacial upload speeds and the frequent need to restart the publishing process.

Failure of primary external hard drive

At some point in this process the hard drive simply unmounted. Disk Utility deemed it unfixable.

The failed drive (bottom): G-Technology 4T USB 3.0 / FireWire 800.

Backup drive to the rescue?

With the primary drive out of commission (oh, well, that happens) I simply pointed Lightroom to my backup photographs on the other 4T drive. It was a slower, cheaper drive but it was a clone of other drive thanks to nightly updating I did with SuperDuper! software ($27.95). Lightroom made this switch flawlessly — the number of photographs in my catalog was exactly the same as it was when using the dead drive. Phew. So I ordered a new backup drive and then continued the uploading to SmugMug, now even more convinced that I needed virtual copies of all my photographs.

But the uploading process still kept hanging, and I eventually discovered that the Publishing process was choking on missing files. But this new drive was just fine, so I was perplexed. After a bit of poking around I discovered that approximately 30,000 originals were missing. The Preview files were there, just no originals. My guess is that the original drive failed over a week or so, and SuperDuper! faithfully copied all the errors onto the backup drive. And because those missing originals were never uploaded to the cloud, I had truly just lost 30,000 photographs. Family pics, nature pics, etc. All gone, forever. I was horrified. I don’t deal with loss well.

Data Rescue 3 to the rescue?

But because the working drive used to have those files, I hoped I might be able to get them back using Data Rescue 3, which I own. It worked, but the files had to be manually placed in the correct Lightroom folder and then rematched to the catalog. This process took weeks and was extremely unpleasant. And to my dismay, I soon realized that a large percentage of the files were damaged. I was seriously considering just aborting the entire process, or perhaps dedicating a third hard drive for housing the tens of thousands of unhomed, potentially damaged files that I’d recovered. The process of fixing the mess could easily take me thousands of hours, all unpleasant.

Disk Warrior to the rescue?

I decided to buy Disk Warrior ($120), which many people said worked wonders on drives that Disk Utility couldn’t repair. Maybe, I hoped, the files were OK on the original drive and only became corrupted during the cloning process. I know that’s delusional thinking, but I was desperate. And Disk Warrior couldn’t repair it. But for some reason it tricked the Finder into allowing the disk to be mounted. And when I tunneled into the drive to see whether the original files were there, they were there — and undamaged.

I then tried to use the clone feature on Data Rescue 3 to make a copy of the damaged drive. That process was slow but appeared to be working. E.g., at one point in the process it needed another 56 hours to complete step 2 of 3.

But the after four days the estimate kept growing, eventually suggesting it might take several years. I decided to abort because the drive was likely to fully die before the clone ever completed. Or I would die of old age, waiting. Both seemed likely.

It was agonizing. I knew the files were there … but the drive was so slow for some sectors that I couldn’t get them. E.g., if I used the finder to drag a file from the damaged drive to a new location, it choked. Sometimes it worked but there were thousands of files that simply couldn’t be copied.

FreeFileSync to the rescue

After some searching online, I discovered several programs that were capable of slowly copying directories of files. FreeFileSync turned out to be just the ticket. I used it to copy whole months of photographs from the damaged drive to the new one. Each copy process might take days but it would plug away even though read speeds would go to near zero for hours. Identifying the missing files and getting them all copied took weeks, but it worked. I ended up giving money to the brilliant folks at FreeFileSync.

Lessons I’ve learned

After approximately two months of near constant work I have recovered every single photograph. In case helpful, here are some tips to avoid a similar mess.

  1. Regularly check Lightroom for missing originals and for originals that are missing Previews. The reason for the latter is that Lightroom cannot make previews from present but damaged originals.
  2. Only after I am convinced that files are present and healthy do I invoke a synchronization action to update my backup drive. I use FreeFileSync.
  3. A single backup drive is insufficient. I purchased an additional, portable drive (LaCie Rugged) that I now update every week but unplug and keep in a fireproof lockbox. I’ll probably get a fourth drive that I can keep at a friend’s house.
  4. Online backups are critical. But I already knew that.
  5. Don’t unduly stress a drive with massive Lightroom tasks. I decided to buy a LaCie SSD drive for this reason (because no moving parts).
  6. Check drives occasionally to see whether they are going bad. Disk Utility is OK for this but I might also invest in DriveDx.

Now I can get back to taking and posting nature pics. I’ve missed it.

Nature pics from Zolfo Springs

Photographs from a week in Zolfo Springs, Florida. In case you’re wondering, the namesake spring was capped with concrete in 1960s, and the large pool it fed has been filled with dirt. But still plenty of nature to photograph.

Cuban tree frog (Osteopilus septentrionalis) on a post

Cuban tree frog (Osteopilus septentrionalis). Apparently arrived in Key West in 1931 or earlier. They eat each other as well as any of the native tree frogs, which is likely why native tree frogs are becoming so rare in the area. Reported to take up residence in bird boxes, too, which is thought to annoy birds and result in their declines. Also fond of hiding in boxes that house electrical components, which doesn’t end well. And they clog toilets. I could go on. All considered, a really awful species for Florida. And it’s spreading to other states.

Juvenile Cuban tree frog (Osteopilus septentrionalis) on lemongrass

Juvenile Cuban tree frog (Osteopilus septentrionalis) resting on a lemongrass leaf. Yes, cute, but don’t be fooled.

Black-tailed red sheetweb spider (Florinda coccinea)

Black-tailed red sheetweb spider (Florinda coccinea) with morning dew. I wonder whether the red color evolved to lure prey, perhaps mimicking a fruit or flower. Would be interesting to compare capture data with similar sheetweavers that have black bodies.

Condylostylus mundus

A male Condylostylus mundus. Longlegged flies (Dolichopodidae) are hard to photograph and ID, but I got lucky here. There just aren’t that many that are blue and chunky like this. Females of this species are green.

Copulating pair of common lovebugs (Plecia nearctica)

Copulating pair of common lovebugs (Plecia nearctica). Probably the most photographed fly on the planet. So definitely had to take one more. In addition to mating, they are fond of hanging out in large numbers on freshly painted surfaces. I have no idea why.

Geranomyia virescens (possibly)

Possibly Geranomyia virescens, a crane fly. The long proboscis is for drinking nectar, not sucking blood. It’s just a harmless pollinator.

Black stink bug (Proxys punctulatus)

Black stink bug (Proxys punctulatus). Here’s the BugGuide info page. The University of Florida has a very nice description of the species, too. Like many insects it’s gone through a half-dozen different Latin names, which makes it hard to locate interesting natural history tidbits that are often only available in old books and journals.

Hyalymenus sp. feeding on scarlet sesbane (Sesbania punicea) fruit

Hyalymenus sp. feeding on a scarlet sesbane (Sesbania punicea) pod. BugGuide says there are three species in Florida but offers no help in distinguishing them. Per some experts it could be the case that all three species are variants of a single species. If you know the answer, please leave a comment on the post or go directly to one of the iNaturalist observations I’ve made.

Yellow ants (Monomorium floricola) on cow skull

Yellow ants (Monomorium floricola) on a cow tooth. This species likes to nest in arboreal cavities so this is a fitting location. If you’d like to see closeups, antwiki has them.

Conura sp. on tomato leaf

Conura sp. (Chalcididae) on tomato leaf. I have no idea which species. And there are a lot in Florida.

Chelonus sp.

Chelonus sp. They’d land for approximately 1 second, then launch themselves for more hovering. So super hard to get a decent photograph. An odd-looking wasp because the abdomen is almost completely smooth due to fused tergites. Using wing venation alone this appears to be Chelonus kellieae (per figure in Marsh 1979), but that’s a species from Costa Rica and I’m not sure it’s in the United States. But I bet it is; its primary host, the potato tube moth (Phthorimaea operculella), is found in Florida. There were thousands of these wasps hovering over the lawn in the morning, so they are certainly eating something that’s in great abundance. They are egg-larval parasitoids (the egg is deposited into the host egg but doesn’t kill the host at that point).

Polistes bellicosus at nest

Polistes bellicosus nest. You can see an egg in the top left cell. They were not happy with me being so close. It was mutual.

New World banded Thynnid wasp (Myzinum sp.)

New World banded Thynnid wasp (Myzinum sp.). I think this is Myzinum maculatum because it has clear wings with brown tips; plus coloration and banding matches that of individuals ID’d as such (e.g.). Also, one source says the species has a cleft front claw (figure 7), and I think that’s the case here. You can tell this is a male by the curved pseudostinger at tip of abdomen. The larvae are parasitoids of scarab beetle larvae.

White-footed leafcutter bee (Megachile albitarsis)

White-footed leafcutter bee (Megachile albitarsis) resting with a section of leaf cut out of air potato (Dioscorea bulbifera). It’s one of thousands of species described by Ezra Townsend Cresson, a guy now long dead but who lived five blocks from me in Swarthmore, PA. I’ve even been to his garden several times.

Twice-stabbed lady beetle (Genus Chilocorus sp.)

Twice-stabbed lady beetle (Chilocorus sp.). Given size of spots I’m wondering whether this might be a cactus lady beetle (Chilocorus cacti). But Chilocorus stigma is super common in Florida and is said to have larger spots in the south, so I’m unsure. I should have taken a photograph of the ventral side. One of life’s regrets. They eat scale insects.

Eudiagogus maryae on scarlet sesbane (Sesbania punicea)

Eudiagogus maryae on scarlet sesbane (Sesbania punicea). Members of the genus are known as Sesbania clown weevils, appropriately. The larvae eat Sesbania, too.

Immature Surinam cockroach (Pycnoscelus surinamensis)

Immature Surinam cockroach (Pycnoscelus surinamensis). Parthenogenetic, apparently.

Assembly moth (Samea ecclesialis) on fern frond. Zolfo Springs, FL.

Assembly moth (Samea ecclesialis) on fern frond. Also called the stained-glass moth (hence the species name, I assume). Larvae eat Mexican clover (Richardia brasiliensis) but perhaps other plants as well (very little information available for such a common moth).

Eggplant webworm (Rhectocraspeda periusalis) on tomato

Eggplant webworm (Rhectocraspeda periusalis) on tomato. I spent a whole day identifying this. I’d assumed that any pest of tomato would be easy to identify, but I guess it’s rare to find it on tomato.

Graylet moth (Hyperstrotia sp.) larva on scarlet Sesbania (Sesbania punicea)

Graylet moth (Hyperstrotia sp.) larva on scarlet Sesbania (Sesbania punicea). ID is very tentative, however, and is just based on visual matches to similar caterpillars on BugGuide and in Wagner’s book. Here’s the iNaturalist observation.

Caterpillar of pale-edged Selenisa (Selenisa sueroides) on scarlet sesbane (Sesbania punicea)

For once, a caterpillar that was easy to identify: pale-edged Selenisa (Selenisa sueroides), on scarlet sesbane (Sesbania punicea). In searching for interesting natural history on this species I stumbled onto a paper by Bushwein et al. (1989) describing the larvae’s habit of chewing through flexible PVC tubing in citrus grove irrigation in search of a good place to pupate. The larvae even sealed off the holes they’d chewed, so I guess it was a good place. The authors estimated that some of the larvae travelled almost 5 meters to find such sites, which is pretty impressive. Anyway, if you have PVC irrigation in your citrus grove, make sure to get rid of suitable (Fabaceous) host plants.

Caterpillar of Brazilian skipper (Calpodes ethlius) on canna lily (Canna sp.)

Caterpillar of Brazilian skipper (Calpodes ethlius) in the process of constructing its leaf-fold retreat on a canna lily (Canna sp.). Here’s a closeup of an older larva (after I separated the leaf folds). The BugGuide information page says that the larvae can forcefully eject frass, perhaps as predator deterrent. Always a nice trick.

Larva of yellow-vested moth (Rectiostoma xanthobasis)

Larva of yellow-vested moth (Rectiostoma xanthobasis) found hidden in between two oak leaves tied together with silk. Unlike many moths, the common name here is spot-on descriptive and worth a look. There’s a nice description of this species on page 47 of Marquis et al. 2019.

Caterpillar of Zarucco duskywing (Erynnis zarucco) on scarlet sesban

Caterpillar of Zarucco duskywing (Erynnis zarucco) on scarlet sesban. NB: I opened up its retreat a bit to get a photograph. Here’s the photograph before.

Caterpillar in retreat constructed on scarlet Sesbania (Sesbania punicea)

Caterpillar in retreat constructed on scarlet Sesbania (Sesbania punicea). I’m stumped by this one, however. Possibly in the Family Tortricidae. I’ve photographed it before, and both observations are on iNaturalist … in case you can help ID it. Here’s a photograph of the retreat before I opened it up. Yes, I really enjoy opening up retreats to see who’s living there; so many caterpillars go unnoticed unless you do that.

Underside of splitgill mushrooms (Schizophyllum commune)

Splitgill mushrooms (Schizophyllum commune) growing on dead tree. Common but always beautiful when hydrated.

Small ball moss (Tillandsia recurvata) on trunk of a pine tree

Small ball moss (Tillandsia recurvata) on trunk of a pine tree. It grows on everything, even on exposed power lines. Part of their trick is the silvery trichomes that capture water and funnel it into the plant.

In the unlikely event you’d like to see even more photographs from Zolfo Springs, please see my SmugMug site.