How do I safely store my files?

A printed photo in an album can easily last 20+ years without deterioration.

Digital files are another story altogether: computers fail, hard drives crash, and CDROMs deteriorate.

Reliably and robustly storing digital files used to require an IT staff and rack of expensive hardware. Recent innovations have made this affordable and easy to do, even for non-nerds.

TL;DR: Make sure you have multiple copies, on separate hardware, and preferably at least one copy offsite. Newer filesystems available on NAS devices can provide additional data integrity safeguards. And remember, RAID doesn’t count.

The old axiom: “3-2-1 backups” #

The “3-2-1” backup strategy from several decades ago was one of the more common strategies for keeping files safe.

This strategy requires:

• at last three different copies of every file, with
• at least two different storage formats, and
• at least one copy offsite.

Nowadays, though, you can do better.

Why different storage formats? #

If you store your data on different formats, the hope is that the different formats will have different lifespans. When one copy fails before the other, it gives you time to make a new copy.

There are several problems with this line of reasoning, though.

Optical media

The first issue is that storage formats available to consumers are typically only hard drives and optical storage media (CD-R, DVD-R, or BD-R). The latest optical storage technology (as of July, 2020), is quad-layer blue ray which can store 100GB per disk. External hard drives are regularly in the 6-12TB range, which are 60-120x larger. Backing up a $150 12TB hard drive on optical media would be ridiculously herculean and expensive: it would take 120 BD-R disks, and $1,400 in media, using $12 M-Discs.

Why 3-2-1 isn’t enough: data integrity #

All commonly available storage formats eventually suffer from data degradation or corruption.

Data integrity is the big issue that “3-2-1 backups” doesn’t address: even if you’ve got several copies of your data, how do you know that each copy hasn’t suffered from data degradation?

An example of data degradation #

Your photos and videos are comprised of hundreds of thousands, millions or even tens of millions of bits of data.

If a handful of those bits are “flipped” due to storage defects or media degradation your file can become unreadable.

Here’s an example of an image with only a couple bits flipped:

JPEG bit rot after flipping 0, 1, 2, or 3 bits (source)

Note that the rightmost image contents are 99.999% correct, yet isn’t viewable.

Error correction codes #

Hard drives and optical disks use error correction codes to provide some amount of resiliency to bit rot. Typical ECC allows for several bits in a file to flip, be detected, and be automatically repaired.

ECC can only detect and repair a specific rate of bit rot, however. Data degradation in older storage media may exceed this rate.

Treating the symptom #

In developing PhotoStructure, we found that many of our photos on older hard drives had succumbed to some amount of bit rot. We taught PhotoStructure how to detect and skip over photos and videos that are corrupt, in the hope that you’ll have several copies of a given photo, and one of them won’t have bit rot.

But this feature just treats the symptom: this doesn’t fix the underlying problem.

Overcoming bit rot #

Several advanced filesystems, including btrfs and ZFS, support data scrubbing, which detects and repairs bit rot automatically.

Unfortunately, neither Windows nor macOS support these filesystems (and their newer filesystems, like APFS, still don’t detect bit rot). Time Machine and Backup and Restore don’t detect bit rot either.

So: how do normal people use these fancy new filesystems?

NAS to the rescue #

Network-attached storage (NAS) devices hold several large hard drives and quietly do their work safely storing your files. You can keep using your favorite OS, but you don’t have to worry about bit rot anymore.

Most NAS also support Docker, which is a way of packaging and running third-party applications, like PhotoStructure.

4-drive Synology NAS. Photo by Alex Cheung.

That sounds great. Which NAS should I get? #

• Synology is the easiest and most “plug and play” of these solutions. Be sure to get a newer NAS that supports btrfs.

  Synology’s proprietary hardware comes at a bit of a cost premium than if you built it yourself, and docker support via their UI is a bit limited (which can be overcome by using a tool like Portainer).

• FreeNAS (soon to be renamed “TrueNAS CORE”) is free and fairly easy to install. It uses ZFS, and supports a wide variety of hardware. If building a computer is intimidating and you’d like to try FreeNAS, the FreeNAS Mini comes pre-assembled.

  Note that expanding existing volumes by adding additional disks (which is simple on Synology and Unraid) is not fun on FreeNAS, due to limitations in ZFS. FreeNAS natively uses FreeBSD jails, which are technically excellent but decidedly less popular than Docker. More recent versions of FreeNAS have added docker support.

• Unraid OS, like FreeNAS, runs on hardware you already have, supports XFS and btrfs, has great docker support, but requires a commercial license to use.

• SnapRAID is different from these other alternatives, in that it’s not an operating system nor a filesystem: it’s an open-source application that runs on Linux, macOS and Windows.

  SnapRAID is the only non-realtime data integrity product here: it must be run periodically to build snapshots or check and repair data integrity errors. SnapRAID requires command-line expertise and manual setup.

(PhotoStructure has no commercial affiliation with any of these products. We’d caution against cheaper NAS brands, however, as lax security updates may lead to malware infections and data loss.)

Getting started with your NAS #

1. Consider getting a NAS that has 4 or more drives to offer redundancy and support data integrity checks.

2. Buy 50% more storage than you need right now so you have room to grow in the near future.

3. Enable weekly data scrubbing.

4. Enable snapshotting, if available.

5. Enable monthly S.M.A.R.T. self-tests.

6. Set up your NAS to either apply security patches automatically or notify you to do so.

7. Consider installing a virus scanner and malware detection package on your NAS. Synology has a “security audit” tool as well.

8. Make sure your router has recently-updated firmware

9. Use secure admin and user passwords. Using a password manager like Bitwarden makes this easy to do.

10. Configure your NAS to tell you if it has any errors: you don’t want any disks dying or backups failing without you knowing it.

How to get “at least 3 copies” #

Please understand: RAID is not a backup. Please read that post before continuing.

Satisfy the 2nd or 3rd copy of the “at least 3 copies” rule by copying the entire contents of your NAS to an external hard drive that normally stays powered down and offline. Do this quarterly.

Consider storing this drive near your emergency kit so you can grab it as you leave your house in case of an emergency.

This external drive can also reduce your dataloss if your NAS catastrophically fails, or if you get hit by malware like cryptolockers.

How to get “at least 1 copy offsite” #

Satisfy the “at least 1 offsite” rule by setting up your NAS to back up to a cloud service automatically. Backblaze and tarsnap are both well-regarded offsite storage solutions, and both have solutions that work with your new NAS.

If you don’t want to pay for cloud storage, you can set up another NAS (like at a friend or family member’s house). Both FreeNAS and Synology support NAS-to-NAS replication.

Make sure you configure the replication job to run in the middle of the night, and throttle network bandwidth so you don’t make your friends or family grumpy.

How do I back up files on my phone? #

Resilio Sync (for iOS and Android), SyncThing (for Android), and PhotoSync (for iOS and Android) will automatically back up your phone to your NAS at home.

You install the software on both your phone and your NAS, and then configure your phone to automatically back up to your NAS.

We recommend this approach, rather than uploading to a cloud service, simply to ensure your original files stay intact.

And now that your files are safe… #

You’re welcome to try PhotoStructure, a self-hosted photo management solution that runs both on your desktop, server, or NAS, using Docker.

Hi! I’m Matthew, the author of PhotoStructure. I’m a dad, amateur photographer, and author of open source, adtech, fintech, and edtech software since the turn of the millennium. He/him. I am building a self-hosted, safe, and fast new home for all your photos and videos.
Your memories deserve PhotoStructure. Sign up and try it today.

from Hacker News https://ift.tt/3iWtaxQ