The Noise Jockey Data Storage Guide, Part One: Hard Drives

Bits and bytes are mighty tasty. How does one store them for maximum freshness?

Portrait of the Artist as an IT Professional

Artists, designers, composers, mixers, and audio folks of all stripes must be conversant with the tools of their trade, and in this digital world, that means playing some role in managing hardware and software. This is where your hard-won creative output of blood, bits, and tears will be stored…and possibly lost.

This article is meant to help frame the challenge of selecting hard drives for one’s own home studio. I’m no IT professional, but I’ve been dealing with digital multimedia production for nearly 20 years, so I’ve at least got some perspective as a creative professional. I’ve seen my share of hard drives literally catch fire, glitch out, play the national anthem, and just simply stop working, sometimes one a day for three days in a row. I’ve had to manage IT issues from single machines to small clusters to an entire small studio. What follows, then, is what home-studio creatives of all stripes should consider when thinking about storing their creative output on hard drives.

This article is won’t give you a fish to eat for a day; the goal is to teach you how to fish, so you can feed yourself for a lifetime. Very few specific purchase recommendations will be made so that you can simply make a more informed decision based on your own personal needs.

There are loads of hard drive buyer’s guides online, and Wikipedia is a definitive source for things like RAID levels and hard drive terminology, for anyone who’d like to read more on this topic.

Hard Drive Types

There are two general kinds of hard drives: Mechanical and solid state.

Mechanical hard drives are what we all know, love, and often abhor. They have metal plates with data magnetically encoded on them. They are very susceptible to jostling, physical damage, and magnetic damage. However, they are also reasonably fault-tolerant. When things go awry, utilities or the drive itself can try to work around the problem. They also can give clues as to their imminent failure, such as odd noises, slow mounting or read/write times.

Solid state hard drives are essentially large flash memory devices, like Compact Flash cards. There are no moving parts. They are lightning-fast, small, light, and much less prone to physical or magnetic damage. They suffer from two things: High price and catastrophic failure. They’re extremely expensive per gigabyte, and when a solid state drive fails – not common, but they do, I’ve seen it – it fails immediately, irrevocably, and totally. To use a materials science analogy, mechanical hard drives are like aluminum, and solid state hard drives are like carbon fiber.

Of course, this refers to just the drive itself, which usually lives inside the chassis of a computer or in an external enclosure. Enclosures, in my experience, are just as likely to fail, and often more so, than the drives themselves. Fans, power supplies, and the dreaded power-brick inline transformers of these enclosures are all possible failure points. Knowing how to get a drive out of a dead enclosure is a good skill to have!

Enclosures also tend to influence how these devices are attached to our computers. This naturally takes us to our next topic.

Hard Drive Interfaces

At the time of this writing, USB 2.0, Firewire 800, and eSATA are the most common hard drive connector standards. More exist, from legacy formats to enterprise-level: Firewire 400, iSCSI, Fibre Channel, and more. A little internet research can help you learn more about those other flavors. However, for simplicity, I stick with drives with all three of those initial connector types. USB 2.0 is so universal that I can pretty much take a drive and plug it into anything, which has merits for massive file transfers and emergencies. Firewire 800 is the MacOS X standard broadband connectivity standard (the new Thunderbolt connector notwithstanding), and has decent speeds. eSATA often requires third-party cards to support, but when it’s available it’s the fastest kid on the block. Having at least two options, and perhaps all three (or more), can help leverage your hard drive investment in case you change computers, or computer platforms, in the future.

Types of Hard Drive Storage Needs

Not all hard drives are created equal. Likewise, not all storage needs are the same. It turns out that what you want for your internal system drive may not be the best kind of drive for your daily storage needs, or for your long-term backup needs.

Below, I’ll cover five types of storage needs: System and app storage, scratch disks, external working storage, and long-term project storage.

System and Application Storage

This should be all one’s internal boot drive should be responsible for. This lets the drive stay clean and defragged, since only log files, preferences, and the like will be written to it. Speed isn’t the top concern here, but it does help; a solid state hard drive might boot a machine and launch apps up to twice as fast as a standard 7200rpm hard drive. Most laptops ship with 5200rpm drives, so upgrading to a 7200rpm will make a difference.

These drives should always be backed up with bootable copies. This means using a disk cloner application (I prefer SuperDuper!) to ensure that the copy is fully bootable, so if something disastrous happens, you can just plug in your bootable system backup disk and it’s all good.

Scratch Disks

Scratch disks are where apps – and sometimes RAM-constrained systems – write temporary files to facilitate things like undo, revert, and rendering crossfades or audio effects. The funny thing about scratch disks is that you want ultimate speed and don’t need backup of its data, since ideally no user-written files will be stored there at all. This is theoretically a good use for solid state hard drives, which boast lightning-fast access but aren’t as large as standard hard drives at the same price.

In fact, the smaller the drive, the faster the access speeds will usually be. Scratch disks should be small and as fast as possible without compromising their utility.

External Working Storage

This is what most of us know, love, and hate about hard drive storage. What will you be writing to and  reading from for your actual work, like session files? There are a number of approaches to this, which will vary based on each artist’s risk tolerance and budget, but it all boils down to balancing speed, capacity, reliability, redundancy, power consumption, and cost.

• Speed. 10,000rpm drives can be fast, and solid state drives are faster still, but one pays for such speed. Striping multiple slow drives together as RAID 0 can really boost read and write speeds, at the cost of buying two or more drives and not gaining anything in the redundancy department (see below). But some applications, like video editing, demand the fastest disk access possible.
• Capacity. You need a fair amount of space for your files, especially if you’re involved in high-definition audio and video, but sometimes too much space will cause problems. Pro Tools, at the time of this writing, can only play back from, or record to, volumes of 2TB or less. Most other apps do not have this limitation.
• Reliability. All drives will fail. The more drives you have, the more frequently you will have drive failures, statistically speaking (so think about this when you consider multi-disk RAIDs, below). The cheapest drives usually aren’t the most reliable. Many consumers are unaware that there are specific enterprise-class hard drives out there that are held to higher quality standards than others, and these can be good values relative to piece of mind.
• Redundancy. How many copies of your data do you have? The answer must be “more than one” if you are to be reasonably secure in preserving your life’s creative work. Many folks simply buy one drive and back it up to a second drive of the same size. This works, but it’s not a constant process, since this is done manually or via scheduled synching. Others opt for the RAID option (level 1, 5, 6, or 10), which ensures that every write to disk is duplicated automatically. One drive goes down and, based on what RAID level you’ve chosen, you either have a backup or the remaining drives can rebuild a replacement disk automatically. RAID 1 is pretty slow, while RAID 5 (one of the more popular configurations these days) is a compromise between increased speed and distributed copies of data.
• Power Consumption. Part of the TCO – total cost of ownership – of a hard drive is how much you’ll pay on your electricity bill for using that hard drive. There are now “green” versions of hard drives that can draw less than three watts when not in use. If you’re running one or more RAIDs with lots of platters a-spinning, this can really add up!
• Cost. Greater speed, reliability, and redundancy all cost more. Your workaday storage isn’t a good place to skimp, but do remember that hard drives won’t last forever. Also remember that every external hard drive you buy, you’re buying a hard drive that you can’t remove yourself without voiding the warranty, and you’re paying for another power supply and case that isn’t working for you when it’s unplugged for long-term storage.

Long-Term Project Storage

That huge project is over; you’ve got raw recordings, tons of track bounces, massive sessions. Where do you put all that when you’re done?

The first choice is online or offline storage…this means whether it’s on a drive that’s normally spun up and plugged in, or if it’s archived on physical media that is being stored somewhere that’s not attached to your workstation or network. This “hot” or “cold” storage, as I tend to call it (as the term “online” means something different these days), depends on how often you might need it.

At the moment, there is, for better or worse, no other storage device that holds as much for as long for the price as a hard drive. Hard drives are also more shelf-stable and reliable than DVD-R’s. They may be a little heavy, but they are small, and the cost per gigabyte is low.

This has given rise to the hard drive dock, into which bare hard drives are inserted, mounted, copied to, and taken offline. This provides redundancy and backup at a very low cost per terabyte, and by spending most of their time offline, bare hard drives should last a good deal longer than a hard drive that spins all day long. Speed doesn’t matter as much in this usage, either: Sure, backing up to them may take a while, but do it when you go to bed at night and it won’t make a bit of difference, and buying slower drives can save money. Hard drive docks are also great for troubleshooting: If an enclosed hard drives goes down, you can extract it and pop it in a dock to see if the fault lies in the drive or the enclosure’s power supply.

The second decision is, if you opt for “cold” storage, whether you store the data locally or on a remote network (“in the cloud,” as they say these days). The primary challenge with cloud-based storage is transfer speeds. Every recording session I do, no matter how short or informal, will usually generate at least a gigabyte of data. Even though I have a fast internet connection, it’s asynchronous, as most of them are: The upload speeds are slower than the download speeds. For my purposes, I can’t make cloud computing work given how much data I generate. Note, however, that the new (at the time of this writing) service Gobbler is optimized specifically for Pro Tools users.

It’s great for backing up your personal and business files, though. Speaking of which…

Backups

Backups are usually considered copies of your data that aren’t in daily use. Therefore, the extra disks sitting in your RAID aren’t backups; they’re for daily redundancy in case of failure. Backups are meant to be retrieved when some seriously bad juju goes down. In my opinion, everyone needs both onsite and offsite backups. Why? Two simple reasons.

• Offsite backups, depending on how far away they physically are, won’t help you when a disk dies and you need to immediately repair or replace it when you’re in the middle of a project. I store mine more than 20 miles away from my home, and it would take 2 hours to retrieve them.
• Offsite backups, by their very nature, are kept less up-to-date than onsite backups…unless your backup drives are co-located in a data center, but that’s too complex and expensive for most home studios.
• Onsite backups won’t help you if your studio is burglarized, burns down, floods, or collapses in a tornado, tsunami, or earthquake. These are not paranoid doomsday scenarios. They can and do happen all the time, almost anywhere.

There are too many backup strategies to list, so let’s stay focused on hard drives for this purpose. I prefer to match capacities between the real, “live” drive and the backup, but to use drives from different manufacturers, or at the very least, have the live drive and the backup drive be from very different serial number ranges. This way if there’s a manufacturing defect in a drive, you can at least know that the other drive has less of a chance failing around the same time for the same reason.

Whether you use a hard drive dock with bare drives, fully duplicative external hard drives, or even cloud-based network backups, do it. One massive data loss on a project without backups can cost you money, but more importantly, it will betray a client’s trust in your professionalism.

Local vs. Network Storage

This article isn’t going to focus on network-attached storage, or NAS, devices for larger studios, so I won’t get into robust, scalable rackmount RAID units, JBOD, Fiber Channel and iSCSI devices. However, NAS can have a place in the home for a few reasons.

Many of us with home studios share a network with housemates, significant others, or families. Some of us have multiple computers within our home studios. In either case, sometimes multiple machines need to access the same files, and needing to have one computer dedicated serving files just isn’t cost-effective or sensible. That’s where NAS comes in.

NAS devices are either hard drives that plug into other devices via USB to act like NAS devices, like the Apple Airport Extreme Base Station, or are hard drives with embedded computers inside of them, and they plug into an Ethernet network just like a computer would. The only reason to consider this approach is for multiple computers to access the same files without needing an external host computer to provide the shared hard drive volume on the network. NAS devices and drives largely follow the same purchase decision points as single drives to, above, but backing up a very large NAS device can be tricky and/or expensive, so be sure to think about what your backup strategy will be before you traipse off and buy a 16TB rackmount NAS!

However, the speed of accessing that NAS is dependent on your Ethernet network. With gigabit network switches being so standard and cheap, this isn’t a big deal for small files. Factor in hefty media files, though, like audio or video and the fact that many studios rely on WiFi, which is further bandwidth-constrained compared to wired gigabit Ethernet, NAS can definitely try your patience.

So…How do I do it?

I don’t skimp on hard drives or redundancy, but I do absolutely have a budget within which I need to operate. How I have my hard drives arranged and how they’re used will be the topic of my next article. Stay tuned!