News

Practices Make Perfect (Backups) Wednesday, February 21, 2018

We've been getting a lot of questions lately about the APFS image bug (recently documented by Mike Bombich, author of CCC) where, when an image container can't grow, APFS doesn't reliably indicate an error, which can silently lose data. Worse still, double-checking the data seems to indicate it's OK—even a checksum succeeds—until you eject, at which point the data is lost.

This looks to be another manifestation of a problem we've known about for years (and documented in the User's Guide back in 2006): when an image can't grow, whether due to disk full errors, or because the host volume doesn't support large files (eg FAT32/EXT2), writes to that image can fail catastrophically.

Failure is Inevitable, Success Requires Planning

Whether the problem is a bug in the OS or a hardware failure, failure is inevitable. But if you implement a successful backup plan, you can protect your data, even in the face of this sort of adversity.

So let's take a moment to talk about "best practices" for backup devices and methods.

Simpler is Better

This should be obvious, but is often ignored in the search for convenience (or expedience): the fewer layers there are between your computer and the storage device, the more reliable your backups are going to be.

What do I mean by that?

This: you should write directly to a local drive. That's going to be your most reliable, bootable solution. Don't write to an image stored on that drive. Don't format it as a foreign file system (like NTFS) and use a special driver to access it. Write. Directly. To. The. Drive.

Of course, things can still go wrong! But recovery will be easier and faster. Remember, a single bad sector isn't likely to take out your whole backup...but it could destroy an image.

You Can't Start Up From a Network Drive or Image

This is pretty basic, but important. You can only boot from a backup written to a properly partitioned and formatted, locally connected, non-network drive.

You can't boot from a disk image (it's not a "real" drive), whether it's stored on a local or network drive.

But I Want to Store More Than One Backup on a Drive!

If you need to store more than one backup on a physical device:

  • If you're on 10.12 or earlier, partition the drive into the number of volumes you need to back up. So, three source volumes to back up? Three partitions on the backup drive.
  • If you're running 10.13 or later, format the backup drive as APFS, and use APFS's very flexible "volumes" as your backup destinations, one per source volume.

We hear all the time that people are using images to "make more efficient use of space". That's the wrong thing to do—there's just no other way to put it. An image needs to be able to grow to its maximum size, unimpeded, no matter what. Images do not necessarily "hug" the data inside them, and may grow to full size even when their contents are smaller.

If the image cannot grow, it's quite likely you will corrupt that image and lose the data in it. So you need to have all the space available anyway!

High Sierra's native APFS volumes do this better, anyway. All the volumes share the same storage pool, and allocate that storage intelligently, managing that storage better than images ever did. Use them.

But I Want to Store Backups on the Same Volumes as Some Data!

Don't ever do this, except in a short term emergency situation.

What people seem to forget is that the data they're storing their backups alongside also needs to be backed up. And if you're storing your backup on that same volume, you're probably not backing up the data.

This is a mistake. A huge mistake that we see way too often.

Typically, we'll hear from people who work with large amounts of data, such as photographers, designers or musicians. They'll move their "older" work to an "archive" volume, and they'll want to store backups of their current work alongside that archive.

But they'll often forget they need to back up their archive too!

Don't be a goofus. Having an archive volume is fine. Keep it separate from your backup volume. And back it up.

But I Want to Back Up to a Network Volume!

OK, so we've covered the cases where you've got local drives. The best thing to do, to local drives, is to never use images unless you absolutely need to. Never.

But what about a network drive?

First, I truly believe you should never, ever use a network drive as your only backup. It's fine to have a network backup as a secondary backup. But by its very nature, it's going to be the least reliable one. It's not only subject to the potential flakiness of images, it's also subject to the flakiness of networking in general.

It's one thing when you try to access a web site and the page doesn't load. It's another thing entirely when the network "burps" when you are updating the structures of a backup volume.

You've probably already seen the result of this kind of failure. That message Time Machine displays that says "In order to improve reliability, we've started a new Time Machine backup"? That's because the image failed...and your backups have been completely lost. (Kind of a mild sounding message for a catastrophic failure, don't you think?)

So, here's my advice for networked backups.

If you're using a desktop, and you have a modern NAS device that supports iSCSI (like a Synology or QNAP), purchase an iSCSI initiator (like the Studio Network Solutions GlobalSAN initiator, or ATTO's Xtend SAN), create an iSCSI volume, format it natively for the Mac (as HFS+ or APFS) and back up to it directly.

Yes, iSCSI volumes are also containers, but those containers are managed on the far side of the connection, rather than the near side. As such, a near-side failure is much less likely to corrupt the container...and more closely resembles a typical local drive failure, which are almost always repairable by Disk Utility.

If you have a laptop, or can't use iSCSI, create a share that has more space than you need to back up, and back up to a sparse bundle. If your device doesn't support sparse bundles, use a sparse image.

Broad Spectrum Protection

Remember, you should never rely on a single backup device, or a single backup program. No matter what you're using for your backups...use something else too.

So, if you're using SuperDuper!, also use Time Machine, preferably to a separate device. Have multiple backups, with both SuperDuper! and Time Machine. And use something like Backblaze, so you get an offsite backup!

Backup Frequency

As I've indicated in the User's Guide, I suggest a daily, weekly and monthly backup with SuperDuper. It's easy to do - just make three schedules to different drives.

I set up my weekly and monthly backups "on connect" - that is, when I plug in the drive (which I keep separately), SuperDuper! launches, copies and then (since I've set an "On successful completion" action), ejects the backup and quits.

My daily backup is left unmounted but connected. SuperDuper! launches every day, mounts the drive, makes its copy, and then unmounts the backup.

I leave Time Machine on its default schedule, so it's doing a roughly hourly backup.

And Backblaze is backing up continuously.

Be Smart, and Don't Cheap Out

As I've said many times, no one was ever sad because they had too many backups.

Drives are inexpensive. It costs less then $100 (more like $60) to get a 1TB USB3 external drive. For $10 more you can get 2TB. 512GB External SSDs are less than $200. Spend the money. Your data's worth it.

Don't let your data loss story serve as a warning to others. Be the hero who planned ahead and saved your family's precious photographs. We're happy to be the invisible partner alongside you as you save the day.

Just remember the most important rule of all...

All Apologies Wednesday, January 31, 2018

Turns out, a few of you have drives that don't have low-level media IDs...something we thought all drives had. And they did, at least, all the drives we and the external testers tried did.

Well, now we know...and, now we don't crash. Sorry about that.

Fix now available in v3.1.4 - download away!

SuperDuper! 3.1.4

When I’m 64 Wednesday, January 31, 2018

Way back when, not so long ago, we transitioned the main SuperDuper! application to 64-bit. As I said then, we were 95% 64-bit, and 5% 32-bit.

Today, we're releasing SuperDuper! 3.13, which—after extensive testing—moves the copy engine to 64-bit as well. macOS 10.13.4 is going to start giving obnoxious warnings about 32-bit app usage, so we figured this was a good time to release the update, even without any significant user-facing feature changes.

I'd love to use this blog post to tell you that, because we're now 64-bit, we're 200% faster, but, well, no. Internal testing shows no significant gains or losses from being 64-bit. In fact, we have to do additional work packing and unpacking some file system structures, which are laid out (for legacy reasons) for 32-bit apps.

Of course, we fixed a bunch of other things, too. One specific change was to improve handling of source drives with bad recovery volumes.

We're not sure how people's drives get like this, but sometimes there will be a number of "Untitled" 650MB volumes on their drives. Disk Utility will indicate one of them is the Recovery volume, but when we try to mount and copy from it, we get an error, because the drive is bogus and can't be mounted.

Previously, we would stop and give an error. Now, we log the problem and generate a warning that explains how to resolve the problem. (Basically, you need to reinstall the OS, which refreshes the OS under your existing applications and data, and recreates/fixes Recovery.)

While we were at it, we fixed a few things in the last 10.9 compatible version (3.1.1), back ported some other fixes we made in 3.1.2, and updated it. Unfortunately, since it already had a version number, I just updated the binary. Bad form, I know, since there are now two 3.1.1s out there. But if you're using Mavericks, and can't create schedules with your existing 3.1.1, download 3.1.1 again and reinstall.

So yeah, we're older and we're losing our hair, but you really do still need us. Update inside the app, or download away!

SuperDuper! 3.1.3

12 Years a Bug Friday, January 12, 2018

Shipping software sucks.

It's not that we don't want to put the software in the hands of our users. And it's not that we're not proud of the work. We do, and we are.

What sucks is that you always ship with bugs. Always. Some of them you know about. Some of them you don't.

Anyone who does this for a living knows how this works. You keep track of everything you find in the software. Some things are reproducible. Some things aren't. You draw up general approaches for investigation for the latter, and propose fixes for the former.

You then prioritize things, hash out a schedule, and decide what's worth fixing and what you can, in the end, live with...for now.

"We need to fix this post 2.0"

During the run-up to SuperDuper v3.1.2 (available today, see below), we were dealing with this whole investigate-prioritize-allocate-test cycle, when someone brought a sheet to our attention that's been a known low-priority thing for years.

12 years, to be precise.

It's the basic "Stop the copy in progress" sheet that comes up when you want to stop a copy. It's something you can do by accident, and throw away a lot of progress, so we show a sheet that asks you whether you really want to stop.

Sensible, as far as it goes. But during the development of 2.0, there were a lot of potential cases that would have to be dealt with during a cancel, and while we were working through them, and how to present them to the user, a temporary cancel sheet was put in place.

A sheet that was immediately logged as "this has to be fixed" in our tracking database, because it was poorly worded, confusing and (please don't hurt us) had "Yes/No" buttons at the bottom. Enjoy this example of our overwhelming UI genius:

Stop Copy Sheet

So many problems with that sheet. It's wrong, it uses crazy terminology, it doesn't match the rest of the app, it uses Yes/No buttons. I don't even...

There was a lot to do for v2.0, and development proceeded, other tasks took precedence, this sheet isn't used very much, and eventually we shipped 2.0 with the sheet in place, with the bug tagged as low-priority, known-terrible, embarrassingly bad, "we need to fix this post 2.0".

That cycle repeated over and over. It was an easy fix, but other things took precedence. This stuff happens. Shipping software sucks.

Well, it was terrible, and it's now 12 years post 2.0. So for the new year? Start celebrating, because it was fixed.

New Stop Copy Sheet

More Fixes!

This version also fixes a bunch of other relatively minor things that should please one or more of you:

  • On some systems, the default settings didn't save if you had available snapshots and then went into Options, changed something, and exited.
  • If your drive had a "#" in its name, an erase-then-copy backup would generate an error.
  • Renaming a drive wasn't always reflected properly in the UI.
  • Sometimes the Stop button would beep and not stop
  • In some regions, snapshot date parsing could cause a crash at startup or drive selection

General cleanup, nothing too exciting, but a fix is a fix, and now they're available to you. Download away!

Download SuperDuper! v3.1.2

Snapshot Surprise! Friday, December 08, 2017

We've got something under the tree for all of you, and you get to open your present early.

Executive Summary

SuperDuper! 3.1 is out, and now allows Time Machine-like restoration from bootable APFS copies.

Wait, What?!?!

Oh, decided to join us for the details? Welcome to an ocean of words: prepare to get your feet wet!

Well, I told you there would be more surprises in store, and, so, here we are. Since the release of 3.0, we've not only been copying from a snapshot when using an APFS source. In addition, as some of you have noticed, we've also been creating snapshots on the backup.

This is why.

Turn Back the Clock

In SuperDuper! v3.1, you may notice a small change to the Copy Now button: a little up-triangle glyph. If you've selected an APFS volume as your source, you can click that triangle, and up pops a list of the snapshots available to copy from.

Copy Snapshot Selection Popover

This means you don't just have to copy from the drive as it is "now" (the default choice). You can select from any existing snapshot, and we'll copy the state of the files as they were at that time.

But why would you want to do that?

Well, in High Sierra, the system takes a snapshot of your drive before it installs a software update. Having a problem with an update and want to check if it was different before, but forgot to back up first? No problem: use SuperDuper! to copy from the snapshot taken just before the update was installed, start up from the copy, and check the behavior.

Even better, if you're backing up to an APFS volume as we encourage, this works on a bootable copy, too! Since v3.0, we've been taking a snapshot of the backup volume before every copy we make. That means there's not just one available backup on the drive—if you've been using Smart Update, there are many! Start up from your backup drive, click the triangle, and you'll be presented with a list of available snapshots. Pick one, "Copy Now", and you've restored a day ago's backup, or a week ago's.

But Wait—There's More!

In fact, not only can you use SuperDuper to copy from these snapshots, you can even open Time Machine, select your backup volume, and see older versions of files, deleted files - they're all being saved, automatically, every time you Smart Update. Even though you're not backing up your backup to Time Machine. Because that would be silly.

This new feature is not intended to replace or replicate what Time Machine does. Snapshots are managed by the system, and at present they have some lightly-to-not documented constraints. You need to have about 20% free in a container to create a snapshot, and the system consolidates and removes snapshots according to its own logic.

As I've said before, an effective backup strategy should have multiple parts: SuperDuper!, Time Machine and an online service. The more (and more varied) backups you have, the safer you are. This new feature gives SuperDuper!'s bootable backups some cool new capabilities and an additional measure of protection should you make a mistake.

Now How Much Would You Pay?

Pretty awesome, right?

Needless to say, this is the kind of feature we love to introduce. It's completely transparent to the user, and is built on the extensive work we did to support APFS in v3.0. That support wasn't done at a surface level: we worked hard to research and make use of the more advanced features, like snapshots, and planned for the future in ways that truly make the backup experience better for everyone.

Including you. Update away!

Download SuperDuper! v3.1

It’s Not a Turkey Monday, November 20, 2017

One of the weird parts of doing a public beta, as we did for 3.0, is that the release of the final version doesn't quite feel the same. So, even though it's only been two weeks since 3.0 came out...it feels like it was released months ago.

The 3.0 release has actually gone incredibly smoothly. The extended beta managed to attract a large number of diverse system configurations, and that helped us ensure that any post-release issues would be...obscure. And, indeed, they have been!

Pulling the Right Levers

A program like SuperDuper! needs to make choices about how to interface with the system. There are many different layers, and we try to choose appropriately, depending on what we're doing.

To ensure a maximum level of compatibility, it's incredibly important to work at as high a level as possible for any given activity. For example, when SuperDuper! needs to erase a drive, it would be a huge mistake to actually jump down to the driver level and start mucking with drive structures.

Instead, we do the sensible thing: we ask Disk Utility to do it for us.

Talk to the Command

Disk Utility, though, is what's called a command-line tool: you can run it yourself, from the Terminal application, if you want to: type diskutil to see what it does, and man diskutil for its documentation. Much of macOS is built upon these Unix tools, and (where appropriate), so is some of SuperDuper.

When you talk to a tool like this, though, you have to both check its "return value" (basically, a rough "I worked!" or "I failed!"), and its text output (which says why it worked or failed, or gives you an answer to a question you asked).

Some of these tools (diskutil, hdiutil, etc) will format their output in plist format: basically, a structured XML document that's far more predictable, easier to parse, and more reliable. So rather than returning "Drive formatted as HFS+", it might return (intentionally simplified)

<format>hfs+</format>

which is obviously less friendly to people, but a lot better for programs trying to figure out what's going on.

This plist output's advantage is that its structure is strictly defined. It has a prologue, a series of tags and values, and an end. This is all guaranteed.

Samsung!

So, when a few users started getting some crazy, inexplicable errors (typically, an error indicating that a volume that was there couldn't be accessed), we were shocked to find, after doing some extensive investigation, that a Samsung SMART driver (based on an open source project) was actually polluting diskutil's output with a bunch of debugging/progress junk.

Needless to say, this prevented parsing of the output, and resulted in a stream of invective that did not diminish for many minutes. So frustrating. But fixed.

Various Improvements

That was the big one. But we also fixed a number of other things:

  • Finally fixed the update animation bug! So, if you install 3.0.1, the next release is going to show up properly in all cases. This time for sure!

    This took such a long time to fix because it was entirely timing related and didn't happen in the debugger. But, we figured it out. So embarrassing. Gonna miss you, you little rascal. (Please don't darken our UI door ever again.)

  • Settings conversion related quit issues
  • More reliable sleep prevention during a copy
  • Supports encrypted destination volumes on 10.12
  • Restored 10.9 support

I also took some time over the past week to turn on SSL for Shirt Pocket, so site access and downloads are all being done securely.

Happy Thanksgiving

So, with that, from our families to yours, have a Happy Thanksgiving, and enjoy SuperDuper! 3.0.1, which is available via the built-in updater, from the web site, or here:

SuperDuper! 3.0.1 Download

Bad Craziness, Good Release! Wednesday, November 08, 2017

The short version

Blah-blah-blah what-the-heck-are-you-talking-about-Dave so many words oh, 3.0 is released? Got it.

The wordy-but-interesting version

Sometimes, on macOS, you run into behavior that really makes you go "huh".

An example: as you likely know, you can create a disk image with Disk Utility and format it as APFS. Which works fine.

And, if you've been reading the blog, you probably also know that APFS volumes reside in a container, which itself resides on a GUID partitioned drive. So, multiple layers: drive, partition, container, volume. You can see all those in Disk Utility.

But once you create the image, what you see in Finder is simpler: a single ejectable volume, with the name you gave it. So, you can open it, copy to it, eject it, and there you go.

Great, right?

Not so much

Ah, but programmatically, things are different.

If an application opens an image the old way, and then unmounts its volume using diskutil, umount or hdiutil, it doesn't do what it used to do.

You can easily see this in Disk Utility, too. After you create the image, try selecting the volume and clicking the eject button. When you do that, you'll see that while the volume ejected, the image is still sticking around in the sidebar. And if the image is hosted on an ejectable drive or network volume, and you then try to eject the host volume, you'll find the it can't be ejected.

Worse, if you force eject the host at this point (hey, folks—don't force eject!), you can damage the image.

Not good.

Huh.

Worlds within worlds

So, what's going on is that there are still two things "attached" from the image: the container and the partition. They don't get "released" after all the volumes are ejected for some inexplicable reason. And so, any application that mounted and ejected images can no longer rely on the previous behavior.

Mischief, managed

Fortunately, this is another situation that SuperDuper! handles for you. If you back up to an image file, you don't need to know what I've just explained: we deal with the details, and eject all the "parts" of an APFS image without any manual intervention.

As you've heard many times from me: it just works.

So while you may not want to know why, or how all this is accomplished...isn't it comforting to know that someone's figured it out and has your back?

#Finally

With that last bit of explanation, I'm happy to say that we've reached the end of this particular voyage. SuperDuper! 3.0 (release 100!) is done, and you'll find the download in the normal places, as well as in the built-in updater, for both Beta and Regular users.

SuperDuper! 3.0 has, literally, many hundreds of changes under the hood to support APFS, High Sierra and all versions of macOS from 10.9 to the present.

SuperDuper! 3.0 is the first bootable backup application to support snapshot copying on APFS, which provides an incredible extra level of safety, security and accuracy when backing up. It's super cool, entirely supported (after all, it's what Time Machine uses...and it was first overall), and totally transparent to the user.

It doesn't stop there, of course. We're already in progress on the next version, and we've got some really great things planned.

Thanks to all of you who participated in the beta, because the final release couldn't have happened without you.

And to those of you who didn't participate in the Beta: you are going to love the final result.

Without further ado, download SuperDuper! 3.0

Enjoy!

The Beta 7 Spooktacular! Wednesday, November 01, 2017

Gee, do you think the original plan was to release this update on Halloween? Spoooooky!

Poof!

When “Erase, then copy” is selected, SuperDuper! has always endeavored to match the format of the source. That is, if the source is Mac OS Extended [HFS+] (Journaled), and the destination is Mac OS Extended (Case Sensitive), on an erase, the destination would end up matching the source, adding Journaling and removing Case Sensitivity.

As I mentioned in one of my first High Sierra posts, we always knew this wouldn’t work when going HFS+ to APFS: you just can’t format an APFS container as HFS+ safely, since it can affect more than just the one volume.

We’ve discovered during our broader Beta rollout that, due to weird bugs in Disk Utility, formatting an HFS+ drive as APFS is unreliable too. Sometimes the drive just “vanishes” and doesn’t re-mount. Sometimes it fails for no reason. Sometimes it makes the one volume unreadable until it’s erased again.

It’s kind of scary crazy. But that seems to be par for the early version High Sierra course. It’s relatively solid in the main, but really weak at the margins.

...But I Don’t Do That

So, we’ve made a change in our “magic” behavior to deal with these issues. Basically, we continue to do what we’ve always done. But if the “major” part of the file system doesn’t match (that is, HFS+ and APFS), we just maintain the existing format of the drive.

So, if you copy APFS to an HFS+ volume with erase, we’ll erase the destination but leave it as HFS+.

Hopefully we’ll be able to change this as macOS stabilizes further.

Enter the Void

I regret to say that we’ve fixed the “black flash” bug, so you’ll no longer have a moment to stare into the blackness as it stares back at you.

Perhaps that’s for the best.

Spaced

We didn't notice (but should have) that our various numbers were now proportionally spaced rather than monospaced since we built against a newer SDK with different targeting when we made the UI 64-bit.

Of course, someone—and you know who you are—noticed right away. Embarrassing, and fixed.

It’s Log, It’s Log!

One of the things we’ve done is improve our diagnostic logging, so when people send in a bug report, we get a better feeling for the state of the drives attached to the system, and the problems that might be going on with the system, should there be any.

We changed this a bit to only collect the extended log information when copies fail.

In earlier OS versions we would include a bit of the system log, so we could see what hardware was doing at the time of the copy. That stopped working in recent macOS release, so we've changed it to use log show.

The problem is, log show -last 1m, including a kernel predicate so it only returns low-level kernel logging, can be slow. Not only that, but due to bugs in the logging subsystem, it can incorrectly return much more than one minute of logging. We've seen it return almost a gigabyte of log data!

Even though we also limit the data to a short snippet at the end of all that text, we have to wait for it to be collected, and when the system grabs too much, it can take a while to retrieve and then discard the extra stuff.

The end result of all this is that we only do it when copies fail, and we put up a modal sheet while it's going on.

And Also, Too

Previously, when users sent in a report, we only included the log information when they reported an "Error during copy". We were a bit too clever for our own good there, since people didn’t always choose the right issue type, and then we’d have to ask for the report to be sent in again. We now always attach the log when using “Send to Shirt Pocket”.

These are small things, I know, but they'll help me help you more quickly and efficiently, should you ever need support.

I Can't Quit You, Baby

There are still some issues with quit that we're aware of...we're working on them.

Various and Sundry

There has also been a bunch of polishing and fixing things at the margins, thanks to user reports from highly unusual configurations, but as before, things are working really well on the vast majority of systems, which we’re really happy about. It’s just kind of boring to write about how great things are working for most people!

So you get the point. And now you get the results. Download away!

Download SuperDuper! 3.0 B7

Workaround Turtle Wednesday, October 25, 2017

Any program of sufficient complexity ends up having to work around behaviors that are undesirable. That might be a usability issue caused by default framework behavior, due to the interaction model selected by the designer. Or it might be an OS issue that only shows up in some situations (the recent iOS calculator bug is a good example of that).

Most, though, don't have to deal with the huge variation of hardware, software, OS, etc that a program like SuperDuper! has to handle.

The vast majority of the beta cycle has been focused on getting coverage of as many systems as possible. With a broad range of different setups and configurations, we can then find issues in the way macOS 10.13 is dealing with those configurations, and try come up with workarounds so, when when we release the GA version of SuperDuper! 3.0, the broader user community will have a smooth experience.

A lot of these changes have been focused on attached volumes, incorrect state information being returned by macOS calls or tools in some configurations and states, and coming up with ways to prod and probe until macOS returns the correct data.

It can be challenging to do, since you don't want to break "working" setups, and any workarounds should only come into effect in situations where they're needed. That way, when the OS problems are fixed, everything continues to operate.

All of that is a longwinded way of saying, we're happy to release Beta 6 of SuperDuper! 3.0, which has even more workarounds for macOS issues, covering a broad variety of situations we've seen in the past week or so. (It's workarounds all the way down!)

The new beta should work better on every system, and it fixes a problem with the beta autoupdate mechanism as well...so the next release's autoupdate should work better.

Download away, and thanks!

Click to download SuperDuper! 3.0 B6

Fraternal Twins Monday, October 23, 2017

(I really wanted to embed the teaser trailer for Peter Greenaway's A Zed and Two Noughts above, but alas it's a tad NSFW. Really great movie, though.)

Content warning: some strong language is used in this post. Not without reason.

So, here's something that you might not expect me to say:

Because of the way APFS "file clones" work, no program operating at the file level, including SuperDuper!, can make an exact physical copy of every possible APFS volume.

That's right. There are cases where we can't make an exact copy of your APFS volume. And Time Machine can't either. Nothing can.

That doesn't mean the copy is bad! It just means it might not be as space efficient as the original.

Doomed! Doomed! (Well, maybe not so doomed.)

Remember back in this post where I talked about the demo where Craig showed how fast it was to copy a gigantic amount of data?

I explained back then it was because the files aren't being copied. Rather, APFS creates new directory entries for the files, but references the same data blocks. So nothing is copied, which is fast. This is documented in Apple's APFS Guide.

From the user's perspective, these are different files. They're not like hard linked files, where changing one copy changes the others (not that most users know what hard links are). As far as users are concerned, they're totally separate, even if, at the file system level, they share the same data.

In APFS, if one of the cloned files is changed, even by a single byte, that changed data 'splits off' from the rest, and the files are now physically, and not just logically, separate—some of the data blocks now have two copies: the original ones, and the modified ones.

This process continues as the files diverge further.

The amount of logical drive space taken by the copies is twice the original, of course. However, the amount of actual space taken is, effectively zero...until the files are changed. At which point the space taken is the original plus the number of modified blocks.

This is all handled for you by APFS. You don't really have to think about it.

Quantum Theory?

Until you do have to think about it.

Consider this case: you have a 1TB APFS drive, and three 333GB files, named A, B, and C. So the drive is nearly full.

You then create a folder, and copy the three files into that folder with Finder. Of course, you'd expect the copy to fail and the drive to fill...but it doesn't.

In fact, if you look at the volume's size with Get Info, you may be surprised to see it has the same amount of data on it as was there before you made the copies. But, if you look at Finder's size for the folder, you'll see you now have 2TB of data on a 1TB drive. It's like magic!

At least until you change one of the files.

But now, select those files and folders with Finder and try to copy them to another 1TB drive. What happens?

The drive fills.

A Shitty Analogy

You can't copy it to the same size drive! But why?

The reason is there's no (public) way to find out that two files are actually sharing the same data (they might even only be sharing some of the same data, as I explain above). So, when copied, the "clone" relationship is broken, as is the ten-pounds-of-shit-in-a-five-pound-bag magic. You now have a full ten pounds. It doesn't fit...so you end up covered in shit.

But What If You...

Yes, we know:

  • What if you kept track of checksums of every file on the drive, and then made "clones" for each file based on whether the files had the same data?

    Leaving aside how ungodly slow that would be (think about trying to match ten million files to each other via checksums every time you copied), remember that cloning operates at a block level, where some blocks may be shared and some may not be. At a file level, it just won't work.

  • How about using hard links?

    That won't work either: clones and hard links are not semantically equivalent at all, since changing one of the hard linked files would change all of them, by definition.

  • Just ask the file system!

    While there are APIs to create clones, there's nothing there to find out whether two files are clones... and also, the shared data is at the block level, so still, no.

  • Time Machine does it!

    Well, not really. Time Machine does seem to be able to determine if two files are clones (which I assume it's doing with private APIs, because I can't find any documented APIs to determine if two files are clones). When it backs up cloned files, it uses hard links to represent them (since HFS+ doesn't support clones, and Time Machine can only back up to HFS+ volumes), and when it restores, it checks to see if those files are clones (which it tracks in a special database), and restores them as clones to APFS...unless they're restored to an HFS+ volume, where all bets are off.

    But even in the best case, restoring to APFS, when files get 'separated' when they're changed, again, only the part of the file that was changed is separate. The other blocks are still shared. So even though they've jumped through hoops to maintain the clone relationship, there are lots of cases where Time Machine's own copies will increase in size too, and it happens more and more as the files diverge further.

  • You guys are so smart, you figure it out! Why are you asking me?

    Geez, don't get so defensive!

We're All in this Together

So, as you can see, given the low-level behavior, there's really no solution, even when you're Apple.

What does this mean for you? It means you can get in cases where data that fits on a source drive won't fit on a destination, even when the drives are exactly the same size.

To avoid problems, you need enough space to store the full logical size of the data (that is, with all the "clones" separated) when you copy, unless you're copying the entire container at a sector level.

We Good?

Again, this doesn't mean your backup isn't good! It is! It has all your data!

What it does mean is that the data isn't stored as efficiently on the backup. So, it might not fit on your drive when you back it up. And it also might not fit when you restore, if the backup ends up larger than the capacity of the source.

That's easy to check, and the solution is also pretty easy: have plenty of free space on your drives, folks. It's always been good advice, and given all this hidden behavior that happens with cloned files, it's even more important with APFS.

OK, back to the code mines...

Page 1 of 6 pages  1 2 3 >  Last »