Somewhere in the world, it’s November 22nd.

SuperDuper! v2.0 emerged from our engineering labs and was placed into the hands of our first external testers one year ago today.

Over this past year, we’ve released—on average—a build every two weeks. We’ve done thousands upon thousands of test runs, and ended up with over 80 individual testers who put the product through its paces and offered incredibly valuable feedback. We’ve torn into nearly every aspect of SuperDuper!, and have tried to improve the experience in as many ways as we could.

It’s not an easy thing, writing backup software—and it’s not easy to ask testers to depend on a Beta version, no matter how thoroughly tested it was before put into their hands. Every tester contributed to improving the reliability, usability and polish of the product as the months went on.

We had 13 external testers who ran over 200 individual backups each over that time, and one who ran over 600 (655, as of this writing). Major props to each and every tester: you all know who you are, and you should feel incredibly proud.

So, we’re coming to the end of the long process. The web site is ready to go. The press release is written. The documentation is complete. The software is just about as ready as we think we can make it.

Get ready, people. SuperDuper v2.0 is nearly in your hands.

We hope you love it as much as we do.

I come before you today to sing the praises of Bruce Lacey, my collaborator on SuperDuper!

Bruce doesn’t get a lot of direct play in these pages, mostly because it’s my personal blog, but SuperDuper! wouldn’t exist at all without him.

A few years ago, Bruce and I were both testers on some software that most of you use on a daily basis. Bruce had written a little tool—“Seed Volume Utility”—that made dealing with testing a lot easier. What was great about SVU is that it had a terrific copying engine, fast and reliable. Good stuff.

I saw a lot of potential there, and I approached Bruce with a proposition: if he’d agree to have me publish the program, I’d redesign the UI, write documentation, take care of marketing and support, and work with him to do the product planning. Basically, he could concentrate on the programming—the fun stuff—and leave the rest to me.

I’d been doing this with my own products for a while (since 1983), but had always shared in development, so this was a new thing for me. Bruce thought it over for a while, and—even though we’d never met in person—he took a chance, and agreed. We’ve been working together ever since.

You may not see him up here every day, but you certainly benefit from his hard work. Bruce spends a lot of quality time implementing, polishing and improving SuperDuper!—not to mention dealing with my constant tweaking of wording, behavior, layout, functionality—and always with dedication, good humor, and a shared focus on producing a high quality product.

So, while I might use I a lot when I talk about the stuff that’s going on with SuperDuper!, behind the scenes here Bruce is working like crazy to turn our shared product dreams into coded reality.

Here’s to you, Bruce!

As of about five minutes ago, I finished the new SuperDuper! v2.0 User’s Guide. Barring any egregious errors (there’s always one), I won’t have to revisit again for a little while… a relief.

I hope that all of you out there in Blogland find it an improvement and if not—when you get to see it—please drop me some feedback.

On to the next task!

Yeah, clichéd title, sorry.

Well, the Computer America show went pretty well last night—it’s amazing how fast an hour can vanish. For those interested, here’s an archive of the show that’ll be up for about a week.

Thanks to Craig and the staff of Computer America for asking me to appear!

Writing a manual can be fun, or it can be tedious, but it’s usually a combination of both, which is where I am right about now. But Fun Land is now clearly visible in the rear-view mirror.

At this point the guide is 64 pages long, and it’s pretty much content complete. 64 pages for a program that has a total of 8 windows or so, and maybe 30 controls. Shouldn’t this thing just say “install, and have at it—best of luck, we’re all rooting for you”?

It’s not so much the UI that needs so many pages: as you (may) know, I’ve tried to make the UI as simple as possible. And I think I’ve done a decent job with that. But you can do a lot of things with that UI, and those things are often nerve-wracking, and performed by users who have never done anything like this before or—if they have—they’ve never understood what they were doing.

In the new guide, I’ve reworked much of the existing material, and have added more based on the common questions and tasks I’ve seen users do over the past year or so. There are many sections right up front that cover these common tasks, and I’ve tried to answer the common questions about those tasks right in the section.

Each can pretty much stand alone, and be read without referring to the remainder of the guide. So, there’s some repetition, but I hope it improves the usability of the document. Not that there was anything terribly wrong with the existing User’s Guide—we’ve had great feedback on it. But, constant improvement, in all things. No stone left unturned. Except maybe the about box. Maybe.

What’s left is lots of proofreading, tweaking, and re-shooting a huge pile of screen shots.

And then, when the whole thing is finally locked down, Adobe Acrobat hell.

As I said in the previous post, Acrobat, on the Mac, doesn’t automatically take the structure of a document, and cross-references therein, and generate appropriate tagging for PDFs to produce the nice table of contents and hyperlinking that a long document, like this one, needs. Which sucks. So, my last task is taking the large number of cross-references—which all had to be hand-formatted to look like hyperlinks—and manually tag the PDF to support them. And then, tag the whole thing to generate a real table of contents that’s separate from the main text.

Needless to say, by the end, I’ll be well into Tedious Territory.

Looks like I’ll be joining Craig Crossman on his Computer America radio program Monday, November 14th, from 11pm to Midnight, EDT.

Don’t know what we’ll be talking about, but you can listen online or at various stations across the country—hope you can tune in!

I’m deep into the documentation for SuperDuper! v2.0, and have already reworked my original plan significantly, which is both a good and bad thing.

Let me explain.

The original SuperDuper! User’s Guide tried to follow the UI, in the sense that it was relatively narrative, and encouraged you to read the whole thing. Rather old school, and while this worked well for many, it required a lot of reading to get to the point where you were actually making a backup.

A short while later, given the feedback we were receiving, I extensively rewrote the early sections (3-5) to be more task oriented, and so tweaked it worked pretty well for most.

But still, there was a need for more task oriented material—basically, a “How Do I?” guide (something I’d also written years ago for our Track Record product). So, as v2.0 progressed, I looked through our support database and worked on various task-based bits of this new “Quick Start” guide, intending to have two completely separate manuals—one quick, one detailed—covering both learning styles, and most common tasks.

Given the number of individual tasks and screen shots needed to illustrate the process, it wasn’t long before the “Quick Start” guide ended up with well over 30 pages. Not so Quick any more.

I decided at this point that the two manual idea is basically unworkable, so I re-integrated the two, retaining the task-based approach for most of the Guide, concentrating on various common activities. The latter part goes through the various UI elements in more detail, and has advanced topics like the Sandbox, shell scripts and the like.

I’m also trying to make everything easier to navigate, which involves a lot of tweaking with Acrobat: unfortunately, the Mac version doesn’t extract the structure of the document from Word (the PC version does), so all the links and tags need to be created by hand—something that’s best left to the end of the writing. Don’t want to do it more than once!

It seems to be coming along, though there’s more writing and screen shots and layout and the rest. But, the fact that the documentation is being finalized also means that we’re quite close to release, and that’s a good feeling—now, there’s not just a single gull: there’s a whole flock of them!

A recent post on Jonathan “Wolf” Rentzsch’s Tales from the Red Shed reminded me to give a bit more of the philosophy behind what we’re doing in SuperDuper!

Wolf points out that we don’t do “temporal versioning”—i.e. traditional “incremental backups”—and he’s right.

The Technical Problem
Doing versioning “right” requires both a non-native file format and a database of what’s been going on over time. (I don’t consider the technique used by some programs—stuffing old versions in special folders on the backup media—a reasonable solution, since it pollutes the original and complicates restore. And, yes, you could store in a parallel location with dated folders… but read on.) You need to be able to reconstruct this database from the backup media. And you need rather extensive UI to manage this stuff. (I could keep going, bringing up other issues like the patent problem, but those are the big issues involved.)

This, by definition, significantly increases the complexity of the program’s back and front ends—which makes the program much harder to QA properly. As Wolf says, it’s incredibly important this stuff works. Of course, that’s our problem—it’s our job to ensure that the features we implement are well tested and work.

The User’s Problem
That added complexity has another major problem: it can alienate and confuse users, and a proprietary, single-vendor format leaves them without an alternative should a problem arise. So, it’s important that any solution be easy to understand, usable, and not have any “lock in”.

Staying Balanced
So, to determine whether that complexity is worth adding, it’s important to ask—when do most people need to restore? In general, we’ve found that “regular users” (and by that, I mean real “end users") need to use their backups when:

• They’ve made a “bad mistake”, like accidentally deleting an important file, or overwriting one (this kind of mistake is almost always recognized immediately)
• Their drive (or computer) fails catastrophically, requiring a full restore
• They sent their computer in for service, and it came back wiped clean
• An application they installed, or a system update, caused their system to become unusable/unstable

None of these situations require much other than a high-quality, up-to-date, full copy backup. (The last has a better solution than a backup—a “Sandbox”—which we offer in SuperDuper! as well.)

Covering the 99% Case
Given that, it’s pretty easy to see that most end users don’t need to retrieve a two-year-old (or even six-month old) version of a file from a backup. (An archive is a different thing: I’m talking about backups.) It’s just not that common a case. Developers, on the other hand, do need older versions of files, but they should be using a version control system: something a backup should absolutely not be.

But, it is possible that a user won’t notice a problem in a “bad file” until they’ve already overwritten their backup, thus losing any chance of recovery with a “full copy”. I suggest that while this is a problem for some, we have a good solution: rotate more than one full backup.

Any need for this kind of “temporal rollback” can be significantly reduced with a single rotation—say, on a weekly basis—and nearly eliminated with two—a weekly and a monthly. It’s incredibly rare that, on a non-archival basis, you’d need to go back more than four weeks. (It’s similarly likely that a daily incremental would become difficult to manage, and thus “recycled”, in this kind of timespan.)

Storage Space is Cheap and Plentiful
The only real disadvantage? It takes disk space, something that was incredibly expensive and limited when these other schemes were originally invented (floppies, anyone?). But, these days, disk space is cheaper than cheap, with the “sweet spot”, Mac-boot-compatible 200-250GB FireWire drives going for $150-$200. And most “normal” users can store a lot of backups on a 250GB drive or two.

Simple to Understand
The advantages to this kind of approach are many, not the least of which is that a non-technical user can easily understand what’s going on. It’s incredible how many people are confused by conventional backup terminology—“incremental”, “differential”, backups “sets” and the like. And, complicated storage mechanisms require a significant amount of expertise to perform a full recovery in the event of that all-too-common disaster: the total drive failure. (Look, for example, at what you have to do with Retrospect or Backup 3 should you lose your boot drive (very common)—where the vast majority of people also store their “Backup Catalog”. Yes, it can be done. Even if the program works properly, it can take days to recover.)

Simple to Restore
With SuperDuper!, recovery in that situation is literally a matter of booting from your most recent backup. And restoration—which, should you be on deadline, you need not do immediately—is just a matter of replacing the drive and copying back.

Individual files are also easy to restore: just drag and drop from the backup. (Yes, applications without drag-and-drop install, or system-level files, are harder, but can typically be reinstalled/archive-and-installed should that be necessary… or, see the Safety Clone/Sandbox for another rather unique idea...)

The Other 1%
I know this all sounds terribly simplistic to those who run data centers, or large corporate networks, and for that kind of user, it is. And, I have no doubt that some users have need of more complex systems, with the ability to roll back to any given day during a six-month period—or whatever timeframe they choose to work within.

User It or Lose It
SuperDuper!’s approach is the kind of thing that regular end users can do, and feel confident about. And, with that confidence—and with the ease of use and understanding we provide—they’ll actually back up!

Even the most perfect program can’t work unless that happens—so, in some ways, it’s the most critical thing of all.

Every day, I get asked one question over and over—namely:

My backup is failing, and it’s saying that it couldn’t disable ignore permissions! What’s going on?

This started the day Tiger shipped, and has continued on a hundreds-of-times-a-week basis since. And the worst part? It’s a bug in Tiger. So, put on your waders—we’re going in, hip-deep.

Basically, here’s the deal. OSX has a file called “volinfo.database”, which is stored in a normally-hidden-from-you folder named “/var/db”. It’s a simple text file that looks something like this:

290AC3DC4B28C8FC: 00000001
9A081A3451A451C7: 00000001
435F42E92DC994B5: 00000001
C3301F4FEBB1BC41: 00000001
F36BADBD7ACFBD6F: 00000001
AB47F9CD780BF0A8: 00000001
76E8B50DFBBBD9BB: 00000001
893D1C0A293603C7: 00000001

(That’s my real one.)

That looks like gobbledegook to most, but it’s really a database of the volumes you’ve got connected to your Mac, listed by UUID (a low-level internal identifier that’s constant, even if you rename the volume), followed by the state of their “Ownership” flag: 0 if ownership is off, 1 if it’s on.

When you attach a new drive to the system, the UUID of the drive’s volumes are added to the database, along with their “ownership” setting. And, when you check/uncheck it in Finder’s Get Info panel, the value changes here, too.

Apple ships a command-line tool, vsdbinfo, that allows programs to check and adjust the ownership state of a given volume.

It’s very important for SuperDuper! to ensure that ownership is turned on when it makes a copy. If it’s off, the system does strange things with the file’s owner and group (see Floating ownership nearly sinks us for the story of that particular horror). So, we check it, with vsdbutil, during the “preparation” phase before starting a copy.

Which brings us to Tiger. When Tiger installs, it goofs up when it manipulates this file. So, instead of what you see above, you get:

290AC3DC4B28C8FC: 00000001
9A081A3451A451C7: 00000001
435F42E92DC994B5: 00000001
C3301F4FEBB1BC41: 00000001F36BADBD7ACFBD6F: 00000001
F36BADBD7ACFBD6F: 00000001
F36BADBD7ACFBD6F: 00000001
F36BADBD7ACFBD6F: 00000001

I’ve looked at hundreds of these at this point, and it’s pretty clear what the problem is: the installer, or something it runs, has neglected to put in a carriage return. Two lines run together, and the system can’t parse the database any more—so it always says that ownership is disabled, even when you try to turn it on. (In the case above, one volume has multiple entries because the system keeps failing to turn ownership on… even though it’s doing the right thing, the very fragile parser can’t move beyond the bad line.)

It’s a really simple bug. And while I’ve reported it, it hasn’t been fixed. We’ve waited, provided our users with a workaround, and waited, and waited… and it’s just not getting fixed.

This is really frustrating for our users, because things don’t work in a mysterious (and ungrammatical) way. And it’s frustrating for us, because it makes us look bad, incompetent and/or lazy. Honestly, we’re not.

So, since it’s not getting fixed by Apple, we decided on an alternate approach. I did the research necessary to ensure the problem was consistent, and to figure out how we could fix it for them. After looking at hundreds of these, and confirming that the problem is as described above, we’ve integrated a fix into v2.0.

Specifically, we examine the structure of this text file. If it’s missing the carriage return, we correct it, leaving the semantics intact (all volumes with ownership on/off stay the way they are).

In our testing, this solves the problem 100% of the time… and I, for one, won’t miss directing people to the FAQ entry that, as of this writing, has been viewed 5325 times since April.

I’m sure our users will be much happier, too!

As two of you know, SuperDuper! has provided an AppleScript interface for some time. We’ve used it to provide the rudimentary scheduling in the v1.x series, and we’ve enhanced as necessary to improve the experience in v2.0.

But, one rather tricky thing fell out during the process: users can (and will) schedule any number of backups in ways that might overlap, or even start them at exactly the same time. And, given the nature of a multi-processor machine, and the fact that each “scheduled job” is an AppleScript external to SuperDuper! that has no knowledge of the other scripts around it, there had to be a way to serialize execution or the scripts would all interfere with each other.

In the v1.x series, we’d put a rudimentary system in place: the script can ask about the current status about SuperDuper! and—if SuperDuper! is busy running another copy, it’ll say just that, and the script just waits until it’s done. This was mostly done to prevent the script from “grabbing” the interface away from a user who’s actively using SuperDuper!.

But, with v2.0, it’s trivial to schedule two copy jobs for the same time, and with the old technique both could be told that SuperDuper! was, indeed, idle. And then, both would try to run at the same time. Clearly, it’s unreasonable to ask users to schedule their own backups in a way that would take this into account: we had to handle it internally.

Of course, another way to handle this is to break SuperDuper! into a true background engine, and have the UI and scripts all “drive” this process (or processes). But, beyond breaking existing scripts, doing so required enough reengineering to push it well beyond the scope of this particular release, much as it would have been cool. Gotta ship sometime, you know?

So. Bruce and I talked about this for a while, and Bruce came up with the idea of just changing the status into a queue. The first process who asked for status is at the front, and only the guy at the front gets the real status at any given time: every other script always gets “busy” until they move to the front of the queue.

Nice, simple solution and—apart from various tricky bits that always plague any solution—it seemed to do the trick.

Which led to another problem. To try to match user expectations, the scheduled jobs try to leave SuperDuper! in the same state it was in when they started. So, if SuperDuper! is running, we leave it running; if not, we quit when we’re done.

The scripts are responsible for this state preservation. And, once again, when launching at the same time, on fast systems both would see SuperDuper! as not running, and would take responsibility for quitting. To avoid quitting the application out from under the others, each would be waiting for the other script to finish… and then, due to the deadlock, SuperDuper! wouldn’t ever quit.

Great.

More discussion, and another simple solution: the quit command basically waits until the queue is empty, and all external processes are done, before it does its thing. And this simplifies the scripts, too, because they just don’t have to worry about it.

And, all without changing the script dictionary… not too bad!

So, we’re sweating the details here. Or trying to.