Excuse the brevity of this post, but—as you might expect with a new update–things are kind of busy around these parts.

The update's gone well, except for some bandwidth issues we had to deal with, and some MySQL issues on my server that happened due to some issues with the way FogBUGZ handles large attachments (really, really crappily, typically crashing MySQL after eating huge amounts of memory). But, apart from continued delays for outbound mail due to volume, it's good.

But, as I said a number of posts ago, the only thing I can promise about a big update like this one is that there'll be a few problems. They're not affecting many people, but to summarize:

• Some users are reporting that the status display stops updating in the middle of the copy. The backup does actually complete eventually, but the progress bar doesn't move, and the file counts stop.

  We're not quite sure what's going on here: it's not something we've been able to reproduce at this end. But we're looking into it. If you're experiencing this issue, drop a note to support.

• The updater has been giving the occasional problem. This is frustrating, because we know of some problems there and have tried to "pre-emptively" handle the problem with the update notice, which indicates that you should download the update manually if you experience a problem.

  Unfortunately, that hasn't been clear to people, resulting in a huge volume of incoming mail (we're talking thousands in the past few days)... so sorry for the delays in responding.

• 10.2/10.3 users were being offered the 2.5 update. Due to an error in the update description XML, 10.2/10.3 users were being offered the update, even though it wasn't compatible with their systems. The XML has been corrected, so that shouldn't happen any more.

• Some users are having problems starting up from their copies. On some systems, copies start up but then programs immediately begin crashing. This is corrected by re-prebinding the copy, but given the way Leopard works this shouldn't be necessary. We're hoping this is a Leopard bug that'll be corrected in an update of OS X, but we're investigating it here, too.

• Users with AntiVirus programs are having the occasional problem. This is kind of expected, given the way that AntiVirus programs work. In general, we suggest that you configure your AntiVirus application so that it ignores the backup volume.

  That said, if the AntiVirus program is on, an incorrect result code returned from a system call is indicating a file that vanishes between two points of execution is a folder, when it's not, and that's causing a weird "(null)" error that repeats a large number of times in the log. We're implementing a workaround for this case in our copy engine.

  Note that if you get a "permission denied" type of error and you're running AntiVirus, you should configure your AntiVirus program to not scan the backup volume, or turn off its "auto protect" feature while you're backing up.

• Already mounted images, or images with special characters in their filename, cause an error. Due to a regression (caused by an attempt to workaround another bug), SD! fails if a sparse image we're supposed to copy to is already mounted. We also fail if the image has "special" characters (such as a quote) in it. We're working on a fix for this.

• Networked Time Machine backups, on the same volume as local Time Machine backups, aren't preserved. If you're using the same volume for "networked" and "local" Time Machine backups, and you try to store a SuperDuper! backup directly to the same volume, we preserve the "local" backups but not the "network" ones. (That's a mouthful, sorry.)

  We're working on a fix for this as well.

• Custom copy scripts that used "Exclude spotlight files" are failing. This script is no longer needed, so users who have included it should modify their copy scripts to remove the reference.

I think that's about it. There's nothing terribly major in there, fortunately, and we're working on getting the problems corrected as quickly as we can.

Thanks to everyone who's reported problems, and also to the vast majority of people who haven't encountered any!

And what could make it more official than a press release and a download?

Press Release

Shirt Pocket is happy to announce that SuperDuper 2.5 is now available as a free update for all users. The new version includes full Leopard support, including the ability to store a bootable backup side-by-side with a Time Machine backup on a single volume, and the ability to copy Time Machine archives to other drives for backup purposes or to move to a larger drive without losing history.

In addition, version 2.5 improves many aspects of our 2005 and 2006 Macworld Eddy-award winning application, including improved Spotlight handling, a "Run Now" feature for scheduled copies, icon preservation for destination volumes and various performance improvements.

"What more appropriate day to release a terrific new version than SuperDuper Tuesday? We're really happy with the way version 2.5 works with Leopard", said David Nanian, owner of Shirt Pocket, in his standard stump speech. "Our new feature set is great on its own, plus it's an excellent complement to Time Machine, adding the ability to have a bootable backup alongside your Time Machine archive -- and we've done all this without any increase in complexity. With SuperDuper!, recovery from a disk crash is just a matter of rebooting from the backup!"

Of course, SuperDuper 2.5 still has all the capabilities SuperDuper! is famous for, including: the ability to easily schedule backups; additional imaging options; more control over shutdown; better AppleScript support; hundreds of UI improvements; Growl support; and a readable, complete, task-based User's Guide.

SuperDuper supports both Intel and Power PC Macs running Mac OS X 10.4 or later, and is a free update for existing users. The unregistered version will perform full backups for free. Registration costs $27.95, and includes many additional timesaving features, including Smart Update for faster backups, Scheduling, and others.

More information, as well as a download link, can be found at http://www.shirt-pocket.com/SuperDuper.

About Shirt Pocket

Shirt Pocket, based in Weston, Massachusetts, was formed in late 2000 as a Macintosh-only shareware creator and publisher. Shirt Pocket's first product, the 2004 Eddy Award winning netTunes, lets customers control iTunes on one Mac from any other Mac on the network with iTunes own intuitive user interface. launchTunes, Shirt Pocket's second product, made iTunes' playlist sharing practical by automatically launching iTunes on remote servers when needed. SuperDuper!, the 2005 and 2006 Eddy Award winning disk copying program that allows mere mortals to back up and restore their systems accurately and confidently, was released in January 2004.

Shirt Pocket was started by David Nanian, co-founder of UnderWare, Inc, and one of the original authors of the BRIEF programmer's editor and Track Record bug tracking system.

...we salute our testers, and the patience of our users.

It's going to be a SuperDuper! Tuesday.

A few days ago, we were happy to get a workaround for the bug mentioned in a previous post, so we went about implementing and testing—and things were working quite well.

Unfortunately, we've found a case where the workaround doesn't, um, work. I think I have a way of dealing with it, and so we're going to implement Workaround Part IV: The Reworkening and run it through the wringer.

Again.

These tests are really time consuming, as you might guess: it can take eight hours to rebuild a complex-case volume (so I try to keep one building while another is testing), and then more hours to run the test. I can't begin to tell you how frustrating it is to have a set of simple tests work and then find that the thing we found rearing its ugly head again in another (more complex) scenario that can (and did) happen in "real life".

This is why we don't just implement a fix and release, of course: we want to ensure it works before we toss it out there, and so we smoke test, test internally and then—if internal tests are OK—test externally, every time. We always are hoping for green lights at every stage. Sometimes, 99% of the way there, you get red. Then, the failure needs to be investigated, understood, reproduced...

Anyway, we'll leave that between me and my ulcer. If my new idea works, this will hopefully only delay things a few days, which will be plenty of time to fill the comments with complaints of our laziness, incompetence, lack of communication and general suckitude. Have at it!

Note: I've had to turn comments off for a little while because the subscriptions to comments are overloading my outbound servers (every time someone posts something, it's sent to the 200 people above their post), causing delays for regular support mail, etc. You can still review existing comments by clicking the title of this post. I'll turn them back on after the backlog clears. Sorry for any inconvenience: it's not that I don't want to hear what you want to say...

Last post, I mentioned that we were bitten by a bug that showed up during late-in-the-game testing that didn't make a lot of sense, and was quite nasty in certain complex situations. This bug caused the "release process" to grind to a halt.

Well, I'm happy to say that, as of about two minutes ago, I've managed to figure out what's going on.

Basically, a folder can become "magic" in some situations, and even when the conditions that made it "magic" are reversed, the "magic" sticks around when it shouldn't. Unfortunately, this "magic" acted as a sort of "protective spell" on the folder, and was preventing us from doing anything.

Unfortunately, there's no 'external' visibility for when the "magic" sticks around, so we were seeing something that basically didn't make any sense. On top of that, it's new behavior in Leopard, which is why we've never seen it before. Fortunately, thanks to Amit Singh's recently-updated-for-Leopard hfsdebug (thanks, Amit -- love the book, too), I was able to drop down into the guts of HFS+ and determine what's happening.

Now that the problem's understood, we can implement an effective workaround. The workaround will mean that in some situations it'll re-copy a bit more than it should when Smart Updating. But, at least the result produced will be correct: and the workaround will break the spell, and remove the "magic".

Which is—let me tell you—a relief. (For those of you out there who have hit this kind of WTF-roadblock, where you have no idea what's wrong and thus can't even estimate how long it's going to take to figure it out and fix it, you know what I mean.) And for any Apple engineers reading, the (incorrect) behavior is described in rdar://5687977.

So, anyway, now that that data-integrity-related bug is getting wrapped up, we're back to putting together a final test build (should be in the next day or two), a bit of time to let our test group run their scenarios, etc.

So, barring another similar issue (please, no) showing up during testing, as I indicated in the comments of the previous post, it should be a week or so...

We look to still be on schedule, so hopefully you'll have the new version (which, by the way, I've decided will be 2.5, not 2.1.5) in a week or so.

We've been working through a problem with copying hard folder links in a complicated source volume recently that's got us a bit stymied—what we're seeing doesn't make sense—but hopefully we'll get it wrapped up shortly.

Sorry for the briefness of the entry: both busy and sick with some stomach flu... more when there's something new/interesting to say.

I know it feels like that from the outside. But from the inside, we've been putting out quite a few external beta releases (11 at last count), incorporating feedback from our testers, and carefully observing what users with 2.1.4—who have tried to use it with Leopard—have tried to do.

Before I start, though, please recognize that I'm describing things we noticed back in October. So, this is not causing additional delay—these are some of the things that contributed to the delay in the first place.

Paying Attention to Users

We've always observed our users very carefully. And, after Leopard's release, we noticed one big thing: users with 2.1.4 and Leopard have been trying to store their SuperDuper! backup to the same volume they've been using for Time Machine. But there's a significant problem with that: namely, since the Time Machine backup isn't on the source volume, Smart Update is going to try to delete it.

Now, Time Machine "protects" its own backups through a number of complex mechanisms, so no damage is done—except the backup fails.

Choices

In some ways, this is quite similar to the "storing other files alongside a backup" case that has come up before (and has an entire section of the User's Guide dedicated to it)—basically, we could request that users partition the destination drive, put their Time Machine backups on one volume, and their SuperDuper! backup on another.

But, even with Leopard's new "live partitioning" capability, that's a pain, isn't obvious, requires a support round trip, doesn't meet user expectations, and isn't transparent.

It also wasn't something we took into consideration during our original compatibility design. We focused on the various new capabilities of the OS, ensuring that we worked well, but since we were used to partitions—and knew that Time Machine liked as much disk space as possible—we gave it a lot of space to work with and left it alone.

So, once we started seeing this—which was after Leopard's release, of course—we had to make a choice. Push responsibility to the user to do the right thing, or anticipate and handle it inside SuperDuper! so users wouldn't be immediately frustrated.

But I Don't Want to Use Time Machine!

We certainly understood—and understand—that some users don't want to use Time Machine. But the problem is, a large proportion of Leopard's users do, and we have to serve the needs of both groups. We can't come out with a "Leopard-compatible" release that doesn't work properly with Time Machine.

And we couldn't really release a "quick update" that just copied the basics first, and then an update that handled this case right. Had we done that, we would have still needed to take a full external cycle to test the copy engine changes we needed to make, and then run the same extensive cycle again when we made the low-level changes to handle this case.

So, we bit the bullet and made the necessary changes to handle the case properly, the way users would expect: it just works.

So, if you Smart Update a volume that you're using for Time Machine, it works just fine. Your Time Machine backups, and your bootable SuperDuper! backup, and stored on the same volume, side-by-side, without interfering with each other, with no need to partition or do anything special (other than use Smart Update, of course).

On top of that, you can still copy that volume—that is, a volume that has both Time Machine backups and a bootable SuperDuper! backup—to a 3rd volume to make a second backup, and everything there works great, too.

This is technically more complicated that you might expect, but all of that is under the covers. On the surface, there are no visible changes.

Side Benefits

So, this is the main reason why we were able to improve the handling of Spotlight and preserve icons as I discussed in the last post: we had to handle the Time Machine case, but it benefits all users by improving the way Spotlight (which everyone basically uses) and icon preservation (which a small, but vocal group use) works.

Why Focus on Time Machine?

Which, of course, is an extension of But I Don't Want to Use Time Machine!, above.

The fact is, Apple considers Time Machine to be the most important, most visible new feature in Leopard. They've put a lot of marketing muscle behind its rollout, and its tendrils are in everything, from the default desktop to Quick Look.

As I've discussed before, it's a big, good change, and it benefits all users, even those who use SuperDuper! We're truly two complementary technologies, and we need to ensure that SuperDuper! works perfectly with it to ensure that SuperDuper! itself remains a viable product.

And if the product remains viable, we'll be able to continue improving it for everyone—something we all want!

It's Not All Time Machine, Though—Really!

Another interesting thing we found as we got reports from new users, that we never encountered internally: a very old bug in the Migration Assistant (described in Slowly going insane while waiting for a fix back in 2005) is back, with a vengeance. But there's a new variation: rather than having a missing carriage return, there are now also blank lines in the file, which result in the same failure.

So, we've had to implement a second workaround to fix this problem, too.

How Much Longer?

Anyway, enough of that. At this point, we're basically locked down. We have a few UI tweaks to complete, and we should have a final release out to everyone within a few weeks.

You'll notice this is the first timeframe I've provided other than "soon"... and that means I'm quite confident it'll be in your hands shortly. So, once again thanks for your patience, and I really think you're going to be pleased with the update.

Another week, another Leopard Status Update. Please, hold your cheers and applause until the end.

Two parts

Although most people think about SuperDuper! as a single application, it's actually comprised of a number of parts. The two main ones are the UI application (which you interact with) and the Copy Engine, which actually performs the copy.

When changes are made to the UI component, testing takes significantly less time, because it's much easier to ensure that those changes are working properly, and improper operation is relatively low risk: if a button doesn't dim properly in some conditions, it's an easy bug to work around for an end user.

Changes to the copy engine, though, are extremely high risk. Errors made in copying files affect the backup itself, and often occur silently: everything looks like it's working, but, not so much.

Testing the tests

This is one of the reasons that this update is taking so long to release. Due to the nature of the changes in Leopard, we've had to make significant changes to the engine. And while we, of course, have extensive test suites internally, those tests are "necessary" but not "sufficient".

Final bits

As I've mentioned before, we did, of course, have betas of Leopard, and we were working with them. But a beta is just a beta: changes are made up until the shipping version, and that was the case with Leopard as well.

But, there's another consideration, too: we can't do our "big" external test cycles until the "public" has the real bits. Yes, we can roll to testers who have the same beta we have, but when you're making the kinds of changes we had to make this release, it's important to roll in new testers who have different file setups and different patterns of copying.

(Tester fatigue is an issue that all developers are constantly struggling with: it's not so much that a tester is "tired", but that they fall into a pattern of use, and rarely venture outside it.)

So, our internal tests couldn't be run until we received the final bits (we never release test versions to our testers that are risky, to ensure they can really use the test release as a production copy), and our big external test couldn't start until after that.

That testing -- of which there have been multiple rounds -- has been going very nicely indeed.

More progress

So, last week, we locked down the "cloner changes": given feedback from the testers and our external tests, and running against 10.5 and 10.5.1, we're pretty confident that the engine's working properly.

That's a good thing.

We've added some new capabilities to the engine, too. We can now preserve the contents of the Spotlight index on the destination (which also preserves its indexing status in a more reliable way) and preserve the "File System Event" database that Leopard uses. More on this in a later post.

While we were doing that, we decided to do one more thing.

Save the icons

We had always felt that the volume icon -- being a file on the source drive like any other -- should be copied to the backup drive, since you'd want to restore the original drive to the same condition it was originally in, and that included the icon.

Well, we had a lot of pushback on that choice (to put it mildly - I think the thread that discusses this issue and the workaround we came up with originally is the longest one on the discussion forums), and we decided that we were wrong. Since you rarely restore, and your backup drive is constantly visible, it was best to preserve the destination icon. Plus, if you restore with SuperDuper!, we'll preserve the icon of the destination for the restore, so it's all good.

The changes we made, above, made this possible. And so, the destination volume icon will now be preserved.

More coming

So, a lot more progress has been made. Testing's going really well. And I'll talk a bit more about some other new stuff later.

So, another quick update on our progress on the Leopard version of SuperDuper! (which will work better than the 2.1.4 on Tiger, for those of you still there).

Link Hard

I'm happy to say that we've worked through the last few issues with hard links and the complexities of copying Time Machine backups. Yesterday, I was able to copy and restore an over 7 million file Time Machine drive successfully, and restore it. After some scanning, Time Machine picked up exactly where it left off, happily accepting the "new" drive as the right one.

This means that you can successfully copy your Time Machine backup to a different drive as a backup, or to a new, larger drive, without losing your history.

It also means, as I mentioned last post, that SD! is properly handling some of the new on-disk representations in Leopard.

Another test cycle

Now that we've successfully passed our internal tests for this, we're going to package it up (with some other new stuff I haven't talked about) and roll it to our external testers for them to bang on to make sure we haven't missed anything.

Good progress

So, we're making good progress. I'll write about some of the wackiness we've run into later on (example: did you know that journaling has to be on to be able to successfully copy a drive with hard links? Me neither.), but for now, back to the grind.

By popular demand (and demand and demand and demand), here's a quick update on the status of SuperDuper!'s Leopard update (likely to be called 2.1.5). Sorry it's been a bit quiet here. I've been getting a lot of email, and it's hard to find the time to write posts. I'll try to update the blog with status a bit more often in the future.

SuperDuper! is working well in our internal tests and for our external testers. Right now, we're ensuring that some of the (currently) lesser-used features of the Leopard file system are being handled properly, from both a copying and status standpoint.

Some technical details

The details of all these things are more than a bit obscure: the issues we deal with are things that end users should never need to think about.

For example, Leopard allows applications to "hard link" folders, something that's never been allowed before. Time Machine relies on this capability, but it's available to any application. Proper handling of these hard linked folders is important, and has its own challenges, including how you handle the counts and sizes of files on the drive for the status display, etc.

Not all parts of Leopard handle these new on-disk structures consistently, and as such we're working through the issues bit-by-bit, making sure we're doing things as right as we can, and in a way that won't break when (or if) -- things change in the future. Or when a new, Leopard-specific application you install starts to make more use of this stuff...

Another obscure example

One thing we found during testing was that users with Bluetooth keyboards were having trouble. The keyboard wasn't maintaining its pairing relationship with the system when starting up from the copy.

This was not actually due to any kind of file or metadata copying problem (as we initially feared): everything was being copied as it should, so when we ran our various comparison/verification tools, things checked out fine. Instead, it was due to a change in the way Leopard recreates (or, in this case, no longer recreates) a standard system folder (specifically, /private/var/run), which is certainly not the first place you'd look for this kind of thing, and running down the problem (and verifying the fix) took time.

Timeframe

So, we're working hard to get things right. We want to make sure that when you make a copy of an HFS+/HFSX Leopard volume with SuperDuper!—regardless of what new features were used—the copy will be correct.

Of course, we keep being asked—with various level of politeness—for a "timeframe" for release. Again, we're not targeting a drop-dead release date. It all depends on how testing continues to go after each beta release, whether any bizarre things are found that weren't covered in our test suite, and how long those things—if found—take to research and handle properly. It's not going to be in the next few days: we're hoping within a few weeks. I'll keep you informed about our status here.

Waiting sucks

I know. Believe me, I know. Waiting sucks. I understand your frustration: it's frustrating on this end, too. Once again, thanks for your patience as we try to maintain the level of quality you expect from us.