SuperDuper!

Slowest! Update! Ever! Sunday, December 16, 2007

I know it feels like that from the outside. But from the inside, we've been putting out quite a few external beta releases (11 at last count), incorporating feedback from our testers, and carefully observing what users with 2.1.4—who have tried to use it with Leopard—have tried to do.

Before I start, though, please recognize that I'm describing things we noticed back in October. So, this is not causing additional delay—these are some of the things that contributed to the delay in the first place.

Paying Attention to Users

We've always observed our users very carefully. And, after Leopard's release, we noticed one big thing: users with 2.1.4 and Leopard have been trying to store their SuperDuper! backup to the same volume they've been using for Time Machine. But there's a significant problem with that: namely, since the Time Machine backup isn't on the source volume, Smart Update is going to try to delete it.

Now, Time Machine "protects" its own backups through a number of complex mechanisms, so no damage is done—except the backup fails.

Choices

In some ways, this is quite similar to the "storing other files alongside a backup" case that has come up before (and has an entire section of the User's Guide dedicated to it)—basically, we could request that users partition the destination drive, put their Time Machine backups on one volume, and their SuperDuper! backup on another.

But, even with Leopard's new "live partitioning" capability, that's a pain, isn't obvious, requires a support round trip, doesn't meet user expectations, and isn't transparent.

It also wasn't something we took into consideration during our original compatibility design. We focused on the various new capabilities of the OS, ensuring that we worked well, but since we were used to partitions—and knew that Time Machine liked as much disk space as possible—we gave it a lot of space to work with and left it alone.

So, once we started seeing this—which was after Leopard's release, of course—we had to make a choice. Push responsibility to the user to do the right thing, or anticipate and handle it inside SuperDuper! so users wouldn't be immediately frustrated.

But I Don't Want to Use Time Machine!

We certainly understood—and understand—that some users don't want to use Time Machine. But the problem is, a large proportion of Leopard's users do, and we have to serve the needs of both groups. We can't come out with a "Leopard-compatible" release that doesn't work properly with Time Machine.

And we couldn't really release a "quick update" that just copied the basics first, and then an update that handled this case right. Had we done that, we would have still needed to take a full external cycle to test the copy engine changes we needed to make, and then run the same extensive cycle again when we made the low-level changes to handle this case.

So, we bit the bullet and made the necessary changes to handle the case properly, the way users would expect: it just works.

So, if you Smart Update a volume that you're using for Time Machine, it works just fine. Your Time Machine backups, and your bootable SuperDuper! backup, and stored on the same volume, side-by-side, without interfering with each other, with no need to partition or do anything special (other than use Smart Update, of course).

On top of that, you can still copy that volume—that is, a volume that has both Time Machine backups and a bootable SuperDuper! backup—to a 3rd volume to make a second backup, and everything there works great, too.

This is technically more complicated that you might expect, but all of that is under the covers. On the surface, there are no visible changes.

Side Benefits

So, this is the main reason why we were able to improve the handling of Spotlight and preserve icons as I discussed in the last post: we had to handle the Time Machine case, but it benefits all users by improving the way Spotlight (which everyone basically uses) and icon preservation (which a small, but vocal group use) works.

Why Focus on Time Machine?

Which, of course, is an extension of But I Don't Want to Use Time Machine!, above.

The fact is, Apple considers Time Machine to be the most important, most visible new feature in Leopard. They've put a lot of marketing muscle behind its rollout, and its tendrils are in everything, from the default desktop to Quick Look.

As I've discussed before, it's a big, good change, and it benefits all users, even those who use SuperDuper! We're truly two complementary technologies, and we need to ensure that SuperDuper! works perfectly with it to ensure that SuperDuper! itself remains a viable product.

And if the product remains viable, we'll be able to continue improving it for everyone—something we all want!

It's Not All Time Machine, Though—Really!

Another interesting thing we found as we got reports from new users, that we never encountered internally: a very old bug in the Migration Assistant (described in Slowly going insane while waiting for a fix back in 2005) is back, with a vengeance. But there's a new variation: rather than having a missing carriage return, there are now also blank lines in the file, which result in the same failure.

So, we've had to implement a second workaround to fix this problem, too.

How Much Longer?

Anyway, enough of that. At this point, we're basically locked down. We have a few UI tweaks to complete, and we should have a final release out to everyone within a few weeks.

You'll notice this is the first timeframe I've provided other than "soon"... and that means I'm quite confident it'll be in your hands shortly. So, once again thanks for your patience, and I really think you're going to be pleased with the update.

Updating the updates Wednesday, December 05, 2007

Another week, another Leopard Status Update. Please, hold your cheers and applause until the end.

Two parts

Although most people think about SuperDuper! as a single application, it's actually comprised of a number of parts. The two main ones are the UI application (which you interact with) and the Copy Engine, which actually performs the copy.

When changes are made to the UI component, testing takes significantly less time, because it's much easier to ensure that those changes are working properly, and improper operation is relatively low risk: if a button doesn't dim properly in some conditions, it's an easy bug to work around for an end user.

Changes to the copy engine, though, are extremely high risk. Errors made in copying files affect the backup itself, and often occur silently: everything looks like it's working, but, not so much.

Testing the tests

This is one of the reasons that this update is taking so long to release. Due to the nature of the changes in Leopard, we've had to make significant changes to the engine. And while we, of course, have extensive test suites internally, those tests are "necessary" but not "sufficient".

Final bits

As I've mentioned before, we did, of course, have betas of Leopard, and we were working with them. But a beta is just a beta: changes are made up until the shipping version, and that was the case with Leopard as well.

But, there's another consideration, too: we can't do our "big" external test cycles until the "public" has the real bits. Yes, we can roll to testers who have the same beta we have, but when you're making the kinds of changes we had to make this release, it's important to roll in new testers who have different file setups and different patterns of copying.

(Tester fatigue is an issue that all developers are constantly struggling with: it's not so much that a tester is "tired", but that they fall into a pattern of use, and rarely venture outside it.)

So, our internal tests couldn't be run until we received the final bits (we never release test versions to our testers that are risky, to ensure they can really use the test release as a production copy), and our big external test couldn't start until after that.

That testing -- of which there have been multiple rounds -- has been going very nicely indeed.

More progress

So, last week, we locked down the "cloner changes": given feedback from the testers and our external tests, and running against 10.5 and 10.5.1, we're pretty confident that the engine's working properly.

That's a good thing.

We've added some new capabilities to the engine, too. We can now preserve the contents of the Spotlight index on the destination (which also preserves its indexing status in a more reliable way) and preserve the "File System Event" database that Leopard uses. More on this in a later post.

While we were doing that, we decided to do one more thing.

Save the icons

We had always felt that the volume icon -- being a file on the source drive like any other -- should be copied to the backup drive, since you'd want to restore the original drive to the same condition it was originally in, and that included the icon.

Well, we had a lot of pushback on that choice (to put it mildly - I think the thread that discusses this issue and the workaround we came up with originally is the longest one on the discussion forums), and we decided that we were wrong. Since you rarely restore, and your backup drive is constantly visible, it was best to preserve the destination icon. Plus, if you restore with SuperDuper!, we'll preserve the icon of the destination for the restore, so it's all good.

The changes we made, above, made this possible. And so, the destination volume icon will now be preserved.

More coming

So, a lot more progress has been made. Testing's going really well. And I'll talk a bit more about some other new stuff later.

Time Marches On! Monday, November 26, 2007

So, another quick update on our progress on the Leopard version of SuperDuper! (which will work better than the 2.1.4 on Tiger, for those of you still there).

Link Hard

I'm happy to say that we've worked through the last few issues with hard links and the complexities of copying Time Machine backups. Yesterday, I was able to copy and restore an over 7 million file Time Machine drive successfully, and restore it. After some scanning, Time Machine picked up exactly where it left off, happily accepting the "new" drive as the right one.

This means that you can successfully copy your Time Machine backup to a different drive as a backup, or to a new, larger drive, without losing your history.

It also means, as I mentioned last post, that SD! is properly handling some of the new on-disk representations in Leopard.

Another test cycle

Now that we've successfully passed our internal tests for this, we're going to package it up (with some other new stuff I haven't talked about) and roll it to our external testers for them to bang on to make sure we haven't missed anything.

Good progress

So, we're making good progress. I'll write about some of the wackiness we've run into later on (example: did you know that journaling has to be on to be able to successfully copy a drive with hard links? Me neither.), but for now, back to the grind.

Leopard Status Update Friday, November 16, 2007

By popular demand (and demand and demand and demand), here's a quick update on the status of SuperDuper!'s Leopard update (likely to be called 2.1.5). Sorry it's been a bit quiet here. I've been getting a lot of email, and it's hard to find the time to write posts. I'll try to update the blog with status a bit more often in the future.

SuperDuper! is working well in our internal tests and for our external testers. Right now, we're ensuring that some of the (currently) lesser-used features of the Leopard file system are being handled properly, from both a copying and status standpoint.

Some technical details

The details of all these things are more than a bit obscure: the issues we deal with are things that end users should never need to think about.

For example, Leopard allows applications to "hard link" folders, something that's never been allowed before. Time Machine relies on this capability, but it's available to any application. Proper handling of these hard linked folders is important, and has its own challenges, including how you handle the counts and sizes of files on the drive for the status display, etc.

Not all parts of Leopard handle these new on-disk structures consistently, and as such we're working through the issues bit-by-bit, making sure we're doing things as right as we can, and in a way that won't break when (or if) -- things change in the future. Or when a new, Leopard-specific application you install starts to make more use of this stuff...

Another obscure example

One thing we found during testing was that users with Bluetooth keyboards were having trouble. The keyboard wasn't maintaining its pairing relationship with the system when starting up from the copy.

This was not actually due to any kind of file or metadata copying problem (as we initially feared): everything was being copied as it should, so when we ran our various comparison/verification tools, things checked out fine. Instead, it was due to a change in the way Leopard recreates (or, in this case, no longer recreates) a standard system folder (specifically, /private/var/run), which is certainly not the first place you'd look for this kind of thing, and running down the problem (and verifying the fix) took time.

Timeframe

So, we're working hard to get things right. We want to make sure that when you make a copy of an HFS+/HFSX Leopard volume with SuperDuper!—regardless of what new features were used—the copy will be correct.

Of course, we keep being asked—with various level of politeness—for a "timeframe" for release. Again, we're not targeting a drop-dead release date. It all depends on how testing continues to go after each beta release, whether any bizarre things are found that weren't covered in our test suite, and how long those things—if found—take to research and handle properly. It's not going to be in the next few days: we're hoping within a few weeks. I'll keep you informed about our status here.

Waiting sucks

I know. Believe me, I know. Waiting sucks. I understand your frustration: it's frustrating on this end, too. Once again, thanks for your patience as we try to maintain the level of quality you expect from us.

Time’s Arrow Redux Tuesday, October 23, 2007

Now that the release of Leopard is imminent-ish, I thought I'd do an update of my Time's Arrow post from a year ago regarding SuperDuper! and Time Machine. Let's get into it!

Leopard Compatbitility

First, v2.1.4 of SuperDuper! is not fully compatible with Leopard.

Changes were made to Leopard since our last release in February. We've been following Leopard's development closely, and we have a working version of SuperDuper! nearly ready. Given the nature of SuperDuper!, though, we don't want to release it until we have the final "bits" of Leopard to test with. It should be available shortly after Leopard's release (earlier if we get the final build before the public does--which we won't).

To answer the inevitable question: the update will be free when released.

Note that netTunes is not Leopard compatible due to a Leopard bug I was unable to work around. I'm hoping for a fix in an upcoming Leopard release, since I do not have a workaround for the problem.

Leopard and Sandboxes

When the Leopard-compatible version is released, please realize that Leopard is a pretty different version of the OS. I recommend against using a Sandbox to share a Home folder between Tiger/Panther/Jaguar and Leopard: there have been a lot of changes to the way data is stored, and a Sandbox is not likely to provide sufficient isolation.

Time Machine

As far as Time Machine goes -- in general, I've always thought that some sort of backup functionality belongs in the OS. It's been a long time coming. The fact that it wasn't there left opportunities for 3rd parties, but that doesn't mean Apple shouldn't address the missing functionality.

And so, they have, with Time Machine. Really, I think that's a great thing. People need to back up more often, and I think Time Machine encourages them to do so, and gives them a relatively transparent way to do it.

Time Machine and SuperDuper!

That said, though, Time Machine isn't the be-all and end-all of backup programs. In fact, given how it works, I really do think that SuperDuper! remains both relevant and necessary -- a true complement to the functionality in Time Machine.

First, as is likely obvious, Time Machine is designed to provide automatic "temporal" backup (discussed in broad terms in the post The Ninety-Nine-Per-Cent Solution many months ago). Its primary usage scenario -- and the one that most demonstrations and documentation focuses on -- is to allow quick recovery of files and data that have gone missing, etc.

It does this in a way that's highly integrated with the OS, with a unique UI that's both cool and kinda cheesy (I'll tell you, the 'space' theme hasn't grown on me at all...)... and, as was the case with Spotlight, with a certain amount of application-level impact (something 3rd parties like Shirt Pocket could never mandate).

Fully, Directly Bootable Backups with SuperDuper!

What's important to note is that this isn't, and never was, what SuperDuper! was designed to do.

Our tagline, Heroic System Recovery for Mere Mortals, tries to sum up the whole idea: SuperDuper! is designed to provide excellent failover support for the all-too-common case where things fail in a pretty catastrophic way, such as when a drive fails, or your system becomes unbootable. We do this by quickly and efficiently creating a fully bootable copy of your source drive. Perhaps more importantly, recovery is near immediate, even if the original drive is completely unusable, because you can start up from your backup and continue working.

You can even take your backup to a totally different Macintosh, start up from it, and work while your failed Macintosh is in the shop... then, when it comes back all fresh and shiny, restore things and keep working. And even if the other Mac is a different CPU type, you can still open and edit the files on the backup.

You cannot do this with Time Machine: Time Machine copies are not bootable until they're restored.

In SuperDuper!, system recovery is done with a minimum of fuss and bother, and with respect for your time. Yes, Time Machine can restore a full system, but that's not its strength. Doing so requires you to actually start up from the Leopard DVD (which you'll need to have with you) and then take the time to restore the backup in full, which interrupts your workflow, requires a working, entirely separate destination device, and takes a lot of your time -- at the exact moment when you can least afford it.

So, when Leopard comes out, and Time Machine is released, be assured that SuperDuper!'s fully bootable backups continue to be relevant and necessary. We'll work alongside its rapid recovery of individual files, augmenting that with our rapid system recovery.

And, of course, we'll continue to improve every part of SuperDuper! to make backups faster and easier for all.

Ocean of words Thursday, May 03, 2007

For those who haven't heard me blather at length about all things SuperDuper! and Shirt Pocket, there's a 45 minute interview of me up at MacVoices.

Remember: the game rules require you to drink every time you hear me say "Um", "Uh" or audibly gesticulate. Do not play while driving.

Tip for Seagate SATA drive owners Thursday, April 26, 2007

A little tip for those of you who might have installed Seagate SATA drives into your 3GB/s capable G5, MacPro, NAS device, or whatever.

Looks like recent Seagate drives ship with a jumper installed that limits the drive to 1.5GB/s speeds. While the jumper is documented in the User's Guide that ships with retail packs, it's specifically mentioned as something you might need to install if you have trouble with the drive. And OEM drives don't have any documentation at all.

To get 3GB/s, the jumper should not be present on the outer pins of the jumper block. So -- if you've got one of these drives, check it out: you might get that drive humming along twice as fast!

Small change, big impact Sunday, March 25, 2007

Apple can sometimes make small changes that have unexpected consequences.

Take Partition Schemes, for example.

This is the kind of thing most Mac users never think about. You buy a drive, plug it in, drive comes up, things work.

Done.

What you probably don't know is that most drives came partitioned and formatted for Windows, even when the drive is listed as "Mac Compatible", typically using Master Boot Record as the partition scheme and FAT32 as the format.

Before OSX 10.4.6 or so, this wasn't a big deal, at least for SuperDuper! Since FAT32 isn't appropriate for storing Mac files with full fidelity (and has various other issues, like file size limits, etc), we would instruct the user to erase the drive, using HFS+ (Mac OS Extended) as the format. Disk Utility wouldn't allow HFS+ volumes to be hosted on Master Boot Record partitioned drives, and so the user would need to select the drive, rather than the volume, and erase that way. No big deal.

Then, the Intel Macs came out, and with them, 10.4.6.

To help with something-or-other (probably to allow HFS+ and FAT32 volumes to be hosted on the same drive in separate partitions for Boot Camp users), Apple decided to change Disk Utility so that it allows HFS+ volumes to be hosted on Master Boot Record partitioned disks. Which I'm sure didn't seem like a big deal: in fact, it's quite convenient, and had probably been requested as a feature over the years.

All that's great, except for one thing: Macs don't support starting up from Master Boot Record volumes.

This is clearly not a big deal for most users. But, for SuperDuper! users (and, I'm sure, for those who use similar applications) it's been a big problem: drives that look like they should act as startup volumes won't, and the reason is really obscure.

Suddenly, users who used to use Disk Utility to erase a drive now have to head to the Partition tab (which doesn't even appear if you select the volume instead of the hosting drive), click Options, understand the 3 partition types, and partition their drive appropriately.

So, this one small change -- allow HFS+ volumes on Master Boot Record partitioned drives -- ends up having a pretty big impact on users with external drives. This is mostly due to its implementation (and lack of documentation).

Worse, users aren't steered to the right choice by Disk Utility. In fact, the default is to do the absolutely wrong thing for most: retain a Windows partition scheme.

This probably hasn't had much of an impact on Apple's support, but it's sure hurt here!

Infrant Expansion Sunday, March 18, 2007

The other day, I was pushing at the limits of my existing Infrant ReadyNAS NV setup, and needed to increase its size. Normally, this would be a huge project, but with the ReadyNAS it was incredibly easy to do.

You see, the ReadyNAS uses Infrant's proprietary X-RAID. X-RAID basically RAID 6RAID 5 (see comments, below) with the ability to dynamically increase the total size of the RAID as well.

So, not only will the ReadyNAS run with a single drive faiure (and hot-rebuild the drive), it can dynamically increase the size of the RAID set as well. So, all I had to do was:

  1. Buy four drives of the appropriate size. I went from four 250GB drives with a total size of about 700GB, to four 500GB drives with a total size of about 1.6TB.

    The reason you don't get "all" the space on the drives is because redundant information is spread across each drive that allows any drive that goes "down" to be replaced and rebuilt with no data loss.

  2. With the ReadyNAS on, and in use, pull out the first of the four drives.

    Yeah. Scary. But that's what to do!

  3. Unscrew the four screws that attach the SATA drive to the tray from and attach it to the new.

  4. Slide the new drive into place.

    At this point, the ReadyNAS will automatically rebuild the data that was on the original drive on this new drive. All of this has been done with the unit on and operating.

  5. Wait for the rebuild to complete (it'll send you email when it's done).

  6. Repeat with the next drive.

Yeah. That's it. When you're done, you do need to restart the ReadyNAS to get the volume to expand, but that can be postponed until you're ready to do it... and that's the only time the unit is "down".

Pretty cool, eh?

(Yeah, I know I sound like a pitchman for Infrant, but I'm honestly not affiliated with them in any way at all. I just think it's a great product.)

Good-bye, iSight Saturday, March 17, 2007

Many of you probably know that, for some reason, iSight cameras can get into a weird state where they start causing serious errors with FireWire drives.

The symptoms usually include a bunch of I/O errors while copying to a FireWire drive: failed copies, flaky behavior, crashes. All of this goes away if you power off, disconnect the iSight, wait a while and power back up.

Sometimes, the iSight stops working with iChat (it says the camera's in use when it's not, or the light comes on, but you don't get any picture), and in it was in that second state today when I had a kernel panic.

No data was lost, I'm happy to say, but I decided that I've had enough of this. Since 10.4.9 supports USB cameras, I've switched to a Logitech QuickCam Ultra Vision. (All they need to do is add Super Deluxe Extreme Edition to the end to make it really cool!)

It's not as elegant as the original iSight (it's fixed focus, horizontal format, a bit gaudy), but it's got a nice wide angle lens, built in microphone, works with any monitor, and seems reliable. Plug the thing in, and it works.

Given that you can't buy "real" iSights any more, and based on direct experience, I give it a thumbs-up.

Sorry, iSight. I'll miss you!

Page 6 of 13 pages « First  <  4 5 6 7 8 >  Last »