Like Riding a Bike Wednesday, May 05, 2021

You know it's been a while since you last released a Beta when you don't quite remember the rhythms of the process.

But, it doesn't take long for muscle memory to kick in... and I'm happy to say that it's going really well. A lot of that is due to you: thanks for your support, encouragement and reports of both success and, occasionally, failure.

To deal with the latter (and celebrate the former), here's v3.5 B2.

A few things to mention as we continue this process:

  • Make sure you have "Check for Updates" turned on in our Preferences. You may have turned it off if you were using the v3.2.5 workaround: it's time to turn it on again, since that's how you'll get Beta updates.
  • Some users were confused by the macOS 11.4 Beta requirements. 11.4 is only required for M1 Macs. Intel Macs work fine with SuperDuper under all versions of Big Sur.
  • Startup volumes will often not show up in the Startup Disk Preference Pane in System Preferences under Big Sur. This does not mean the drive isn't bootable—it's a weird bug in Big Sur.

    To start up from the backup, use Option+Power On on an Intel Mac, or hold down the Power button when powering on with an M1 Mac.

    If you do this, you'll note that the Mac's regular drive doesn't appear in the startup disk preference pane...but the backup drive does! It's weird. But the backup is OK! You can breathe!

  • If you only made a Smart Update, and didn't start with an Erase, then copy backup, you'll note that your backup drive seems "empty". That's because you're looking at the "empty" System volume...and the Data volume is mounted by the OS as "no browse"—basically, it doesn't show up readily in Finder.

    It's still there, though. Use Finder's Go To Folder, in the Go Menu, to open /Volumes and you'll see your Data volume.

    For less stress—and who doesn't want less stress?—start with an Erase, then copy backup.

  • We've actually figured out how to get at least a little useful information out of the weird errors system tools produce:

For Beta 2 we've addressed a few things and polished some others: it's a relatively minor update. Details are in the Revision History, but to highlight a few things:

  • Backup on Connect no longer runs over and over if you use Erase, then copy on Big Sur. This had to do with some very low-level changes to some Big Sur APIs that messed with our code that detected whether we were already running a copy for that volume.

    The bug was useful for testing a zillion back-to-back Erase, then copies, though, so there's that. Yeah! We meant to do that! Sure!

  • Erase, then copy was using asr to copy Catalina, which was unintentional. It now does the same thing v3.3.1 did.
  • Send to Shirt Pocket was failing for users in all versions of SuperDuper due to an locally cached response of a value that had been updated on our server. This turned out to be rather interesting, and frustrating, because we don't have a way of "fixing" old copies...although they will "fix themselves" over time...anyway, it won't happen any more in v3.5 B2 and later.

That'll about do it. Enjoy the new version, and thanks for the emails and tweets!

Download SuperDuper! v3.5 B2

Thanks, again, for using SuperDuper...we look forward to your comments.

Finally. Thursday, April 29, 2021

Sound the trumpets!

We're happy (and, frankly, relieved) to announce the immediate availability of SuperDuper v3.5 Beta 1: our first version to fully support Big Sur backups.

That's right: bootable Big Sur backups for Intel and M1. Today!

Shut Up and Give Me My Download

For those who aren't interested in reading about this stuff (you really should), you can download SuperDuper v3.5 B1 from here.

Note If you're on an M1 Mac, you will not be able to boot from a SuperDuper backup unless you are running macOS 11.4 or later. The new macOS version is in beta, and you can get it here: macOS 11.4 Public Beta. (The usual caveats about running a beta OS apply.)

As you might expect, you can't boot an M1 Mac from an Intel Backup or vice versa. You can still restore, though, using Apple's "first boot" restore process, or by using Migration Assistant.

When starting off with the v3.5 beta, you must first make an erase-then-copy backup. On success, you can switch to Smart Update.

Finally, we know Big Sur Erase, then copy image backups don't work yet—Smart Updates work fine—but we wanted to get this out to those of you who have been patiently waiting. Remember: Beta!

As is always the case with our Betas, this version will automatically update to future Beta versions of v3.5 (assuming our update check is on in Preferences) until the final v3.5 is released. At that point, it will install the release version and move to the regular update check.

Let's Hear What He Has to Say!

(Have those folks in a rush gone? OK, good. For the rest of you, here's the deal.)

As we've been saying for some time, release of our Big Sur version had been gated on factors not fully in our control.

For security reasons, Big Sur forced the use of Apple Software Restore, or asr, for backups that contain a System volume (that is, any startup volume).

Until recently, asr didn't work properly with M1 Macs. It would copy the System volume, then throw various errors, including one indicating that the "source volume format was not yet supported".

Of course, the "source volume format" was APFS, something it clearly did support, so this was weird. Clearly they meant something else, like the various new support volumes the M1 Macs use.

But given that asr wouldn't run at all, there wasn't much point trying to copy the System volume. So we decided to wait, thinking that a fix would be just around the corner.

Which, needless to say, it wasn't. Or it was, but given that blocks are longer in San Francisco than they are around here (and, frankly, Apple had its hands full with a bunch of new Macs, iPads and the like), the fix took much longer than expected. So we needed to do something.

Since copying the Data volume was the only thing that worked, and was still fully restorable, we released a workaround while we waited.

This had two major advantages. First, we didn't need to fully test and beta a new version that half worked, sometimes, with caveats...something we were loath to do. Instead, it made it clear it was a workaround.

Second, it meant that we didn't have to "half wrap-up" our in-progress work.

New Stuff Has Come to Light

Of course, we kept plugging away at it and, when we checked the new macOS 11.4 beta, we were happy to see the "not yet" message had gone away, which suggested "not yet" had finally become "can". Or, at least, "might".

More testing and configuring ensued and, as of late Saturday night (livin' the high life here at Shirt Pocket HQ), we were finally able to boot from a SuperDuper copy of an M1 MacBook Pro, made to a regular USB SSD, without any post-copy OS install.

Finally.

As such, we can finally release a Beta that works as expected on both Intel Macs (now) and M1 Macs (assuming you have installed 11.4 Beta 1 or later).

To be clear, this is only possible because Apple made it possible in macOS 11.4. I'm certain it required changes to the M1's startup process and to asr to finally make it happen. Without those changes, we couldn't have done this in a supported way.

I'm really happy the teams involved made the effort. Thanks, folks!

The "Did I Say No Caveats?" Caveats

Remember, though, Apple is not finished. There's movement, and success, but we honestly don't know whether the final, non-beta release of 11.4 will work.

Given things are clearly changing, and heading in the right direction, we thought putting this out now would be useful.

As indicated above, you can't boot an Intel Mac from an M1 backup or vice-versa. You can, however, boot an M1 Mac from a backup of a different M1 Mac, once you've authorized it with a user on the Mac you're booting.

Regardless of processor, you can still restore using Apple's "first boot" restore process, or by using Migration Assistant. So it's not a problem to move from Intel to M1 or M1 to Intel.

Important Note asr copies at a low level and works a drive quite hard. In our internal testing, some SSDs couldn't handle the load without intermittently overheating and generating errors.

This seems to be because the controller was trying to manage freed blocks, or wear-level, while still trying to manage the flow of incoming writes.

When this happened, we started to get obscure errors from asr ("resource busy" is quite common). Allowing the device to settle and cool typically resolved the problem. I'm hoping Apple will improve its handling of this error as development continues.

We had the most success with the Angelbird SSD2GO PKT MK2, with its SSD Manager installed and the most recent Westernal Digital MyPassport SSD (affiliate link).

Erasing a drive from the top/hardware, rather than just erasing the volume, seems to help some drives, too.

Or you can just use Smart Update, which will only copy the Data volume. To wit...

Notes on a Revolution

With macOS releases prior to Big Sur, we could update the OS when changed, regardless of whether it was hosted on a separate System volume (as done under 10.15 and later) or on the same volume (all previous versions).

However, with Big Sur, a system update requires us to use asr to recopy the drive. And asr won't (currently?) just copy the System volume: it's all or nothing.

That's fine with an Erase, then copy backup, because you are asking for a full backup.

But with a Smart Update, fully erasing the drive when an OS changes just isn't appropriate.

So, we decided that under macOS 11 and later, a Smart Update of a "volume group"—that is, a startup volume—will only copy the Data volume, updating your applications and data but leaving the system as-is.

This gives you some interesting options:

  • You can keep your backup under an earlier version of the OS, while keeping your own files and applications up to date
  • It's now possible to restore just your applications and data into an existing OS install, whether through migration or by copying with SuperDuper
  • By updating your backup with Smart Update, and then restoring with Erase, then copy, you can roll back the OS to the version on the backup, while keeping your most recent files and applications intact

You can think of it as a modern Sandbox: you can try a new minor OS version and still roll back if you don't like it...without losing your "new" data. Pretty nice!

Have at it

So there you go. It's been a long, frustrating road, we know—believe me, we know—but we can clearly see our destination. Enjoy the new version, thank you for your patience, and we look forward to your comments and feedback.

Download SuperDuper! v3.5 B1

That’s Big SIR to You! Sunday, January 31, 2021

Hey, folks. Sorry it's been a while, but it's been a busy time. Let's start with the bad news first.

Bad news

As you know, SuperDuper 3.3.1 cannot copy a volume with Big Sur on it. We're currently blocked on some issues I don't have direct control over, and as such I don't have a new version for you that fully supports Big Sur, nor a timeframe for when that will be released.

Right now, as many of you know, v3.3.1 will work with non-boot volumes, but it won't work with volumes that have macOS on them, because it will try to do some of the things that no longer work in macOS 11.

I know that's been a disappointment, but that's where we are with v3.3.1.

Good news!

However, after wracking my brain for far too long, I've come up with a workaround that will let you make the backups you need to save your files, and to supplement your Time Machine backup. And for that, we need to go Back...to the Future!

Huh?

Let me try to explain.

In Catalina, as I explain in Breaking the Tape, Apple split the startup volume into two parts: the System volume and the Data volume. We did a ton of work that year to support this new setup in a way that was transparent to the user; SuperDuper automatically creates the proper volumes, converts the drives to APFS as needed, etc.

Worked great.

In macOS 10.15.5, though, Apple broke 3rd party copy tools in a way that couldn't be worked around without the use of asr, a low-level drive copy tool that has its own issues. They fixed that in 10.15.6...but it was a rather ominous sign for the future.

That ominous sign became terrifying reality in macOS 11. Due to the new Sealed System Volume, use of asr became mandatory if you wanted to make a copy that was bootable. And even that didn't work at all until November 5th of last year—just before Big Sur's official release.

Even now, as of the time of this writing, asr won't make a bootable copy of an M1-based Mac.

So, as of Big Sur, 3rd party tools like SuperDuper can no longer make bootable copies on their own. For that, it's asr or nothing.

It is, indeed, a very sweet solution.

But, 3.3.1 doesn't know that. It tries to do all the special stuff that we had to do for Catalina, and those things no longer work. And so, as you've seen, that copy generates errors or seems to hang right at the start (because it's thrown exceptions that stop the copy).

Didn't You Say "Good News"?

I'm getting there.

SuperDuper! 3.3.1's magic was all about dealing with the split startup volume. It built on the APFS support and scheduling fixes we put into the previous version...and added new things for compatibility with Catalina.

But...what if it didn't do that? What if SuperDuper was...stupider?

Wonderfully Awful

I've been testing this out for a while in-house. and I've come up with a weird-sounding workaround that...works!

Basically, you can use SuperDuper to copy the Data volume of the volume group. The result contains all your data and applications, can be restored in a few different ways...and can even be made bootable.

Note that, as I indicated above, M1 Macs can't readily boot from external drives. There are things you can do, if you have an external Thunderbolt 3 drive (USB-C isn't sufficient), but even that won't work if the internal drive is dead. Unless things change, bootable backups are basically a thing of the past on M1-based Macs.

How?

It's actually easy. To accomplish this, use an old version of SuperDuper—specifically, v3.2.5—to copy the Data volume, which is shown in the older version!

v3.2.5 is well tested, having been on the market for quite some time, and is reliable. So we don't have to worry about doing a broad beta test of a partially complete new release. It's already tested, and I've been busy doing the additional testing necessary to prove it works on Big Sur.

Again, this will make a copy of the data that you need to preserve your stuff, both Applications and Data, while leaving the Sealed System Volume alone.

And it's a valid source for "restore" during a clean install or migration! So restoration is easy and fast should it become necessary.

Neat!

Yeah, I wish I had thought of this earlier.

So, if you're on Big Sur, and you want to copy a startup drive, here's what to do:

  1. Make sure you have your license information handy. You can retrieve it from SuperDuper's Register... page should you need to.
  2. Download and install SuperDuper! v3.2.5 from here.
  3. Remove SuperDuper! from the "Full Disk Access" list in the Security & Privacy preference pane and restart your Mac. This is important, and works around an Apple bug triggered by the change of SuperDuper!'s bundle ID.
  4. Run SuperDuper and follow the steps to allow it Full Disk Access.
  5. If your license is missing, re-enter it from your license email.
  6. Turn off "Check for Updates" in our Preferences so we don't nag you about v3.3.1.
  7. Select the "Data" volume in the source pop-up, and a new APFS backup volume in the destination pop-up, along with "Backup - all files" (or whatever script you want). If you already have a backup volume, you can use Disk Utility to select and delete just the backup System volume, rather than create a new one. After doing this, rename the Data volume to something sensible (remove "- Data"). Note that you may need to repair it with Disk First Aid before it will show up in SuperDuper.
  8. Make your copy as normal, set up your schedule as needed, etc. Your regular Smart Updates will work as expected.

To fully restore, it's easiest to boot to recovery, erase the internal drive you want to restore to, reinstall the OS from Recovery mode, and then, when prompted to restore during the first boot of the fresh copy of macOS, point at the backup. All your data and applications will be brought in automatically.

If you want to make the backup bootable and have an Intel Mac, boot to Recovery (Cmd+R during power on) and install Big Sur to the backup drive. You can then start up from the backup. Note, though, that once made bootable, you can no longer copy to the backup until you delete the system volume as above. So don't do this unless you need to.

Forward-Looking Statements

It seems clear that the future of bootable backups is unclear.

M1 Macs can't be copied in a way that makes them bootable. Bare metal recovery on an M1 Mac isn't possible, since they depend on the contents of their internal drive even when booting externally. And the tools required to make bootable copies of Intel Macs are limited, often fail, and produce inscrutable and undocumented diagnostics when they do.

Everything's a tradeoff, and with the M1 Macs, Apple has given us an amazing new platform, while taking away some of the things that made macOS such a joy to work with. And one of those things is bootable backups.

I have no idea if this is going to change for the better in whatever the next macOS version brings, and have no insight into Apple's future plans.

But I continue to advise multiple backup strategies, including Time Machine (to an APFS volume under Big Sur), SuperDuper! (for a simple copy of your data and applications) and an online backup program (as a last resort).

With that, back to plugging away at a new version.

Thanks for reading, and for using SuperDuper.

Big Sur Wednesday, November 11, 2020

I'm going to keep this short and to the point.

It's never a good idea to update to a just-released major OS version unless you have to. Nobody knows how reliable Big Sur is going to be for regular users. Let someone else find out before you take the jump.

On our end, SuperDuper! will not be compatible with Big Sur on day of release.

As I indicated on twitter, the first time we were able to make a successful bootable copy of Big Sur at all was November 5th. As of this writing, that was six days ago.

Until that point, it wasn't possible.

So, it's going to be a while. Until then, if you must move to Big Sur, use Time Machine.

Under Construction Friday, September 04, 2020

I haven't said anything about Big Sur yet, and since the public beta has been out for a while, and I'm getting more questions about it, it's time.

At present, it's not possible to make bootable copies of Big Sur, even with asr, Apple's own built-in replication utility. As such, we haven't released a Beta, or even an internal Alpha, because it wouldn't meet our own requirements.

So, for the moment, we're holding back, hoping that Apple will fix the issues and allow 3rd party (or even 1st party, given asr) bootable backups. While asr was failing completely in previous builds, in the most recent one it isn't able to back up because the system volume isn't properly 'sealed' (which is ominous, since why wouldn't a standard install be sealed, and if it's not, why wouldn't you be able to back it up anyway).

So, while progress is being made, we're kind of stuck waiting for the king.

In the meantime, my advice for macOS Betas remains as valid as ever: do not install a macOS Beta unless you have a critical business need to do so. These Betas, even when public, are not for general use, and certainly not for anyone who wants a reliable system for day-to-day work.

If you must install it, use Time Machine for backups for now (and, given my advice in Practices Make Perfect (Backups), on a continuing basis in addition to other methods), since it's at a relatively equivalent Beta quality level to the OS itself and is explicitly tested and supported by Apple.

I'll be back when we have additional news to share. For those in the US, enjoy Labor Day weekend!

I Like Batteries! Friday, August 07, 2020

I've probably mentioned on the blog before that I'm a big fan of electric cars.

Starting in 2013 with a first generation Tesla Model S P85, both Z and I have been powered by electrons, with two leased BMW i3s and we've now replaced the Tesla with a Porsche Taycan.

I was a guest on the I Like Batteries podcast, and like a little yapping dog who just won't shut up, I overstayed my welcome, so it was split into two parts. Part one went live today, and part two drops next week.

Enjoy! (That should probably end in a question mark, or be surrounded by scare quotes.)

https://pod.co/ilikebatteries/030-dave-nanian-likes-battery-powered-cars-part-1-of-2

Never Look a Gift Fix in the Mouth Thursday, July 16, 2020

Happy to say that Apple has fixed the bug in 10.15.5 that caused firmlinks to not be connected after being copied by backup programs.

As you may know, this had two symptoms:

  1. The backup looked incomplete, with some folders seemingly empty on the backup drive.

    Note that the backup was complete, but because the firmlinks weren't fully connected, you couldn't see the files on the Data Volume without specifically looking for them in /Volumes (which you could access via Go To Folder in Finder).

  2. The backup would boot about 2/3 of the way, and then shut down.

Both of these symptoms were caused by the same symptom. We provided a workaround on request for users who needed it, and while convenience was impacted, your data was never in danger.

Fortunately, it's fixed in 10.15.6, and a normal Smart Update will repair your backups and all will be back to normal.

Until Big Sur, of course...where the "adventure" continues. More on that when I have something good to say.

Black Boxes and Bugs Thursday, May 28, 2020

During last year's Catalina introduction, Apple's APFS session introduced a new capability: the asr (Apple Software Restore) tool was now able to perform both APFS copies and, used appropriately, "delta" backups.

We were pretty excited about this new feature, until we actually tried to use it.

The new feature basically didn't work until Catalina's final beta. And even when it started working, while fast, it dealt with failures...poorly.

Sometimes it would leave unusually named volumes hanging around. Sometimes it would leave unmanaged snapshots on a volume. Sometimes the destination drive would be unreadable or unmounted.

And in all cases, it would fail...obscurely. With messages like "Error -49173". Or it would just crash.

Rocks and Hard Places

If you've seen that WWDC session, you know that Apple specifically mentions the new asr capability as being perfect for "Backup programs", and we sort of looked at that as code for you're going to be forced to use this eventually. So, during investigation, when we discovered how incredibly buggy and user-hostile it was, even when it started "working", we took the work we'd done to implement asr support and tabled it.

It's one thing to depend on system features to perform necessary functions: after all, to be a good application, you need to build yourself on top of the APIs that Apple provides.

Much of that work is deciding what "level" you're going to interact at. We're constantly choosing between "higher level" tools (like diskutil) or lower-level APIs (like the IORegistry) to accomplish our tasks.

When there's no significant disadvantage to using a higher level tool, we choose that, because it helps to insulate us from future OS changes: Apple tends to update their own tools to work with new OS releases, and so we have a higher chance of minimizing incompatibilities when a major OS is released.

But one of the "hidden" downside is: sometimes that particular box is way too black. When you put in a file and get out a failure, and the resulting errors are so obscure as to be useless, it's a pretty easy decision: don't use that tool.

Another is less obvious: Apple can move away from the "solution" they were pushing at WWDC, and replace it with "the new hotness", and now you've lost control over your product.

Ch-ch-ch-changes

Regardless of what you pick, sometimes things happen that you can't predict. When Apple makes a change and breaks something, as they have in 10.15.5, they break us...and then we need to react within the framework we've established...or replace what we've selected with something else, because that's the only solution available.

In this case, Apple has broken the ability to make new firmlinks. It's utterly unclear why they broke this capability, but they did. And that makes new and erased SuperDuper! backups unbootable.

Note, though (and this is important): Smart Updates of previous Catalina volumes continue to work properly.

We had hoped Apple would fix the problem before 10.15.5's release, but unfortunately, they didn't. And so we're working on an "automatic" workaround based on the asr work we'd done early in the Catalina beta cycle.

If you have need of a fix "now", please reach out to me at the support email and I'll be happy to assist.

Anniversaries Sunday, April 05, 2020

One year after my Dad's death, I was flying back home, out of the first COVID-19 hot zone in the Northwestern US, listening to people cough, feeling the tension in the air.

It was the same flight I caught rushing home, after cutting our vacation short, after he said not to worry: there was plenty of time, and I should enjoy the snow. The same flight where, well before dawn, Mom said to hurry home, things had taken a turn for the worse. The same flight where, on landing in Denver, I found out there wasn't plenty of time.

We were too late. He died while we were in the air.

Today he would have been 92.

The last voicemail I have from him, he was trying to help Z with her ailing Mom, offering advice from a lifetime of medical experience, researching the best approaches to take.

And in the last few months, as we've all gone through this slow-moving wave of death, sickness and horror, he would have wanted to help. He would have found a way.

He was stubborn that way.

He was stupid that way.

He was selfless that way.

Happy Birthday, Dad. We all miss you.

Modern Times Friday, February 07, 2020

Over the years, Shirt Pocket has been hosted on a number of local Unix-based systems. As I recall, it started out on a Cobalt Qube, then moved to a Power PC Mac mini, then the original "tall" Intel Mac mini, and finally to a "short" Intel mini.

These systems served us well, were easy to manage and, important for the time, meant we were fully "in control" of our site and mail.

There were downsides, though.

Apple has obviously moved away from the whole idea of "macOS Server", leaving a lot of the details of keeping things up to date and relevant to the "modern internet"...and eliminating the original "easy to use and manage" reasons for selecting it in the first place.

But even more than that, since the mini wasn't in a data center, extended power or network outages would take our site down completely—an obviously ridiculous situation. A few years ago, a huge snowstorm took cut power for two weeks, and while I was able to find alternate "hosting" for the site (thanks, Jon!), it wasn't ideal. Not to mention when it would happen when I was on vacation.

Fortunately, while macOS Server is no longer really relevant or useful, in its place are a million cloud-based providers, many from huge vendors like Microsoft, Google and Amazon.

A few years ago, I moved from our own Kerio mail server to one of those: Microsoft's Office 365 Exchange service. That went extremely well, and so, with the threat of winter storms looming, it was time to make the change.

As of Wednesday, all of Shirt Pocket is now "in the cloud", as it were. I'm using SSDNodes, a high quality provider with great support for many Unix variants, Docker, fast bandwidth, etc.

Bruce had used them, successfully, to bring up our Paddle-specific micro services, with no downtime during that transition. A good choice.

So, with that experience under his belt, Bruce guided me through the transition (doing much of the work while I looked over his shoulder, learning how to actually work with Docker, Traefic, etc) and the result?

So, we're now up entirely in the cloud. The Mac mini is off, after probably 10 years of always-on (well, mostly-on) service. One of its RAID drives had died and come back to life, and I had a SuperDuper! backed-up SSD ready to go, but now it won't be needed.

You'll get better service with hopefully no downtime, I don't have to worry about the power going off when I'm away, and thanks to Bruce, I now mostly understand how to use, configure and maintain a Docker-based set of services that, in aggregate, look just like the old web site.

While the dusty old graphics and layout still look like they were created in 2005 (yeah, I know - I have to allocate my limited time carefully), and you probably didn't notice anything had changed, the whole thing works...better!

Can't ask for more than that.

And now, after many years of faithful service, the mini gets a well deserved nap. Well done, little guy. Enjoy the rest.

Page 1 of 27 pages  1 2 3 >  Last »