Shirt Pocket Discussions

Shirt Pocket Discussions (
-   General (
-   -   Requesting "Ultra-smart" TM-based backup feature (

avramd 09-12-2009 08:06 PM

Requesting "Ultra-smart" TM-based backup feature

I would like to open a discussion about a possible new feature. Obviously the panacea that everyone wishes either TM or SD! could do is automatically maintain a bootable backup that is up-to-date within the hour.

I believe I just discovered how SD! could do it; I would like a feature where SD combines TimeMachine's backup data, with the inode comparison technique used by timedog, to "UltraSmart Update" to a bootable backup drive by only copying the files that TimeMachine has found changes in since the last time the feature was used on the same target drive.

timedog is able to detect changes between TM backups incredibly quickly because it simply compares inodes. Since TM makes hard links for directories that have not changed, timedog does not have to scan the contents of a directory if its contents did not change. The heavy-lifting of identifying changes has already been done by Time Machine.

Some timedog performance bites: 1.5 secs to identify the changes of a 70-change TM backup of a 2.8GHz MBP w/ 90GB used on the drive being checked), and 2 min to identify the changes of a 20,000-change TM backup of the same drive (this was after updating to iPhoto 9 & letting it update my photo library).

Another benefit of this strategy is that it should be able to run from a network TM backup (e.g. TimeCapsule), since the image file mounts locally as a true HFS+ partition - so the generation of the up-to-date bootable backup could happen on another machine - and you can even take your machine away since TM is already done with it.

Also, you wouldn't need to be this pro-active with SD! - You can just run it say weekly, and let TM do it's thin in the mean time. Then after a crash, you run it again to turn your week-old SD! backup into a <1hr old one.

One obvious caveat of this technique is that the *first* backup would itself have to be generated from Time Machine data, to ensure proper continuity. Rather than building that capability into SD!, it would probably make more sense to generate it using TM's own restore feature.

dnanian 10-06-2009 08:56 AM

Actually, no: the work here is done by fsevents, not Time Machine, really. And improvements to scanning can be made in future versions of SD, but the more coupling we have to complex software systems (Time Machine, fsevents), the more likelihood there is that a flaw in those systems will impact all your backups, rather than just one.

In general, that's unwise. SD! tries to do things in a simple, straightforward way. It's easier to test, more reliable, and the only real downside is a bit of extra time taken. Given that, on my 3-year+ old system, it takes < 18 minutes (16 minutes last time) to update a backup of over 2.5M files to a regular FW400 drive, the time savings isn't worth the additional risk/coupling until those systems are very well established (and perhaps not even then).

avramd 10-06-2009 09:38 AM

Ok, that is a valid point. So let me paint the picture a little differently: How often do you backup with SD!?

Here is how my life currently looks w/rt backups: TM backs up hourly. SD! backs up daily, in the middle of the night.

Two weeks ago, I had a drive failure at about 5pm. I had been working all day. I didn't particularly remember what I had touched that day. Here are the options I had for recovery:

1) swap in the SD! backup drive and be up and running immediately, but with the task of manually pulling from TM everything I could think of that had changed (time consuming), and living with an unknown amount of lost data (risky)

2) ignore the SD! backup drive, and restore from the TM backup being certain that everything I have is as fresh as possible (time consuming but not risky)

I took option 2. But in thinking it through, I realized that I can not think of a single situation in which I would take option 1 (well except for if I had an HD crash at 3am that I knew for sure happened after my SD! backup had completed, but before I had done any other work that TM! caught).

What I really want is an automated way to take an SD! backup and bring it up to date w/rt the latest TM backup. It just doesn't seem reasonable to run SD! hourly - if it takes 16 minutes to run, that means 25% of the time, your computer is running a process that is either pegging the CPU or the disk.

I guess another way to describe what I think needs to exist is an offline utility that can repeatedly update an offline backup based on the latest TM run. Not couple the logic of SD! & TM, just use the TM logic alone, but in a different way. It's reasonable to say that this would be an entirely different utility from SD! - but one that is sorely needed, IMHO.

dnanian 10-06-2009 10:36 AM

In general, I find it pretty easy to know what I've been doing in the past few hours, so if there are critical things I need to restore from another backup, it's relatively easy to find them. But this is a usability problem with Time Machine, most certainly - it doesn't handle "please restore everything from the last four hours" well at all.

But, as I said, I don't think this is, in general, appropriate or desirable coupling between systems. One of the main advantages of using two separate programs to back up is to have additional failure modes covered.

If your Time Machine backup gets corrupted (and a small corruption can have massive impact on a Time Machine volume), your SuperDuper! volume is there for you. Similarly, if you have some critical files you absolutely must restore from your Time Machine backup, that's there too.

Tying both together in any significant way compounds, rather than reduces, any potential errors that either might encounter alone.

JoBoy 10-06-2009 11:13 AM


Originally Posted by avramd (Post 27282)
Ok, that is a valid point... So let me paint the picture a little differently: thinking it through, I realized that I can not think of a single situation in which I would take option 1 (well except for if I had an HD crash at 3am that I knew for sure happened after my SD! backup had completed, but before I had done any other work that TM! caught).

Unfortunately, I've had failures of my system twice in the recent past from a graphics card defect. The first time, I, too, took option 1 for the same reason, but I took option 2 the second time because I was aware of what data would be missing if I took option 2. There is nothing like real-life failures to stimulate thinking along the lines of this thread. However, I agree with the view that tying TM to SD! poses practical problems. There may even be proprietary issues with Apple.

Although it isn't true, it seems like TM is constantly trying to back up data. I've learned to ignore it and continue on with my work despite the constant background noise of the nearby Time Capsule. (I network TC via GbEthernet rather than using an external Firewire drive.)

My wish list includes a Super Duper! that runs simultaneously with my work flow and provides an updated, hourly backup, but not the versioning that is TM's stock in trade. That way, there is room for both apps in the market. TM would continue in its slow, versioning way and SD! would provide a quicker and reasonably current restoration that I would nearly always prefer because it restores a bootable system to my main drive in a very simple, very quick manner. I would use TM to restore an individual file or folder that I had previously deleted.

I have a concern about restoring from any copy that was made immediately prior to a system failure. Can such a copy perpetuate early file corruption that later leads to a system failure? This may be more of a worry than a valid concern, but I'd prefer something not quite so current for that reason. Losing up to 1 hour's work doesn't bother me, but concern over perpetuating corrupted files does bother me. Again, is that a valid concern? Could a really recent backup be programmed to quarantine the files copied during the last few minutes before a crash so that a person could elect to use them or not depending upon the cause of the crash?

All times are GMT -4. The time now is 10:50 PM.

Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2018, vBulletin Solutions, Inc.