Shirt Pocket Discussions

Shirt Pocket Discussions (http://www.shirt-pocket.com/forums/index.php)
-   General (http://www.shirt-pocket.com/forums/forumdisplay.php?f=6)
-   -   Data integrity question- how much verification by SuperDuper? (http://www.shirt-pocket.com/forums/showthread.php?t=365)

stevea 05-12-2005 01:43 PM

Data integrity question- how much verification by SuperDuper?
 
What kinds of verification tests does SuperDuper perform to ensure data integrity of the backup copy?
________
VAPOR TOWER

stevea 05-12-2005 05:02 PM

Quote:

Originally Posted by dnanian
We don't do any kind of "explicit" verification, since the disk controller itself performs that task for us.

Our general take is that you don't ask the Finder to verify things when you copy, and you don't do it when you write a file to disk with an editor (or whatever). That's because a hard disk has "built-in verification" -- its controller flags errors when they occur, on either read or write.

If we hit an error like this, we stop, because there's been a problem. But compare-after-write would both take time and cause unnecessary wear and tear on the drive.

(This is different from writing to a medium that is inherently prone to failure, like "older" Travan tapes, CDs or floppies...)


Yes, the built-in verification prevents media errors or device errors... but, it does not prevent errors or verify that the files are copied correctly to begin with. (For example, are all files copied? are permissions correct? etc.)

Also, the SuperDuper documentation specifies that "[files which Apple recommends not to copy]" are not copied... yet, there is no explicit documentation or verification to determine exactly what is copied and what is not copied.

Thus, it is not possible to determine if the backup copy is identical to the original or not. Presumably, it is not identical... but, not knowing the exact differences, makes it more difficult to determine if data integrity was maintained or not.
________
Detroit diesel v8 engine

dnanian 05-12-2005 05:04 PM

Steve, if you don't trust us to copy the files properly, how can you trust us to "verify" those files?

The files that are not copied are in the scripts themselves. You're welcome to look in there: every file that's not copied is listed.

stevea 05-12-2005 05:17 PM

Quote:

Originally Posted by dnanian
Steve, if you don't trust us to copy the files properly, how can you trust us to "verify" those files?

I don't wish to argue with you. But, availability of a verification feature would at least indicate some concern for data integrity on your part.

On a technical level, it seems that verification is less prone to troubles that plague copying and backup utilities on OS X.

There are lots of other reasons why verification is valuable and important, which I am sure you don't need me to list here.


Quote:

Originally Posted by dnanian
The files that are not copied are in the scripts themselves. You're welcome to look in there: every file that's not copied is listed.

Thanks, that's helpful!
________
Ford falcon (argentina) specifications

dnanian 05-12-2005 05:28 PM

I'm not trying to argue: I just want to make sure you understand our position.

We spend a lot of time testing and ensuring that we do the right thing when we copy. We obviously have a high regard for data integrity, and spend considerable effort ensuring that we do the right thing, and the software is never released to make a date: it's released when it's been thoroughly tested.

We have internal tools that do careful comparisons of source and destination to ensure we're doing the right thing, and we don't release versions that don't pass our extensive battery of tests.

Building those tools into the main application doesn't really offer any benefits. If we ensured we do the right thing beforehand, then it's going to do the right thing. If we didn't, we'd have made a mistake in both the comparison/testing tools AND the program. And thus, you wouldn't see the problem when you relied on our own tools to verify a copy.

If you're concerned about this, I'd suggest finding a comparison/verification tool from a different author -- that way, you'd get some "independence" when looking at the files, thus isolating yourself (at some level, at least) from any algorithmic issues that would effect both sides of our own implementation.

stevea 05-12-2005 05:36 PM

Quote:

Originally Posted by dnanian
Building those tools into the main application doesn't really offer any benefits. If we ensured we do the right thing beforehand, then it's going to do the right thing. If we didn't, we'd have made a mistake in both the comparison/testing tools AND the program. And thus, you wouldn't see the problem when you relied on our own tools to verify a copy.

Thanks. I understand your position, but very strongly disagree with your opinion and logic here.

Obviously, you rely and trust your verification tools and consider them very valuable to you. So, it makes little sense for you to turn around and say they would have little or no value to anyone else.

Do you think you are the only person concerned with matters of data integrity?

How do you think other people would want to trust your software, if you don't even want to make available tests to verify the integrity of its copying?




Quote:

Originally Posted by dnanian
If you're concerned about this, I'd suggest finding a comparison/verification tool from a different author -- that way, you'd get some "independence" when looking at the files, thus isolating yourself (at some level, at least) from any algorithmic issues that would effect both sides of our own implementation.

Of course.

In part, my questions are related to resolving differences in results of using various other tools, both to perform copying tasks and verification tasks.
________
Fz6

dnanian 05-12-2005 05:40 PM

Thanks for your comments.

stevea 05-12-2005 06:02 PM

Thanks for your replies! :)


Another situation-

I am trying SuperDuper to backup a disk that has some problems. (Disk Utility reported some problems, DiskWarrior reported the directory was too damaged to rebuild, Disk Utility repaired the problems and says the disk is OK, but DiskWarrior reports the directory is still too damaged to rebuild.)

Other verification tools (Synchronize! Pro X 3.6.1) is running into errors when comparing the original disk and the SuperDuper backup copy, so I cannot verify the backup completely. And yet, SuperDuper made the backup copy without reporting any errors, so I am trying to analyze and understand what data can be trusted here...
________
E cigarette

dnanian 05-12-2005 06:07 PM

We're very conservative about error handling. If we get back any errors at all during the process, we stop and log the file (as well as a bunch of information about it).

Of course, if the directory damage involves the tree we're walking -- and leaves files out -- there's no way for us to know what's not shown to us by the OS...

What specific errors is SPX running into here? A comparison error, or an inability to walk the tree?

stevea 05-12-2005 06:18 PM

Quote:

Originally Posted by dnanian
What specific errors is SPX running into here? A comparison error, or an inability to walk the tree?

The SPX error: "* An error occurred while getting the location of the parent directory of a file to be written. Current privileges do not allow the operation."

This error appears during a verify-only, so nothing is to be written. I'm not sure if it is a typo in error message or a revelation of some hack or low-level error in getting data for verification purposes.

SPX has its share of bugs and I'm not very happy with it... it's another motivation for this thread- just seeking better solutions.
________
Michigan marijuana dispensaries

stevea 05-12-2005 06:20 PM

Quote:

Originally Posted by dnanian
Of course, if the directory damage involves the tree we're walking -- and leaves files out -- there's no way for us to know what's not shown to us by the OS...


Right. But, as a user, that and other problems are what I want to detect and know about (as I'm sure you understand)! :)
________
LINCOLN MARK SERIES PICTURE

dnanian 05-12-2005 06:27 PM

That's unusual. I have no idea what they're trying to do there...

dnanian 05-12-2005 06:37 PM

(Something weird happened on the forums, and the original reply to this message has moved to a different thread with a deleted parent. Weird: go figure. Here's the original text.)

We don't do any kind of "explicit" verification, since the disk controller itself performs that task for us.

Our general take is that you don't ask the Finder to verify things when you copy, and you don't do it when you write a file to disk with an editor (or whatever). That's because a hard disk has "built-in verification" -- its controller flags errors when they occur, on either read or write.

If we hit an error like this, we stop, because there's been a problem. But compare-after-write would both take time and cause unnecessary wear and tear on the drive.

(This is different from writing to a medium that is inherently prone to failure, like "older" Travan tapes, CDs or floppies...)

Hope that helps!

stevea 05-12-2005 07:02 PM

Quote:

Originally Posted by dnanian
(Something weird happened on the forums, and the original reply to this message has moved to a different thread with a deleted parent. Weird: go figure.

I first posted the original message in a different thread, then, after I realized my mistake, deleted it and reposted here as a new thread.

You were just too quick, and replied to the first message that I deleted in the other thread! :D
________
Ford nucleon picture

dnanian 05-12-2005 07:05 PM

Oh, Jeez, good. I was becoming concerned that things were well and truly hosed here!


All times are GMT -4. The time now is 04:09 AM.

Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.