Shirt Pocket Discussions  
    Home netTunes launchTunes SuperDuper! Buy Now Support Discussions About Shirt Pocket    

Go Back   Shirt Pocket Discussions > SuperDuper! > General

Reply
 
Thread Tools Rate Thread Display Modes
  #1  
Old 05-12-2005, 01:43 PM
stevea stevea is offline
Registered User
 
Join Date: May 2005
Posts: 38
Data integrity question- how much verification by SuperDuper?

What kinds of verification tests does SuperDuper perform to ensure data integrity of the backup copy?
________
VAPOR TOWER

Last edited by stevea; 03-10-2011 at 11:46 AM.
Reply With Quote
  #2  
Old 05-12-2005, 05:02 PM
stevea stevea is offline
Registered User
 
Join Date: May 2005
Posts: 38
Quote:
Originally Posted by dnanian
We don't do any kind of "explicit" verification, since the disk controller itself performs that task for us.

Our general take is that you don't ask the Finder to verify things when you copy, and you don't do it when you write a file to disk with an editor (or whatever). That's because a hard disk has "built-in verification" -- its controller flags errors when they occur, on either read or write.

If we hit an error like this, we stop, because there's been a problem. But compare-after-write would both take time and cause unnecessary wear and tear on the drive.

(This is different from writing to a medium that is inherently prone to failure, like "older" Travan tapes, CDs or floppies...)

Yes, the built-in verification prevents media errors or device errors... but, it does not prevent errors or verify that the files are copied correctly to begin with. (For example, are all files copied? are permissions correct? etc.)

Also, the SuperDuper documentation specifies that "[files which Apple recommends not to copy]" are not copied... yet, there is no explicit documentation or verification to determine exactly what is copied and what is not copied.

Thus, it is not possible to determine if the backup copy is identical to the original or not. Presumably, it is not identical... but, not knowing the exact differences, makes it more difficult to determine if data integrity was maintained or not.
________
Detroit diesel v8 engine

Last edited by stevea; 03-10-2011 at 11:46 AM.
Reply With Quote
  #3  
Old 05-12-2005, 05:04 PM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,923
Send a message via AIM to dnanian
Steve, if you don't trust us to copy the files properly, how can you trust us to "verify" those files?

The files that are not copied are in the scripts themselves. You're welcome to look in there: every file that's not copied is listed.
__________________
--Dave Nanian
Reply With Quote
  #4  
Old 05-12-2005, 05:17 PM
stevea stevea is offline
Registered User
 
Join Date: May 2005
Posts: 38
Quote:
Originally Posted by dnanian
Steve, if you don't trust us to copy the files properly, how can you trust us to "verify" those files?
I don't wish to argue with you. But, availability of a verification feature would at least indicate some concern for data integrity on your part.

On a technical level, it seems that verification is less prone to troubles that plague copying and backup utilities on OS X.

There are lots of other reasons why verification is valuable and important, which I am sure you don't need me to list here.


Quote:
Originally Posted by dnanian
The files that are not copied are in the scripts themselves. You're welcome to look in there: every file that's not copied is listed.
Thanks, that's helpful!
________
Ford falcon (argentina) specifications

Last edited by stevea; 03-10-2011 at 11:46 AM.
Reply With Quote
  #5  
Old 05-12-2005, 05:28 PM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,923
Send a message via AIM to dnanian
I'm not trying to argue: I just want to make sure you understand our position.

We spend a lot of time testing and ensuring that we do the right thing when we copy. We obviously have a high regard for data integrity, and spend considerable effort ensuring that we do the right thing, and the software is never released to make a date: it's released when it's been thoroughly tested.

We have internal tools that do careful comparisons of source and destination to ensure we're doing the right thing, and we don't release versions that don't pass our extensive battery of tests.

Building those tools into the main application doesn't really offer any benefits. If we ensured we do the right thing beforehand, then it's going to do the right thing. If we didn't, we'd have made a mistake in both the comparison/testing tools AND the program. And thus, you wouldn't see the problem when you relied on our own tools to verify a copy.

If you're concerned about this, I'd suggest finding a comparison/verification tool from a different author -- that way, you'd get some "independence" when looking at the files, thus isolating yourself (at some level, at least) from any algorithmic issues that would effect both sides of our own implementation.
__________________
--Dave Nanian
Reply With Quote
  #6  
Old 05-12-2005, 05:36 PM
stevea stevea is offline
Registered User
 
Join Date: May 2005
Posts: 38
Quote:
Originally Posted by dnanian
Building those tools into the main application doesn't really offer any benefits. If we ensured we do the right thing beforehand, then it's going to do the right thing. If we didn't, we'd have made a mistake in both the comparison/testing tools AND the program. And thus, you wouldn't see the problem when you relied on our own tools to verify a copy.
Thanks. I understand your position, but very strongly disagree with your opinion and logic here.

Obviously, you rely and trust your verification tools and consider them very valuable to you. So, it makes little sense for you to turn around and say they would have little or no value to anyone else.

Do you think you are the only person concerned with matters of data integrity?

How do you think other people would want to trust your software, if you don't even want to make available tests to verify the integrity of its copying?




Quote:
Originally Posted by dnanian
If you're concerned about this, I'd suggest finding a comparison/verification tool from a different author -- that way, you'd get some "independence" when looking at the files, thus isolating yourself (at some level, at least) from any algorithmic issues that would effect both sides of our own implementation.
Of course.

In part, my questions are related to resolving differences in results of using various other tools, both to perform copying tasks and verification tasks.
________
Fz6

Last edited by stevea; 03-10-2011 at 11:47 AM.
Reply With Quote
  #7  
Old 05-12-2005, 05:40 PM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,923
Send a message via AIM to dnanian
Thanks for your comments.
__________________
--Dave Nanian
Reply With Quote
  #8  
Old 05-12-2005, 06:02 PM
stevea stevea is offline
Registered User
 
Join Date: May 2005
Posts: 38
Thanks for your replies!


Another situation-

I am trying SuperDuper to backup a disk that has some problems. (Disk Utility reported some problems, DiskWarrior reported the directory was too damaged to rebuild, Disk Utility repaired the problems and says the disk is OK, but DiskWarrior reports the directory is still too damaged to rebuild.)

Other verification tools (Synchronize! Pro X 3.6.1) is running into errors when comparing the original disk and the SuperDuper backup copy, so I cannot verify the backup completely. And yet, SuperDuper made the backup copy without reporting any errors, so I am trying to analyze and understand what data can be trusted here...
________
E cigarette

Last edited by stevea; 03-10-2011 at 11:48 AM.
Reply With Quote
  #9  
Old 05-12-2005, 06:07 PM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,923
Send a message via AIM to dnanian
We're very conservative about error handling. If we get back any errors at all during the process, we stop and log the file (as well as a bunch of information about it).

Of course, if the directory damage involves the tree we're walking -- and leaves files out -- there's no way for us to know what's not shown to us by the OS...

What specific errors is SPX running into here? A comparison error, or an inability to walk the tree?
__________________
--Dave Nanian
Reply With Quote
  #10  
Old 05-12-2005, 06:18 PM
stevea stevea is offline
Registered User
 
Join Date: May 2005
Posts: 38
Quote:
Originally Posted by dnanian
What specific errors is SPX running into here? A comparison error, or an inability to walk the tree?
The SPX error: "* An error occurred while getting the location of the parent directory of a file to be written. Current privileges do not allow the operation."

This error appears during a verify-only, so nothing is to be written. I'm not sure if it is a typo in error message or a revelation of some hack or low-level error in getting data for verification purposes.

SPX has its share of bugs and I'm not very happy with it... it's another motivation for this thread- just seeking better solutions.
________
Michigan marijuana dispensaries

Last edited by stevea; 03-10-2011 at 11:48 AM.
Reply With Quote
  #11  
Old 05-12-2005, 06:20 PM
stevea stevea is offline
Registered User
 
Join Date: May 2005
Posts: 38
Quote:
Originally Posted by dnanian
Of course, if the directory damage involves the tree we're walking -- and leaves files out -- there's no way for us to know what's not shown to us by the OS...

Right. But, as a user, that and other problems are what I want to detect and know about (as I'm sure you understand)!
________
LINCOLN MARK SERIES PICTURE

Last edited by stevea; 03-10-2011 at 11:48 AM.
Reply With Quote
  #12  
Old 05-12-2005, 06:27 PM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,923
Send a message via AIM to dnanian
That's unusual. I have no idea what they're trying to do there...
__________________
--Dave Nanian
Reply With Quote
  #13  
Old 05-12-2005, 06:37 PM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,923
Send a message via AIM to dnanian
(Something weird happened on the forums, and the original reply to this message has moved to a different thread with a deleted parent. Weird: go figure. Here's the original text.)

We don't do any kind of "explicit" verification, since the disk controller itself performs that task for us.

Our general take is that you don't ask the Finder to verify things when you copy, and you don't do it when you write a file to disk with an editor (or whatever). That's because a hard disk has "built-in verification" -- its controller flags errors when they occur, on either read or write.

If we hit an error like this, we stop, because there's been a problem. But compare-after-write would both take time and cause unnecessary wear and tear on the drive.

(This is different from writing to a medium that is inherently prone to failure, like "older" Travan tapes, CDs or floppies...)

Hope that helps!
__________________
--Dave Nanian
Reply With Quote
  #14  
Old 05-12-2005, 07:02 PM
stevea stevea is offline
Registered User
 
Join Date: May 2005
Posts: 38
Quote:
Originally Posted by dnanian
(Something weird happened on the forums, and the original reply to this message has moved to a different thread with a deleted parent. Weird: go figure.
I first posted the original message in a different thread, then, after I realized my mistake, deleted it and reposted here as a new thread.

You were just too quick, and replied to the first message that I deleted in the other thread!
________
Ford nucleon picture

Last edited by stevea; 03-10-2011 at 11:48 AM.
Reply With Quote
  #15  
Old 05-12-2005, 07:05 PM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,923
Send a message via AIM to dnanian
Oh, Jeez, good. I was becoming concerned that things were well and truly hosed here!
__________________
--Dave Nanian
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
PROBLEM: (BUG?) Copied disc (or not?)- files not visible in Finder stevea General 7 05-13-2005 10:47 PM
Minor concerns regarding SuperDuper giba General 1 05-02-2005 05:06 PM
Security of SuperDuper Backup Data? Zeigh General 1 08-08-2004 12:29 PM
Another review: MaMUGs looks at SuperDuper! dnanian General 0 01-26-2004 09:26 AM


All times are GMT -4. The time now is 10:14 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2024, vBulletin Solutions, Inc.