Shirt Pocket Discussions  
    Home netTunes launchTunes SuperDuper! Buy Now Support Discussions About Shirt Pocket    

Go Back   Shirt Pocket Discussions > SuperDuper! > General

Reply
 
Thread Tools Rating: Thread Rating: 2 votes, 5.00 average. Display Modes
  #1  
Old 06-10-2005, 02:13 AM
steve steve is offline
Registered User
 
Join Date: Jun 2005
Location: Billings, MT
Posts: 4
Question Concerned about hard links

I'm a new owner of SuperDuper, and I've been doing some testing under Mac OS X 10.3.9.

One discovery has me a bit puzzled and concerned. After doing a simple clone operation to create a bootable volume, I see that hard links are apparently not preserved. This can be observed in Terminal by using the "ls -li" command. Source files (not directories) with a link count higher than 1 inevitably show up on the target with a link count of 1.

Examining inode numbers adds more evidence that hard links are lost. Instead of multiple directory entries sharing a single inode number, each directory entry ends up with a unique inode number. That means that the original file, which had multiple directory entries pointing to it, has been duplicated so that each directory entry points to its own copy.

This seems like a waste of space, but more importantly, could lead to software malfunctions if hard links are assumed to exist when files are updated. Do we know that all those hard linked files are never manipulated by Mac OS X, so there's no need to worry? Couldn't third party software use hard links and get tripped up by this too?

Hard links are especially abundant in /usr/share/...; there are hundreds and hundreds of them. (A simple test directory is /usr/share/zoneinfo/US.)

For comparison I tried cloning using the asr command in Terminal, and also Carbon Copy Cloner (which uses the ditto command). They both play it safe and preserve hard links.

I'm new to SuperDuper, and I'm no expert on unix either, but this seems like an issue worth exploring. Anyone more savvy about these things have any thoughts?

Steve
Reply With Quote
  #2  
Old 06-10-2005, 03:53 AM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,358
Send a message via AIM to dnanian
Steve,

You are, indeed, right: we unroll hard links -- but from what I can tell, ditto does too, sometimes. Here's how I tested: I created a disk image named "test", and -- from /usr/share/zoneinfo -- I did:

ditto US /Volumes/test/US

When I did this, all the hard links were unrolled, kind of as I would expect.

ASR is rather different, since it does a very low-level block copy, and doesn't deal with files. It also can't subset the drive (which, clearly, can't always guarantee the 'original' inode will be copied, so if you don't unroll you end up with a bad copy).

ditto only seems to copy hard links if the file with the original inode is copied. I tested this by touching "c" and hard linking "c.copy" to it. I then copied both files to the root of the test volume, and they were preserved as hard links.

We've not encountered a problem, yet, with what we're doing, but I agree that it's something that we should take a look at -- and we shall. Thanks for reporting it.
__________________
--Dave Nanian
Reply With Quote
  #3  
Old 06-11-2005, 02:56 AM
steve steve is offline
Registered User
 
Join Date: Jun 2005
Location: Billings, MT
Posts: 4
Dave,

Thanks for the quick response. I hope your time at WWDC was useful and enjoyable!

I should have been more clear about how I used /usr/share/zoneinfo/US for testing. The hard linked files in that directory aren't hard linked to each other, so copying that directory by itself doesn't seem very revealing. In my tests, I cloned the entire boot volume, then checked /usr/share/zoneinfo/US on the target to quickly see what happened to link counts. Sorry for the confusion.

Also, I did not test ditto directly. I simply used Carbon Copy Cloner, and observed that hard links were preserved. According to CCC's log, it uses a series of ditto commands like this:

sudo /usr/bin/ditto -rsrcFork /'Applications' '/Volumes/<mytarget>'/'Applications'
sudo /usr/bin/ditto -rsrcFork /'usr' '/Volumes/<mytarget>'/'usr'
etc.

I would imagine that as long as all hard links to a particular inode are within one of the copied folders (like usr), ditto wouldn't need to unroll any of them. (Apparently that condition holds in Mac OS X's hard links, because I found no reduction in link counts within the CCC generated clone.)

Regarding asr, I used it like this:

sudo asr -source / -target /Volumes/<mytarget> -erase -verbose

In this simple application (source not an image), it apparently does not do a low-level block copy. My evidence is that modification dates of all symbolic link files (on the target) were updated during asr's operation. Ditto does the same thing. (I don't understand why in either case.) Note that asr and CCC/ditto took almost exactly the same amount of time to complete the clone, suggesting they're using the same copying method.

Incidentally, to fully explore the results of a cloning operation, I generate complete volume listings (sudo ls -lonTRA > ~/desktop/list.txt) of source and target, and compare them with FileMerge. It's time consuming, but interesting. That's how I discovered the loss of hard links with SuperDuper, and confirmed their complete preservation with asr and CCC. The symbolic link updating phenomenon with asr and CCC is a real nuisance, because it causes huge numbers of uninteresting differences.

Thanks for looking into the hard links issue. I'd love to see a FileMerge source/target listing comparison that shows only expected differences, rather than hundreds of disquieting extraneous ones caused by quirks of the cloning method.

Steve
Reply With Quote
  #4  
Old 06-11-2005, 03:23 AM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,358
Send a message via AIM to dnanian
Understood, Steve. We're looking into the case. I'm not sure ditto preserves the links if they go across "copy operations" (in other words, if in your example something in /usr is linked to something in /Applications, it likely won't have its link preserved).

It's a bit tough, because unlike a soft link a hard link can't be created until the target exists, and there's no real "target" as such since they're all really the same file.

In any case, we're looking at it. Thanks again.
__________________
--Dave Nanian
Reply With Quote
  #5  
Old 06-11-2005, 03:25 AM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,358
Send a message via AIM to dnanian
(Note, though, that -- again -- this has not proved to be a significant issue in real world use. It's "wrong", though, which is why we're looking at how to fix it, but it's never caused any problems because hard links are pretty rare in an OSX installation.)
__________________
--Dave Nanian
Reply With Quote
  #6  
Old 06-12-2005, 12:16 AM
steve steve is offline
Registered User
 
Join Date: Jun 2005
Location: Billings, MT
Posts: 4
Dave,

Thanks for your reassurance that unrolling hard links has not caused any trouble. I agree with your observation that ditto probably can't preserve links "across copy operations".

For fun, I checked to see how many hard links are in my OS X installations. My method was to find non-directory files with non-zero link counts, using commands like those below.

cd /Volumes/whatever
sudo find . ! -links 1 ! -type d | wc -l
sudo find . ! -links 1 ! -type d -ls > ~/desktop/'hardlinks.txt'

Hard links are pretty rare in 10.2.8, only 266 total.
/Library = 3
/System = 3 (these 3 are linked to the 3 in /Library)
/sbin = 4
/usr = 256

But 10.3.9 has lots more, 2839 total.
/Library = 3
/System = 3 (these 3 are linked to the 3 in /Library)
/usr/bin = 22
/usr/share/man = 1112
/usr/share/terminfo = 1478
/usr/share/zoneinfo = 221

I hope you find these results useful. It's been fun digging around in OS X's file system, but I'll quit bugging you now. Thanks for a fine product, and amazingly responsive support!

Steve
Reply With Quote
  #7  
Old 06-12-2005, 11:26 AM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,358
Send a message via AIM to dnanian
Thanks for the additional information. We're actively pursuing a solution.
__________________
--Dave Nanian
Reply With Quote
  #8  
Old 06-13-2005, 03:01 AM
sjk's Avatar
sjk sjk is offline
Registered User
 
Join Date: May 2004
Location: Eugene
Posts: 252
Thanks for noticing and reporting this issue, Steve. I've wondered what was causing broken hard links of files on certain system volumes but had never correlated it with SD!. Awhile ago that might have become a significant space-consuming problem for some hard linked large media files if they'd been backed up using SD!.

If anyone's interested, running "find /path/of/folder -type f -links +1 -ls" in a Terminal shell is one way to locate and list hard linked files. The second column of the output is the inode number.

Hopefully there will eventually be a fix for this. Thanks for investigating it, Dave.
Reply With Quote
  #9  
Old 06-14-2005, 01:16 AM
steve steve is offline
Registered User
 
Join Date: Jun 2005
Location: Billings, MT
Posts: 4
You're welcome, sjk. I was afraid maybe this whole topic was too esoteric to bother about, so I'm glad you found it useful.

And thanks for posting your suggested command for listing hard links. It's certainly more sensible than the bass-ackwards method I posted. I guess you can tell I'm not a unix guy, eh?

Steve
Reply With Quote
  #10  
Old 06-19-2006, 06:27 PM
garoomph garoomph is offline
Registered User
 
Join Date: Jun 2006
Posts: 29
DarwinPorts maintains hard links to all files it installed in /opt/local below /opt/local/var/db/dports/software. As SD unrolls these while backing up, the clone may be significantly larger than the source. This could cause problems when restoring a drive that was nearly full before it failed.

At the moment I can't think of applications relying on changing file contents through hard linking (as opposed to the usual "unlink on overwrite").

To me it generally feels wrong that a "clone"'s filesystem should be differ from its source in such a fundamental way.
Reply With Quote
  #11  
Old 06-19-2006, 06:29 PM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,358
Send a message via AIM to dnanian
SuperDuper! does not unroll hard links: we replicate the hard links on the destination. (These messages are from a long time ago: the issue with hard links was corrected many, many moons ago.)
__________________
--Dave Nanian

Last edited by dnanian; 06-19-2006 at 06:32 PM.
Reply With Quote
  #12  
Old 06-19-2006, 06:39 PM
garoomph garoomph is offline
Registered User
 
Join Date: Jun 2006
Posts: 29
Quote:
Originally Posted by dnanian
SuperDuper! does not unroll hard links
Really? Always?

Of course you can't answer for DarwinPorts, but in the following (ls -i) sitecopy was installed before a HD->Clone->HD restore and mplayer afterwards:

436867 /private/opt/local/var/db/dports/software/sitecopy/0.16.0_0/opt/local/bin/sitecopy
426257 /opt/local/bin/sitecopy

426249 /opt/local/bin/mplayer
426249 /private/opt/local/var/db/dports/software/MPlayer/cvs-20060408_0+darwin_8+macosx+real/opt/local/bin/mplayer
Reply With Quote
  #13  
Old 06-19-2006, 06:46 PM
dnanian's Avatar
dnanian dnanian is offline
Administrator
 
Join Date: Apr 2001
Location: Weston, MA
Posts: 14,358
Send a message via AIM to dnanian
If it did unroll something, it's a bug: we specifically check for and recreate hard links.

I just ran a simple test: I created two disk images, "linktest1" and "linktest2". I touched a file and then created a hard link to it:

Code:
Daves-15-MacBook:/Volumes/linktest1 dnanian$ touch foo
Daves-15-MacBook:/Volumes/linktest1 dnanian$ ln foo foo2
Daves-15-MacBook:/Volumes/linktest1 dnanian$ ls -l
total 0
-rw-r--r--   2 dnanian  dnanian  0 Jun 19 17:44 foo
-rw-r--r--   2 dnanian  dnanian  0 Jun 19 17:44 foo2
Daves-15-MacBook:/Volumes/linktest1 dnanian$ ls -i
19 foo  19 foo2
I then used "Backup - all files" to copy to "linktest2":

Code:
Daves-15-MacBook:/Volumes/linktest1 dnanian$ cd
Daves-15-MacBook:~ dnanian$ cd /Volumes/linktest2
Daves-15-MacBook:/Volumes/linktest2 dnanian$ ls -i
19 foo  19 foo2
Daves-15-MacBook:/Volumes/linktest2 dnanian$
So, it seems to work... checking other known hard links seem fine here, too...
__________________
--Dave Nanian
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Can't boot from firewire disk negativeduck General 32 01-27-2008 04:54 PM
What Hard Drive to use rayc325 General 7 06-18-2006 08:22 AM
Hard drive size needed for backup? Hoosier_1701 General 8 05-20-2006 08:49 AM
SuperDuper! as backup/restore program? Staxman General 17 06-02-2005 02:30 PM
Partition an external hard drive problem andrew_aes General 1 05-25-2005 08:45 AM


All times are GMT -4. The time now is 09:48 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2017, vBulletin Solutions, Inc.