PDA

View Full Version : Clone larger than source, after erase/copy.


mikebore
09-01-2009, 08:27 AM
MacBook Pro, 10.6, SD 2.6.1

Source Volume is a Utilities partition in the MBP, sized 21.47 Gb of which 6.23 Gb is used.

I used Superduper! to make a clone to a FW external partition sized 21.89 Gb, using the erase and copy option. On completion the amount used on the copy is 9.73 Gb.

First thought was connected to sleepimages, but thats the wrong way round, and there is no sleepimage on the copy.

Any thoughts?

dnanian
09-01-2009, 10:58 AM
Please send in your logs using the "Send to shirt pocket" button, Mike. Could be Spotlight or many other things.

mikebore
09-01-2009, 11:50 AM
Hello Dave,

I have run Superduper! again since the one I posted about and the log only shows the latest. Will it be somewhere else?

I can do it again to generate a log if necessary.

dnanian
09-01-2009, 01:14 PM
No, just use the send to shirt pocket button and the last 10 logs will be sent.

mikebore
09-01-2009, 03:11 PM
Ah! I have already sent the log to you under case number 101098...which was a different problem, which happened just after the copy which started this thread.

dnanian
09-01-2009, 03:19 PM
Didn't know that was you.

It's not entirely clear what the additional data is. The right amount of data is copied (about 5.3GB). The volume ends up larger because of Spotlight and prebinding, I think.

mikebore
09-01-2009, 06:18 PM
I just repeated this erase/copy run and immediately on completion used Whatsize to see where the main space is being used.

The top seven items on the source are:

1.98 Gb System
1.37 Gb Library
718 Mb Applications
547 Mb private
500 Mb usr
396 Mb Users
17.8 Mb mach_kernel

and on the copy are:

3.69 Gb System
2.16 Gb Library
1.11 Gb usr
1.06 Gb Applications
434 Mb private
396 Mb Users
75.4 Mb .Spotlight-V100

Total used on the source is 5.82 Gb, and on the copy is 9.13 Gb

All these are in Whatsize Gb not SL Gb.

Does this make any sense to you?

I have sent the log from this run from the application.

EDIT I meant to add that the spotlight file on the source is zero. It is not excluded from Spotlight by the privacy option.

mikebore
09-01-2009, 06:37 PM
I have set two copies of Whatsize running so I can drill down on the source and copy and compare.

There is no single answer to where the size has gone. Thousands of files are different. Just as a random example the Russian Quicktime help index is 56kb on the source and 168 on the copy!

dnanian
09-01-2009, 08:15 PM
That's really weird. Again, looking into it. Maybe the API we're using to copy files is incorrectly expanding the compressed data (which wouldn't make any sense, but I guess it's possible). It's a single call into the OS... weird.

Anyway, again, still investigating. Even if the data might be larger, it's still "correct"...

mikebore
09-02-2009, 12:24 AM
For the benefit of others, your later reply to me on this in the tech support case was "Yes. I think it's because the backup has been fully prebound, whereas the source hasn't".

Thanks. I guess I need to read up on prebinding, I didn't realise it had that kind of effect!

dnanian
09-02-2009, 12:50 AM
Right: this is a theory we're investigating. We don't know for sure yet, because we're still looking into it... which is why the data in this thread is helpful.

mikebore
09-02-2009, 03:20 AM
FWIW a Chronosync bootable copy results in the identical increased size as Superduper!

Unless Chronosync is also prebinding (which I haven't heard) the cause is perhaps something in the OS.

I did a CCC copy too, but it was block level, so unsurprisingly it was identical to source.

mikebore
09-02-2009, 04:34 AM
Two extra bits of info:

1.A copy of my main boot volume also has the 3-4 Gb "discrepancy".....less noticeable on a 150Gb source than a 5 Gb source.

2.A Superduper! (2.6.1) copy of a Tiger volume does NOT have the discrepancy.

My laymans conclusion is that it is something to do with the OS not prebinding.

dnanian
09-02-2009, 09:35 AM
We've done quite a bit of initial investigation into this, and the file size increase is, indeed, due to compressed files expanding during the copy.

This is not necessarily the same thing as the 'recopying', although the two issues are combined in this thread.

Spot checks of Finder, cp, tar and others shows that they're also expanding the files on copy. Since we use the base-level copy calls that Apple provides, and supplement that with additional metadata copies/checks, and the "external" size looks the same, we both assumed the base call was handling this properly and didn't catch it when looking at the file sizes themselves (since they're reported at their uncompressed size).

We're investigating how to avoid this expansion, and continuing to determine why things are being copied that you'd think wouldn't; more as I've got it (although I might move to blogging about this so it's more useful to others and other developers running into the same problem).

mikebore
09-03-2009, 01:35 AM
Thanks for the update.

Main questions are:

1. Is it happening to everyone using Superduper, CCC and Chronosync? or only some people. I would have expected it to be more widely reported.

2. How much of a concern is it? Everything seems to work. Only problem is loss a few Gb of space. Everything seems to work.

BTW ... can't see any mention of "recopying" in this thread...what is that about?

dnanian
09-03-2009, 10:21 AM
It's happening to everyone that's copying file-by-file from what we can tell. It's hard to notice, though, which is why we didn't catch it.

It's not really a concern. When we figure out a way to fix it, you'll simply get back space on your backup.

Some data in /Applications, /Library and /System is being copied each time we back up. We think it has to do with prebinding: modification dates are changing on the backup drive. Still researching.

MacCetera
09-04-2009, 12:47 PM
Main questions are:

1. Is it happening to everyone using Superduper, CCC and Chronosync? or only some people. I would have expected it to be more widely reported.

As an experiment this morning, I ran the latest CCC (3.3b3) and did Mike's equivalent of a smart update of my boot 10.6 drive to my identical SD clone, and there was a similar 3.2 GB discrepancy over approx. 240 GB of files.

This supports thinking the root problem is a change in the function of the API's under Snow Leopard.

I'm going to try a nuke-n-pave of the clone with both SD and CCC today and see if the results yield any other details.

-- Marc

dnanian
09-04-2009, 01:00 PM
Don't nuke and pave. Let us investigate this, Marc... seriously. We have seen some clues about what's going on. Wiping your own system is just going to take your time unnecessarily.

MacCetera
09-04-2009, 01:38 PM
Don't nuke and pave. Let us investigate this, Marc... seriously. We have seen some clues about what's going on. Wiping your own system is just going to take your time unnecessarily.

Oh well... I erased the clone, and started a CCC run... then left to pick up my daughter. I was hoping that CCC would attempt a block copy, but it's flying along with a file-based copy - currently about 65 of my 240 GB done at 40 minutes.

I'm expecting to see the same bloated results, and taking your advice will leave it be (with the exception of probably running a SD smart update just to make sure my scheduled run at 1 am tonight doesn't bail).

FWIW, I enjoy this sort of stuff... empirical knowledge has value :)

-- Marc

...and some time later:

The file-based CCC copy was larger than the source - in a comparable range to the SD differences.
The SD smart updated backup schedule for this pair had to be recreated because erasing the target volume in DU prior to the CCC test yielded a new UUID for the volume, which naturally didn't pass the SD sniff-test.
After recreating the backup schedule, SD ran through the CCC-created copy like it was its own.