PDA

View Full Version : LS: Why is Smart Update so large?


SteveL
08-30-2009, 01:25 AM
Just curious, but if I run SD! (2.6.1), surf, e-mail, and then run SD! again, some 2.5GB are updated. In 10.5, less than 1GB would have been updated. Seems odd, at least to me (but, what do I know!).

dnanian
08-30-2009, 09:02 AM
We've had some reports of this, and we're investigating to see why some files under Snow Leopard files are comparing different when users indicate they haven't really changed anything.

Better copying too much than too little, though. :)

robbchadwick
08-30-2009, 01:42 PM
I upgraded to Snow Leopard Friday evening. I ran disk permissions before and after, of course. I also did a complete erase & backup with SuperDuper ... which took about the usual time for me ... 60 gb of files ... a little over an hour on USB (newer MacBook but without FireWire).

I just now ran Smart Update (which I do every 24-36 hours BTW); and Smart Update took almost 30 minutes. It normally takes about 5 minutes ... no more than 10 when I've changed a lot of files during the day.

No complaints. I still love SuperDuper ... couldn't live without it ... but it does take longer now.

dnanian
08-30-2009, 01:44 PM
Let's see if this changes over time. We're investigating what might be going on, but it's not clear at present, certainly not something that would change 5 minutes to 30...

MacCetera
08-31-2009, 05:53 PM
I'm also hoping that the cause of slow smart updates with a Mac OS X 10.6 / SD 2.6.1 configuration gets attention. My 250+GB boot drive clone has gone from about 10 minutes to almost 20, but worse, my nightly 800+GB Time Machine clone has jumped from under 4 hours to over 9... and this is on an 8 core 2.8GHz MacPro talking exclusively to internal SATA II hard drives.

After discussing this with Dave, there's nothing different in what SD 2.6.1 is doing, so hopefully there will be some "a ha" moment and a 2.6.2 will appear that returns smart update's efficiency.

-- Marc

SteveL
08-31-2009, 06:40 PM
I literally ran SmartUpdate with SD! 2.6.1, rebooted, and ran SmartUpdate again. Amazingly, over 2 GB of my 35 GB drive were updated. Hmmm

dnanian
08-31-2009, 09:10 PM
I think this MIGHT be a side effect of prebinding. We're looking into it.

whaler
09-01-2009, 10:08 AM
I'm another who is having the problem with very L-O-N-G backups.
What used to take 3 - 4 minutes is now taking 20-22.
I thought that if I started all over again and erased the drive and did a complete new backup, it might help.
It didn't.
I sure hope that a fix will be found!

dnanian
09-01-2009, 11:00 AM
I believe this is a side effect of prebinding, as I said above. Investigation has shown that most of the time is spent in /Applications /System and /Library, which would help to confirm that. We're looking into it.

MacCetera
09-01-2009, 05:53 PM
I believe this is a side effect of prebinding, as I said above. Investigation has shown that most of the time is spent in /Applications /System and /Library, which would help to confirm that. We're looking into it.

Dave, after playing with the unbelievably slow backups on my MacPro, I don't get why the prebinding is an issue for overall speed, as that is in the final phase and not performed on non-boot drives.

Maybe this will be helpful:

Today my now 9+ hour Time Machine drive clone was still percolating at 10 this morning, so I popped open Activity Monitor and saw that SDCopy was eating 87-100% of a single core with 2 threads. I'm running SD 2.6.1 right now to clone my MacBook Pro drive under 10.5.8 before upgrading it to 10.6 later today, and SDCopy is only eating at max 15% of one core with an average of around 1-3%. That puppy really beats itself up under 10.6.

-- Marc

dnanian
09-01-2009, 08:13 PM
That's not at all normal. For one thing, we're not threaded. For another, SDCopy is really not significantly different under SL. I think something else is going on there (assuming that OSX itself isn't eating the core when we call it).

That doesn't have much to do with the other issue here, which is that Smart Update is copying things over, which I believe does have to do with prebinding.

nkhester
09-01-2009, 08:26 PM
With reference to the prebinding possibility... I noted that when I was making my final SD backups before updating to 10.6 (today), that an inordinate amount of time was spent in the final step of adjusting prebinding on the backup. Normally this step would have required virtually no time. This was true both for full backup and sandbox.

Don't know if this provides any hint or not...

robbchadwick
09-01-2009, 08:46 PM
The huge amount of extra time seems to come at the beginning. Once SD gets beyond a certain point, it zips along like it used to. I do agree that the pre-binding at the end seems to run a little longer but for me that's not the lion's share of it.

Just to update, what I used to do in about 5 minutes is now taking 20-30 minutes. I know you are working on it; but I thought I'd let you know that time is not clearing up the issue ... although I suppose there really hasn't been a huge amount of time since the weekend. :-)

dnanian
09-01-2009, 10:53 PM
Well... adjusting prebinding is new to 2.6, so I don't think it would have taken no time... but it can take longer under SL. We're trying to find a different way.

And I think people are onto something with the whole 'compressed files are expanding' thing, even though it shouldn't be. We are looking into it. No sense chiming in with a "me too" if you're thinking about it, unless you have more to add: we're trying like hell to get to the bottom of what's up.

nkhester
09-02-2009, 08:23 AM
Well... adjusting prebinding is new to 2.6, so I don't think it would have taken no time... but it can take longer under SL. We're trying to find a different way.

I had failed to notice this added step. So, yes, all I had noted with earlier SD backups is that the last step was accomplished very quickly.

I trust we'll all wait patiently for this to be resolved.

dnanian
09-02-2009, 09:52 AM
Yes, it's important to note (again) that if it's taking a little extra time in some situations in this version, and if the backup takes up more space at a low level, the copy is still reflective of the source and is valid... continued improvements will be made as we determine the cause.

Usually, this kind of thing is investigated in testing, and although we've been in test for a long time, the 'seems a bit slower/added copying' issue came up relatively late, in the "gold bits" timeframe, and then Snow Leopard shipped about 4 weeks earlier than we expected. It was unexpected, and we didn't consider it wise to make risky changes late-cycle, since the copy itself was OK, and there just wasn't enough time to get into the low-level details.

We could have delayed weeks to try to figure it out... but considering how people reacted last time we needed to delay, and since it was a performance issue and not a copy problem, we decided to go with what we had.

Now that we have a bit more time to investigate more completely, we're working on it, and if the fix is riskier, we have the luxury -- necessity, really -- of being able to test it without the same deadline breathing down our neck.

MacCetera
09-03-2009, 01:33 PM
No sense chiming in with a "me too" if you're thinking about it, unless you have more to add...

Adding something :)

Two identical internal MacPro Seagate 1TB SATA II hard drives, with identical GUID single volume partitioning, used as the Time Machine repository and a clone of same. Just completed (yet another nightly 800+ GB smart update):

Source: capacity 999.86 GB, available 184.17 GB, used 815.69 GB
Target: capacity 999.86 GB, available 178.64 GB, used 821.22 GB

Time Machine is turned off during the clone. Not bootable, so no post prebinding operation performed.

-- Marc

dnanian
09-03-2009, 01:49 PM
Perhaps a difference in the spotlight indexes in the two drives, Marc (one's inside the TM archive, too)? Could also be that the compression was preserved in the source but not when copied.

nkhester
09-03-2009, 03:28 PM
It may not be just SD that is bloating its output with OS X 10.6. Over the past two days I've seen similar inexplicable bloating of Time Machine files.

http://discussions.apple.com/thread.jspa?threadID=2142173&tstart=0

MacCetera
09-03-2009, 04:51 PM
Perhaps a difference in the spotlight indexes in the two drives, Marc (one's inside the TM archive, too)? Could also be that the compression was preserved in the source but not when copied.

I have always had Spotlight indexing disabled on the clones. I just checked the exclusion list again, and the TM copy is still listed as excluded. Then I did a du on both the source and target Spotlight dirs:

root@jupiter
/Volumes/Amalthea $du /Volumes/Amalthea/.Spotlight-V100/
175648 /Volumes/Amalthea/.Spotlight-V100//Store-V1/Stores/BA8038CE-F979-4651-912B-3A0FEF836D6C
175648 /Volumes/Amalthea/.Spotlight-V100//Store-V1/Stores
175664 /Volumes/Amalthea/.Spotlight-V100//Store-V1
175664 /Volumes/Amalthea/.Spotlight-V100/

root@jupiter
/Volumes/Amalthea $du /Volumes/Himalia/.Spotlight-V100/
4102416 /Volumes/Himalia/.Spotlight-V100//Store-V1/Stores/38C02202-C9B6-4154-AAC9-B1A851A00BBE
4102416 /Volumes/Himalia/.Spotlight-V100//Store-V1/Stores
4102432 /Volumes/Himalia/.Spotlight-V100//Store-V1
4102432 /Volumes/Himalia/.Spotlight-V100/


So this does explain part of the 5.42 GB difference, but it's not the whole story.

So why would a copy result in expansion of compressed data?

-- Marc

MacCetera
09-03-2009, 04:57 PM
It may not be just SD that is bloating its output with OS X 10.6. Over the past two days I've seen similar inexplicable bloating of Time Machine files.

http://discussions.apple.com/thread.jspa?threadID=2142173&tstart=0

I've been using BackupLoupe to show the detailed differences in TM snapshots:

http://soma-zone.com/BackupLoupe/

-- Marc

dnanian
09-03-2009, 05:47 PM
I'm sorry -- I can't take the time to go into detail on this here, since we're still investigating. But I'm going to write a blog post about it when I get a few minutes of free time to discuss the issue and what we did (or are in process of doing) to fix it.

As I recall (and mentioned above), there's a spotlight index inside the TM backups.backupdb too.

MacCetera
09-03-2009, 06:45 PM
As I recall (and mentioned above), there's a spotlight index inside the TM backups.backupdb too.

There's only one Spotlight location on Time Machine volumes - and none were found within the Backups.backupdb directory...

root@jupiter
/Volumes/Amalthea $find /Volumes/Himalia -name .Spotlight-V100
/Volumes/Himalia/.Spotlight-V100

A more general search yields only spotlight-named things that should be in the backup...

root@jupiter
/Volumes/Amalthea $find /Volumes/Himalia -name *Spotlight*
/Volumes/Himalia/.Spotlight-V100
/Volumes/Himalia/Backups.backupdb/Jupiter/2008-09-22-001142/Europa/Library/Application Support/iDVD/Themes/iDVD 7/CenterStage-Chapters.theme/Contents/Resources/CenterStage-Main-Spotlights.qtz
/Volumes/Himalia/Backups.backupdb/Jupiter/2008-09-22-001142/Europa/Library/Application Support/iDVD/Themes/iDVD 7/CenterStage-Extras.theme/Contents/Resources/CenterStage-Main-Spotlights.qtz
/Volumes/Himalia/Backups.backupdb/Jupiter/2008-09-22-001142/Europa/Library/Application Support/iDVD/Themes/iDVD 7/CenterStage-Main.theme/Contents/Resources/CenterStage-Main-Spotlights.qtz
/Volumes/Himalia/Backups.backupdb/Jupiter/2008-09-22-001142/Europa/Library/Spotlight
/Volumes/Himalia/Backups.backupdb/Jupiter/2008-09-22-001142/Europa/Library/Spotlight/GBSpotlightImporter.mdimporter
/Volumes/Himalia/Backups.backupdb/Jupiter/2008-09-22-001142/Europa/Library/Spotlight/GBSpotlightImporter.mdimporter/Contents/MacOS/GBSpotlightImporter
... Backups.backupdb entries repeated for every TM session ...

Perhaps you're confusing this with a .Spotlight-V100 directory within each TM sparse bundle for networked backups. But there is ONLY one.

By the way - lest you think that I'm being critical or pedantic about this - I'm just VERY HAPPY that SuperDuper handles cloning TM disks at all... NOTHING else out there can even touch them except a block based cloner... and for that support I am EXTREMELY thankful...

-- Marc

dnanian
09-03-2009, 06:51 PM
Quite possibly... I don't remember the details exactly, since I did this research almost two years ago... :)

nkhester
09-04-2009, 11:14 AM
It may not be just SD that is bloating its output with OS X 10.6. Over the past two days I've seen similar inexplicable bloating of Time Machine files.

http://discussions.apple.com/thread.jspa?threadID=2142173&tstart=0

The explanation (at least, partial) for the bloating of TM is that I found that WD Drive Management is incompatible with OS X 10.6. It was adding a spurious message to Console every 10 seconds whether drives were connected or not. It appears that each entry is unique and is seen by TM as a "change"! This caused TM to bloat by about 10GB daily.

Needless to say I've disabled the WD program without any "visible" effect other than TM is no longer bloating hourly.

jsmatney
10-13-2009, 11:54 PM
so - did 2.6.2 resolve the "binding" issue?



side note:
i tried to upgrade from 2.6 to 2.6.2 under auto update this evening, and after 5 minutes of watching the download spin, i aborted. i am moving to o.s. 10.6 shortly so i guess i need 2.6.2

but the long download time was troubling. was this just a one time quirk?

note: i tried download on tue eve at 10 pm est.

i have updated superduper many times in past - past as lightning.

dnanian
10-14-2009, 08:24 AM
2.6.2 resolves the re-copying issue completely, yes.

Downloads might have been a bit slow last night because a lot of people hit the site at around 9pm.