LS: Why is Smart Update so large?

[dnanian]

Yes, it's important to note (again) that if it's taking a little extra time in some situations in this version, and if the backup takes up more space at a low level, the copy is still reflective of the source and is valid... continued improvements will be made as we determine the cause.

Usually, this kind of thing is investigated in testing, and although we've been in test for a long time, the 'seems a bit slower/added copying' issue came up relatively late, in the "gold bits" timeframe, and then Snow Leopard shipped about 4 weeks earlier than we expected. It was unexpected, and we didn't consider it wise to make risky changes late-cycle, since the copy itself was OK, and there just wasn't enough time to get into the low-level details.

We could have delayed weeks to try to figure it out... but considering how people reacted last time we needed to delay, and since it was a performance issue and not a copy problem, we decided to go with what we had.

Now that we have a bit more time to investigate more completely, we're working on it, and if the fix is riskier, we have the luxury -- necessity, really -- of being able to test it without the same deadline breathing down our neck.

[MacCetera]

Originally Posted by dnanian

No sense chiming in with a "me too" if you're thinking about it, unless you have more to add...

Adding something

Two identical internal MacPro Seagate 1TB SATA II hard drives, with identical GUID single volume partitioning, used as the Time Machine repository and a clone of same. Just completed (yet another nightly 800+ GB smart update):

Source: capacity 999.86 GB, available 184.17 GB, used 815.69 GB
Target: capacity 999.86 GB, available 178.64 GB, used 821.22 GB

Time Machine is turned off during the clone. Not bootable, so no post prebinding operation performed.

-- Marc

[dnanian]

Perhaps a difference in the spotlight indexes in the two drives, Marc (one's inside the TM archive, too)? Could also be that the compression was preserved in the source but not when copied.

[nkhester]

It may not be just SD that is bloating its output with OS X 10.6. Over the past two days I've seen similar inexplicable bloating of Time Machine files.

http://discussions.apple.com/thread....42173&tstart=0

[MacCetera]

Originally Posted by dnanian

Perhaps a difference in the spotlight indexes in the two drives, Marc (one's inside the TM archive, too)? Could also be that the compression was preserved in the source but not when copied.

I have always had Spotlight indexing disabled on the clones. I just checked the exclusion list again, and the TM copy is still listed as excluded. Then I did a du on both the source and target Spotlight dirs:

root@jupiter
/Volumes/Amalthea $du /Volumes/Amalthea/.Spotlight-V100/
175648	/Volumes/Amalthea/.Spotlight-V100//Store-V1/Stores/BA8038CE-F979-4651-912B-3A0FEF836D6C
175648	/Volumes/Amalthea/.Spotlight-V100//Store-V1/Stores
175664	/Volumes/Amalthea/.Spotlight-V100//Store-V1
175664	/Volumes/Amalthea/.Spotlight-V100/

root@jupiter
/Volumes/Amalthea $du /Volumes/Himalia/.Spotlight-V100/
4102416	/Volumes/Himalia/.Spotlight-V100//Store-V1/Stores/38C02202-C9B6-4154-AAC9-B1A851A00BBE
4102416	/Volumes/Himalia/.Spotlight-V100//Store-V1/Stores
4102432	/Volumes/Himalia/.Spotlight-V100//Store-V1
4102432	/Volumes/Himalia/.Spotlight-V100/

So this does explain part of the 5.42 GB difference, but it's not the whole story.

So why would a copy result in expansion of compressed data?

-- Marc

[MacCetera]

Originally Posted by nkhester

It may not be just SD that is bloating its output with OS X 10.6. Over the past two days I've seen similar inexplicable bloating of Time Machine files.

http://discussions.apple.com/thread....42173&tstart=0

I've been using BackupLoupe to show the detailed differences in TM snapshots:

http://soma-zone.com/BackupLoupe/

-- Marc

[dnanian]

I'm sorry -- I can't take the time to go into detail on this here, since we're still investigating. But I'm going to write a blog post about it when I get a few minutes of free time to discuss the issue and what we did (or are in process of doing) to fix it.

As I recall (and mentioned above), there's a spotlight index inside the TM backups.backupdb too.

[MacCetera]

Originally Posted by dnanian

As I recall (and mentioned above), there's a spotlight index inside the TM backups.backupdb too.

There's only one Spotlight location on Time Machine volumes - and none were found within the Backups.backupdb directory...

root@jupiter
/Volumes/Amalthea $find /Volumes/Himalia -name .Spotlight-V100
/Volumes/Himalia/.Spotlight-V100

A more general search yields only spotlight-named things that should be in the backup...

root@jupiter
/Volumes/Amalthea $find /Volumes/Himalia -name *Spotlight*
/Volumes/Himalia/.Spotlight-V100
/Volumes/Himalia/Backups.backupdb/Jupiter/2008-09-22-001142/Europa/Library/Application Support/iDVD/Themes/iDVD 7/CenterStage-Chapters.theme/Contents/Resources/CenterStage-Main-Spotlights.qtz
/Volumes/Himalia/Backups.backupdb/Jupiter/2008-09-22-001142/Europa/Library/Application Support/iDVD/Themes/iDVD 7/CenterStage-Extras.theme/Contents/Resources/CenterStage-Main-Spotlights.qtz
/Volumes/Himalia/Backups.backupdb/Jupiter/2008-09-22-001142/Europa/Library/Application Support/iDVD/Themes/iDVD 7/CenterStage-Main.theme/Contents/Resources/CenterStage-Main-Spotlights.qtz
/Volumes/Himalia/Backups.backupdb/Jupiter/2008-09-22-001142/Europa/Library/Spotlight
/Volumes/Himalia/Backups.backupdb/Jupiter/2008-09-22-001142/Europa/Library/Spotlight/GBSpotlightImporter.mdimporter
/Volumes/Himalia/Backups.backupdb/Jupiter/2008-09-22-001142/Europa/Library/Spotlight/GBSpotlightImporter.mdimporter/Contents/MacOS/GBSpotlightImporter
... Backups.backupdb entries repeated for every TM session ...

Perhaps you're confusing this with a .Spotlight-V100 directory within each TM sparse bundle for networked backups. But there is ONLY one.

By the way - lest you think that I'm being critical or pedantic about this - I'm just VERY HAPPY that SuperDuper handles cloning TM disks at all... NOTHING else out there can even touch them except a block based cloner... and for that support I am EXTREMELY thankful...

-- Marc

[dnanian]

Quite possibly... I don't remember the details exactly, since I did this research almost two years ago...

[nkhester]

Originally Posted by nkhester

It may not be just SD that is bloating its output with OS X 10.6. Over the past two days I've seen similar inexplicable bloating of Time Machine files.

http://discussions.apple.com/thread....42173&tstart=0

The explanation (at least, partial) for the bloating of TM is that I found that WD Drive Management is incompatible with OS X 10.6. It was adding a spurious message to Console every 10 seconds whether drives were connected or not. It appears that each entry is unique and is seen by TM as a "change"! This caused TM to bloat by about 10GB daily.

Needless to say I've disabled the WD program without any "visible" effect other than TM is no longer bloating hourly.

[jsmatney]

so - did 2.6.2 resolve the "binding" issue?



side note:
i tried to upgrade from 2.6 to 2.6.2 under auto update this evening, and after 5 minutes of watching the download spin, i aborted. i am moving to o.s. 10.6 shortly so i guess i need 2.6.2

but the long download time was troubling. was this just a one time quirk?

note: i tried download on tue eve at 10 pm est.

i have updated superduper many times in past - past as lightning.

[dnanian]

2.6.2 resolves the re-copying issue completely, yes.

Downloads might have been a bit slow last night because a lot of people hit the site at around 9pm.