why averaging only 4MB/s and how to spare bad blocks?

[sjs]

hi
thanks for a nice program.

i am paranoid about bad blocks on my backup drives, so what i do, and i am not sure if this is the right way, is i use Disk Utility>Security Options>Write Zeros and devote 3-4 hours for that.

then when its done, i select that HFS+ drive as the target of Smart Update.

the drive is a HFS+ GUID boot drive, 178GB, 434,000 files, average speed so far is 4MB/s and it started 9 hours ago.

It should be 32MB/s / 2 = 16MB/s since both are on the same FW400 bus.

i did a Smart Update because i was afraid that Superduper's Erase then Copy might do a reformat (does it?) which might possibly delete the drive firmware's bad block sparing table (do you know if that'strue?)

Is it so slow because I chose Smart Update?

Do you know if the sparing table survives reformats and gets appended to, and do you know of any tools that let you examine the table? I'd love to look at the table of a drive I have that I know has bad blocks, then reformat it, if Disk Utility is so willing, then reexamine the table.

PS - it finished. Time Elapsed 9hrs 47min. 4.92MB/s. 434,532 files evaluated. 419,122 files copied.

Do you know why the source is 182,958,063,616 bytes and the target is 182,121,922,560 bytes? that's about 763MB. looking at it now i see you didn't copy over 512MB of swap. wheres the rest, logfiles and computer specific prefs?

thank you very much
steve

[dnanian]

Steve -- writing zeros can help, yes. There's no real protection against bad blocks other than a good backup, RAID mirroring, etc.

You can't make predictions about drive speed based on the protocol maximum speed. That's a streaming speed, and we're doing a lot more than just writing a single huge bytestream.

The other files that are missing are generally files in /tmp, some caches, etc. You can get a full list by examining the copy script you used.

[sjs]

Thank you. I see the plist exception files, that is good to know.

I know, good backups are essential and that is why i purchased your program and it was, in fact, one of my brand new 500GB backup HD's with the bad blocks. Bad blocks stink and they are often overlooked in the logs and overlooked as to reasons things are slowing down.

I have my main data set on 2 500GB HDs and this gets Smart Copied to TWO other pairs of 500GB HDs, rotated bi-monthly.

Its just that when moving this large a data set around takes full days when using FW400.

I did another smart update and despite what you said got more what I expected, 15MB/s, pretty much saturating the FW400 bandwidth with both drives on the same bus. But this smart update was the boot drive, full of smaller files. The other drive from before, the one where the smart update got only 2MB/s to a clean new drive, was full of large movie files.

Could the nature of these files be why? Otherwise, I will assume that that last 5-10x slower sync was due to bad blocks on my backup HD, which I just returned. I would like to understand what code you are using for a smart update. Is this ditto or rsync? And should a Smart Update to a new drive take the same time as a Format then Copy, programatically?

Is the code or pseudo code for these 2 different algoritm available anywhere? Thanks a lot.... Your program is the 1st I've used that pretty much always works. And I've tried them all.
Steve

[dnanian]

Steve: we don't use either ditto or rsync -- our cloner was custom designed and written for speed.

Certainly, when I copy large files I don't get 2MB/s -- I get much more, and slower performance when copying smaller files. I can't explain why your other was going so slowly, but yes, it could have been doing retries at a lower level.

Smart Update will take the same amount of time as an erase-then-copy if the drive is empty.

Our code/pseudo code is not available, sorry!

[sjs]

dave
i'm having similar problems again.
i started a Smart Update on a 300GB drive that was successfully Smart Updated 1 week ago.

in the meantime, not much changed, except perhaps 100 newly tagged mp3s and os x internals...

it started fine and said the apparent copy speed was 250MB/s.
great.
but 3/4 of the way through it slowed down.

now here's the weird part. when i say slowed down, it is almost *stopped*.

i mean, i look at both the external source and external destination, and neither light is on.

absolutely no data being xferred while these lights are off.

yet the CPU is near pegged on a PB G4 1Ghz.

then..every 5-10 seconds or so, the HD light might flash.

its like SD is caught in analysis paralysis mode...thinking about what to do next with 99% of the CPU resources and only once in a while doing an actual copy.

i note 20 minutes later that it is *not* frozen; # of files copied continues to grow; yet HD's remain mostly off.

this is what happened before.

do you know whats going on here?

also top shows a ditto process ... is that you? i thought you said you didn't use ditto but i didn't call it...

[dnanian]

We don't normally use ditto. But if we start getting failures, we retry with different APIs, and use ditto as a "last resort" to see if it'll work.

My guess is that things aren't working. Did you Cmd+L to look at the log? You can do that while it's running...