Shirt Pocket Discussions - Time for 2.4 TB first copy?

Shirt Pocket Discussions (https://www.shirt-pocket.com/forums/index.php)

- General (https://www.shirt-pocket.com/forums/forumdisplay.php?f=6)

- - Time for 2.4 TB first copy? (https://www.shirt-pocket.com/forums/showthread.php?t=5420)

Time for 2.4 TB first copy?

Hi all,

We are doing a fresh copy using Smart Update to a disk image from one Mac Pro RAID to another Mac Pro RAID over a 1000BT network with a 3Com Gigabit switch. (Both machines use RocketRAIDs.) The source volume has 2.4 TB of data.

The disk image on the backup has been 2.02 TB for about 14 hours now, total elapsed time for the copy has been 33 hours.

The effective copy speed shown this morning is 0.16 MB/s. If my math is right and the copy speed is accurate, it will take another 25 days to finish the copy.

I know SD shows negative copy speeds erroneously, but the Effective Copy Speed displayed seems to match the increase in the disk image size.

Is there a benefit to stopping the copy and re-starting it? Our experience is that stopping the copy midway leaves the disk image unmountable and unusable - is there a safe way to stop and re-start?

I'd be interested to hear if anyone else has copied similar amounts of data and the copy times for it.

Thanks to all!

You're copying 2.4TB of data over a network, rather than locally? Yeah, that's going to take a while.

Stopping the copy should not leave an image unmountable/unusable, unless the image is already damaged. The fact that it stopped at 2TB makes me think that the drive has a 2TB file size limit (images can be petabyte sized, so I don't think it's the image itself).

If you look in the log (Cmd+L) are you getting errors at the bottom with retries?

Note that I've just re-checked HFS+, and it also has a really huge maximum file size, so I don't think that's it either... unless the RocketRAID has some weird limits, or the network FS does.

If you look at the Get Info for the image volume, does it have the maximum size you'd expect?

A crossover isn't going to help much, since you're still indirecting I/O through the network and image layers. The fact that you're not getting any errors is a good thing... as is the growth of the image.

(By image volume I meant the name of the image 'disk' in finder - white icon, named "RAID-Daily".)

Sounds like it's chugging along slowly... not sure what to suggest other than "this might be too much data to copy over a network with SD!, given the speeds you're getting", but it doesn't sound like you have an alternative.

The image volume looks right, the same capacity and available as the RAID, and the used amount matches the size of the sparse image. Seems right to me.

A couple of questions:

Is there any potential benefit to stopping the copy and re-starting it? (Restarting both machines, clearing the caches, etc.)

Can we use and make changes to the RAID while it's in the copy process? I'm afraid that the changes made during the copy will somehow make the verification fail.

I suppose if I had to do this again, it would be faster to install the two RAID cards in the same machine and connect both the RAID and the backup and copy that way.

Thanks again for your help.

No, I don't think stopping/restarting will help. It certainly won't make any difference to SD itself. The system, it's hard to say, but conceptually it shouldn't.

If you make changes to files that have already been copied, that should be fine.

Yeah, copying locally would be a lot faster...

Maybe this helped . . .

Dave,

So we stopped the copy, rebooted both machines, cleared their caches, rebooted again, and restarted the 3Com switch.

The disk image mounted easily, and we re-started the copy. It's currently copying around 37 MB/s, although it hits faster speeds from time to time. (See image below.)

I suppose it could always hang up/slow down again, but at least the math doesn't indicate it's going to take three weeks.

Well, that's because you're using Smart Update, so it's skipping a lot of files to start with... it's going to slow down when you get to the point where data's actually being copied.

You're right . . .

You're right, something not working correctly.

I can do a straight Finder copy through the network from one RAID to the other that averages around 60 MB/s or higher. I tried a big file to see if it bogged down after a while, with thousands of images, and it still copied quickly.

The SD! copy is down to below 3 MB/s and falling as it progresses. IF it stayed stable at that rate, at least we could wait it out, but since the rate falls over time, it's like constantly going halfway to our destination.

Since it's a new RAID on the other end, is there some kind of formatting issue that could be keeping the copy from happening?

Did you try copying into the image, though? They're quite different operations.

Right again, Dave, it completely slows to a crawl.

Apparently we crept up in TB size incrementally and never noticed that the backups where taking so long. After a job we'd dump 200 GB at the most on the production RAID, and it would copy overnight, every night to the backup RAID.

Since my 7 GB test copy will now take hours and hours to copy over, what's the best strategy to create a backup? Would it be faster to copy the info over to just another RAID the same size, not to a disk image?

We can't take over three weeks to make the first copy, and I'm worried that this speed is so much slower than before.

Any ideas?

That probably wasn't the clearest post you'll read today.

The backup RAID is new, it only has a couple of sparse disk images from a few computers on the network, small stuff.

The backup RAID could be reformatted into two volumes - one that matches the size of the production RAID, and one that can house the variety of small disk images from the other machines.

If we did structure the backup RAID that way, and set SD! to make the backup the same as the production, would that be a faster method than the sparse disk image setup we currently have?

If you can attach both to the same Mac, you'll be able to go direct-to-direct, which will make things significantly faster. That's probably what I'd do.

Thanks Dave- just to be clear, if we did connect both RAIDs to one machine, should we copy to a disk image or to a volume sized the same as the source?