Shirt Pocket Discussions

Shirt Pocket Discussions (https://www.shirt-pocket.com/forums/index.php)
-   General (https://www.shirt-pocket.com/forums/forumdisplay.php?f=6)
-   -   Backing up to a .DMG File on Server (https://www.shirt-pocket.com/forums/showthread.php?t=2770)

Marco_Polo 08-23-2007 07:42 AM

Backing up to a .DMG File on Server
 
I'm very frustrated over this.

How do I get SD to make automated backups onto an ordinary disk image file that's situated on a remote server?

Alternatively, how can I make SD wait nicely for the image file to be mounted when I use AppleScript? SD does not respect delays coded in the CopyJob.applescript file.

Details...

Due to issues with ever-growing .sparseimage files filling up my server, I decided to make regular backups to an ordinary .dmg file instead. The .dmg file is the same size as my hard drive and resides on a server volume.

SuperDuper won't mount the .dmg file for scheduled backups.

It won't do it from a scheduled backup when I select the .dmg file as the target in the main window and it won't do it from a scheduled backup when I select the mounted volume from the .dmg file as the target.

I thought that I'd just work around this problem using AppleScript to mount the server and disk image before the backup is run. I altered the CopyJob.applescript file. My first attempt looked like this...

Code:

on beforeRunningCopy()
tell application "Finder"
        mount volume "smb://192.168.n.n/BACKUP_VOLUME"
        open file "BACKUP_VOLUME:Backup.dmg"
end tell
end beforeRunningCopy

But SuperDuper's scheduled backup failed with this error:
The automatic copy aborted because SuperDuper! could not locate the Destination volume

It looked for the volume before it was mounted.

I figured that I could work around this problem with a delay to check for the existence of the mounted volume, so I revised my code...

Code:

on beforeRunningCopy()
tell application "Finder"
        mount volume "smb://192.168.n.n/BACKUP_VOLUME"
        open file "BACKUP_VOLUME:Backup.dmg"
set i to 0
repeat until (i = 1000)
    if not (exists "Boot HD:Volumes:Backup") then
          delay 15
          set i to 0
    else
          set i to 1000
          delay 30
    end if
end repeat
end tell
end beforeRunningCopy

SuperDuper's scheduled backup failed with the same error!

Watching it while it worked, I could see that SuperDuper completely ignored the delays in the AppleScript. It started the script going and then tried to find the volume right away without waiting for the script to complete.

How can I get SD to mount the disk image and WAIT for it to completely mount before starting the backup?

dnanian 08-23-2007 09:38 AM

It sounds to me like your code isn't running, or perhaps hasn't been compiled in: onBeforeRunningCopy is going to happen before SD! runs at all -- SD! hasn't even been launched yet.

Note, too: if your DMGs are the same size as the original volume, that's as large as the sparse image would ever grow (since the size of the original volume is the maximum size set). As such, you're not buying yourself any room at all...

Marco_Polo 08-23-2007 07:21 PM

Quote:

It sounds to me like your code isn't running
Yes it is. It definitely is. You made me just paranoid enough that I edited the script to add some beeps to it. It's running.

SD is not waiting for the script to finish executing before it attempts the backup.

I can buy a short delay if I set it to repair permissions on the source volume first, but that doesn't buy me enough time for the image to mount.


Quote:

Note, too: if your DMGs are the same size as the original volume, that's as large as the sparse image would ever grow
A .sparseimage backup of my 250GB startup drive containing roughly 180GB of files quickly swells to fill up my entire 320GB backup drive. The sparse image file just grows and grows uncontrollably.

See this earlier thread for more info on this problem.
http://www.shirt-pocket.com/forums/s...ead.php?t=2103

dnanian 08-23-2007 07:49 PM

I just don't understand how this can be, Marco_Polo. I'm looking at the code here, and it calls the onBeforeRunningCopy before it runs the backup (which loads the settings). The only way I can see it doing otherwise is if the "default" settings (the configuration it'll start up with normally) references the same disk.

As far as the image goes, if your source volume is 250GB, the most the sparse image can grow to is 250GB, since that's set as it's maximum size...

Marco_Polo 08-23-2007 08:33 PM

Quote:

I just don't understand how this can be, Marco_Polo.
Neither do I. That's why I'm asking. Running Mac OS X 10.4.9 and SD 2.1.4.


Quote:

As far as the image goes, if your source volume is 250GB, the most the sparse image can grow to is 250GB, since that's set as it's maximum size...
Runaway growth of sparse image files is a fact. To the best of my ability, I have verified that there is no fix. Unless you have a fix, I'd prefer not to discuss it further.

What I want to know at the moment is how to get SuperDuper to wait until my disk image has mounted before it attempts its scheduled backup.

I just tried the "run shell script before copy starts" option using a shell script with a sleep delay in it and again, SD ignored the delay.

Same bloody error.

If it'd just hold off for another 30 seconds or so, the disk image would finish mounting and it could perform the backup.

Marco_Polo 08-23-2007 08:57 PM

Okay, I've got it figured out.

Since the script is being executed by CopyJob instead of SuperDuper, delays only affect CopyJob. So I told SuperDuper to delay 5 minutes. Here's the code:
Code:

on beforeRunningCopy()
tell application "Finder"
        mount volume "smb://192.168.n.n/BACKUP_VOLUME"
        open file "BACKUP_VOLUME:Backup.dmg"
end tell

tell application "SuperDuper"
delay 300
end tell
end beforeRunningCopy

It'd be nice if SD would make automatic backups to regular .dmg files on a server without all this fooling around... why doesn't it?

dnanian 08-23-2007 09:59 PM

I've just tested the images again here, and runaway image growth is not reproducible here. It just will not occur. The image grows to the maximum size set and no further...

Marco_Polo 08-23-2007 10:35 PM

Okay, I've found my fix for the other problem, so let's talk about runaway sparse image file growth.

The specific equipment necessary to reproduce the issue as I have experienced it includes a Buffalo NAS, an XFS volume with smb and afp shares and SuperDuper. It doesn't matter whether the remote volume is mounted via afp or smb -- either way, the image files grow until the drive fills up.

The image files don't necessarily grow with every backup. I only observe the results when my backup drive fills up and SuperDuper logs a drive-full error. At that point, perhaps 2-5 times each month, I have a sparse image file that has swelled to be considerably larger than the source. I've gotten into a routine of deleting the over-large sparse image file and starting over when this happens, but it finally got annoying enough that I decided to try something else (the .dmg file solution).

I spoke to Apple engineers during a visit to Cupertino and was told that this is a known problem with sparse image files. Perhaps they misunderstood my question. It seems that you misunderstood me in the earlier thread.

dnanian 08-23-2007 10:42 PM

It's not at all unexpected for the image to grow larger then the total size of the files on the source. What's unusual (and, according to my tests, impossible) is to have the image grow larger than its set maximum size...

Marco_Polo 08-23-2007 11:58 PM

Quote:

Originally Posted by dnanian (Post 13661)
It's not at all unexpected for the image to grow larger then the total size of the files on the source. What's unusual (and, according to my tests, impossible) is to have the image grow larger than its set maximum size...

I used this command...

hdiutil imageinfo [/path_to/disk.img]

... to check the sparse image before I deleted it and started using the ordinary disk image for my backups.

It reported the correct amount of used and empty space as corresponded to the amount of used and empty space on the original drive and what was reported in the Finder when I mounted the sparse image, but the image file, itself, was still roughly 70GB larger than that.

I recall that there was a high "resize limit" of roughly 500GB on the image, but assumed that it only came into play if I used hdiutil to resize it from the command line.

Was I wrong in that assumption? Does that number represent the largest that the image can grow without user intervention? If so, then SD or the OS was setting the wrong number when it created the sparse image.

dnanian 08-24-2007 02:01 PM

I've just created another image to test this, and the max size is set properly from what I can tell. You can test this by creating a small source image (e.g. 1GB). Use that as the source "drive" and create a disk image with SD.

Then, check it, and try to copy more than 1GB of data into the image...

Marco_Polo 08-24-2007 08:15 PM

I had a 580MB disk image handy, so I used that as the source and used SuperDuper to make a backup sparse image on my server. The original disk image had 60MB of free space.

The resulting sparse image was 523MB.

I then copied 60MB of files to the original image -- filling it; ran a smart backup; deleted the new files from the original image; and ran a smart backup again...

I repeated this 3 times.

Results...

First run: The sparse image was 583MB
Second run: The sparse image was 583MB
Third run: The sparse image was 583MB

I decided to switch things up a bit and so I deleted a folder, renamed some of the files and swapped a few files between different folders.

SuperDuper reported this error during the next backaup:
| 07:10:27 PM | Error | ditto: /Volumes/Test/3/XXX001.tif: No space left on device

Checking the sparse image size again, it remains at 583MB. Looking at the contents of the disk image, I see that SuperDuper did not delete a folder that I had deleted from the original image. This seems to be an unavoidable issue due to the way that it sync's. It moves down through folder hierarchy alphabetically.

Although this disk image did not swell beyond the size of the original volume, I think that I have demonstrated the mechanism by which the other disk image grows.

Assume that the sparse image is somehow made with the wrong maximum size or that over time and with repeated mounting/unmounting the maximum size somehow gets changed/corrupted... the result of this kind of file-shuffling is that occasionally the amount of data copied to the image will exceed that on the source drive. That image file will grow beyond the size of the source.

You insist that sparse image files aren't supposed to grow like that and I've been unable to reproduce it here, but it happened repeatedly when making nightly scheduled smart updates from my boot drive. I've deleted over-grown image files dozens of times and started over again with the same result.

I don't know what goes wrong with the sparse image files (maybe something records the wrong volume size from my hard drive?), but I have now demonstrated a mechanism by which those image files would grow if they could grow. And I know that they can grow because they do grow.

I'll tell you what I'll do... I've filled up the NAS drive with other stuff since I started backing up to a regular disk image. Give me a week or two and I'll clear off a few GB and start backing up to a sparse image file again. When it fills up the drive, I'll report back and I'll perform whatever reasonable tests you request.

dnanian 08-24-2007 09:12 PM

OK, Marco_Polo. I've really never seen what you're describing -- I've only seen images grow to their maximum size even though there's less data inside them. I'll be interested to hear your results. (It'd also be interesting to know if the image, before it showed this error, was created with expected sizes.)

Marco_Polo 11-17-2007 12:49 PM

Quote:

Originally Posted by dnanian (Post 13679)
OK, Marco_Polo. I've really never seen what you're describing -- I've only seen images grow to their maximum size even though there's less data inside them. I'll be interested to hear your results. (It'd also be interesting to know if the image, before it showed this error, was created with expected sizes.)

Quick update:

For whatever reason, I could not duplicate the issue again.

I swear, the disk images were filling up my backup drive over and over again, each image growing considerably larger than the source-drive. I'd trash an image and start over and then a few weeks later it'd swell to fill up my NAS drive again.

But the problem went away. I've been backing up to a .dmg file for awhile now and during that period I've done OS updates, firmware updates on the NAS and went from SD 2.1.3 to 2.1.4. Something kicked it.

Instead of out of control disk image growth, I started getting disk-full errors -- which is what I'd ordinarily expect. And then I'd delete a few items from the disk image that I'd already deleted from the source and I could continue nightly scheduled backups until the next time the sparse image filled up.

Now, I've upgraded to Leopard and am using CCC, awaiting the SD compatibility update.

dnanian 11-17-2007 12:55 PM

Weird. Perhaps there was a weird bug in OSX that was allowing images to grow beyond their bounds? It's not something I've ever had reported, but I guess anything's possible...


All times are GMT -4. The time now is 09:46 PM.

Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2024, vBulletin Solutions, Inc.