NetBSD - Live Network Backup

dvl writes "It is possible but inconvenient to manually clone a hard disk drive remotely, using dd and netcat. der Mouse, a Montreal-based NetBSD developer, has developed tools that allow for automated, remote partition-level cloning to occur automatically on an opportunistic basis. A high-level description of the system has been posted at KernelTrap. This facility can be used to maintain complete duplicates of remote client laptop drives to a server system. This network mirroring facility will be presented at BSDCAN 2005 in Ottawa, ON on May 13-15."

Mac OS X

I'm not up on my xBSD's, so can someone explain how hard this would be to port to the Mac? This would be perfect for cloning my son's Mac Mini.

use rsync

It's much less network and hardware intensitive and with the right parameters, will keep past revisions of every changed file. Your hard disks will live longer.

Re:use rsync

This is a block level operation, whereas rsync is file level. With this system you can restore the disk image including partitions. Restoring from rsync would require you to create the partition, format the partition and the restore the files. Also, if you need the MBR...

As the article says, this is drive imaging whereas rsync is file copying.

Pros and Cons

This would be an extremely sensitive server system. With everyones harddrive image just waiting to be blasted to a blank harddrive, the potential for misdeeds is staggering. Even in an offical capacity, I really feel uneasy if my boss was able to take a copy of my harddrive image and see what I've been working on. Admittely, yes it should all be work but here we are allowed a certain amount of freedom with our laptops and I wouldn't want to have that data at my bosses fingertips.

On the flipside, this would be a boon to company network admins especially with employees at remote sites who have a hard crash.

Another reason to build a high speed backbone. Getting my 80GB harddrive image from Seattle, while I'm in Norfolk would be a lot of downtime.

Perfect for those moments...

...when you get that idiot (and EVERY company has at least 1 of these guys) who calls you up asking if it's OK to defrag their hard-drive after downloading a virus or installing spyware. Then, when you tell them "NO", they just tell you that they did it anyways.

Now we can just hit a button and restore everything, a few thousand miles away.

The only thing left is to write code to block stupid people from reproducing.

Re:Perfect for those moments...

The only thing left is to write code to block stupid people from reproducing.

Unfortunately the user interface for the relevant hardware has a very intuitive point and shoot interface.

How long before this becomes a hack?

Assuming you can get around bandwidth monitoring, how long before this becomes incorporated into hacking tools. Add this to a little spyware and a zombie network and things get very interesting for poorly secured networks & computers.

Done this for years

I've been using der Mouse to copy files for years. First I user der Mouse to click on the file, then I use der Mouse to drag it to a new location!

Dump?

Doesn't NetBSD support dump -L the way FreeBSD does? This strikes me as a much more powerful and general solution than this custom tool...

Maybe setup is inconvenient.

Maybe setup is inconvenient. Remote backups using dd and ssh (our method) was a bit of a bear to initially setup, but thanks to shell scripting and cron and key agents, it hasn't given us any problems. I've seen a few guides with pretty straightforward and mostly universal instructions for this type of thing. That being said, I do hope this software will at least get people to start looking seriously at this type of backup since it lets you store a copy off-site.

How does this handle active filesystems?

If one tries to clone an FS that is active, can this cloning tool handle open/changin files (often the most important/recent-in-use files on the system)? I remember an odd bug in an Mac OS X cloning tool that would create massive/expanding copies of large files that were mid-download during a cloning.

Automatic Backup for Paranoids?

Isn't there an automated network disk backup tool for paranoids like me?

Well, I'm not really paranoid, but I had some cases where faulty file system drivers or bad RAM modules changed the content of some of my files and where I have then overwritten my backup with these bad files.

Isn't there any automatic backup solution that avoids such a thing? What I have in mind: there should be several autonomous instances of backup servers (which may actually reside on desktop PCs linked via LAN) that control each other on a regular basis. They should also keep back old versions of files as far as disk space allows.

Then, there should be a KDE tray applet showing me the state of the backup server network. It would indicate if servers haven't been cross-checked for some time or if CRC errors or general malfunction problems have occurred.

Wouldn't that be nice? Never ever care again for your backups. It's all done in the background and in a total paranoid manner.

Meh. You can use DRBD on Linux anyway.

Wel, not a solution for BSD people (unless you're running a bsd under Xen and the toplevel linux kernel is doing the DRBD).

Right solution, wrong problem

While this is cool, as I thought when I saw it on KernelTrap, disk mirroring is useful in situations where the hardware is less reliable than the transaction. If you have e.g., an application-level way to back out of a write (an "undo" feature), then disk mirroring is your huckleberry.

Most (all) of my quick restore needs result from users deleting or overwriting files - the hardware is more reliable than the transaction. I do have on-disk backups of the most important stuff, but sometimes they surprise me.

I'd like a system library that would modify the rename(2), truncate(2), unlink(2), and write(2) calls to move the deleted stuff to some private directory (/.Trash, /.Recycler, whatever). Obviously the underlying routine would have to do its own garhage collection, deleting trash files by some FIFO or largest-older-first algorithm.

Just a thought.

Re:Right solution, wrong problem

I'd like a system library that would modify the rename(2), truncate(2), unlink(2), and write(2) calls to move the deleted stuff to some private directory (/.Trash, /.Recycler, whatever). Obviously the underlying routine would have to do its own garhage collection, deleting trash files by some FIFO or largest-older-first algorithm.

Done. [netcabo.pt]

nothing new

Novell Zenworks has had this capability for sometime in production environments. It also integrates with their management tools so it is easy to use on an entire network. To say this technology is newly discovered is a far cry from the truth. They also use Linux on the back end of the client to move the data to the server.

It is nice though to have something like this in the open source world though. Competition is good.

How Soon

How soon do you think this will this be available in the Major Linux distros? I would love to have this for my debian machine. Perhaps I wouldn't have had to spend all last Saturday rebuilding my machine and restoring individual files.

SIGS!!!We don't need no stinkin sigs

Wacky idea

Maybe I should patent this. Ah well, I figure if I mention it now it should prevent someone else from doing so...

I was thinking - I know how Ghost supports multicasting and such. I was thinking about how to take that to the next level. Something like Ghost meets BitTorrent.

Wouldn't it be great to be able to image a drive, use multicast to get the data to as many machines as possible, but then use BitTorrent to get pieces to any machines that weren't able to listen to the multicast (ie it's on another subnet or something) and to pick up any pieces that were missed in the broadcast, or get the rest of the disk image if that particular machine joined in the session a little late and missed the first part?

I think that would really rock if someone wanted to image hundreds of machines quickly and reliably.

I'm thinking it'd be pretty cool to have that server set up, and find a way to cram the client onto a floppy or some sort of custom Knoppix. Find server, choose image, and now you're part of both the multicast AND the torrent. That should take care of error checking too, I guess.

Anybody care to take thus further and/or shoot down the idea? :)

How does this compare to md over a network block

I've used Linux for years to do this using md running RAID1 over a network block device. It works very well unless you have to do a resync. Is this better than that?

I'm asking because I'm backing-up about a dozen servers in real-time using this method, and if this method is more efficient, then I might be able to drop my bandwidth usage and save money.

dd over a LAN

I have done that 12 years ago on AIX with no problems as long as (a) the hd you dd it off from and to are sound and (b) there are no transmission failures beyond what rsh (at that time) would retry and mask.

ghost 4 unix

not the same?

http://www.feyrer.de/g4u/ [feyrer.de]

This is great

I just took one of our mailservers offline a minute ago to do a block-level copy, so this would be fantastic. I develop images for our machines, e.g., mailserver, etc, and then dd them onto other drives. When I update one machine, I then go around and update the others with the new image. This saves me tons of time, and we do a similar thing with desktops and Norton Ghost (although, if I'm not mistaken, this actually a file level copy).

And since we're running OpenBSD on those machines, porting this should be fairly straightforward... although now that I look at it, he adds some patches for sockets... eugh...

Scalability Forking?

How about disk cloning across servers, for on-demand scalability? As a single server reaches some operating limit, like monthly bandwidth quota, disk capacity, CPU load, etc, a watchdog process clones its disks to a fresh new server. The accumulating data partition may be omitted. A final script downs the old server's TCP/IP interface, and ups the new one with the old IP# (/etc/hostname has already been cloned over). It's like forking the whole server. A little more hacking could clone servers to handle load spikes (not just filling total capacity), running simultaneously under DNS load balancing scheme, like simple round-robin host/IP resolution. And cloning across a WAN could offer geographical distribution for disaster preemption. Is this stuff close to being a .deb package yet?

WTF

Why on earth are people always so insistent on doing raw-level dupes of disks?

First of all, it means backing up a 40GB with 2 GB of data may actually take 40GB of bandwidth.

Second of all, it means the disk geometries have to be compatible.

Then, I have to wonder if there will be any wackiness with things like journals if you're only restoring a data drive and the kernel versions are different...

I have been using ufsdump / ufsrestore on UNIX for ...decades!. It works great, and its trivial to pump over ssh:

# ssh user@machine ufsdump 0f - /dev/rdsk/c0t0d0s0 | (cd /newdisk && ufsrestore f -)

or


# ufsdump 0f - /dev/rdsk/c0t0d0s0 | ssh user@machine 'cd /newdisk && ufsrestore 0f -' .. it even supports incremental dumps (see: "dump level"), which is the main reason to use it over tar (tar can to incremental with find . -newer X | tar -cf filename -T -, but it won't handle deletes).

So -- WHY are you people so keen on bit-level dumps? Forensics? That doesn't seem to be what the folks above are commenting on.

Is it just that open source UNIX derivative and clones don't have dump/restore utilities?

The Dark Side of Image Backups

Image backups have great attraction. Restoring is done in one big whack, without having to deal with individual applications. Absolutely everything is backed up, so no worries about missing an individual file. etc. So why haven't image backups replaced all other forms of backup? The reason is the long list of drawbacks.

• All your eggs are in one basket. If a single bit of your backup is wrong, then the restore could be screwed -- perhaps in subtle ways that you won't notice until it's too late to undo the damage.
• Absolutely everything is backed up. If you've been root kitted, then that's backed up too. If you just destroyed a crucial file prior to the image backup, then that will be missing in the restore.
• You really need the partition to be "dead" (unmounted) while it's being backed up. Beware solutions that claim to do "hot" image backups! It is not possible, in the general case, for a backup utility to handle the problem of data consistency. E.g., your application stores some configuration information on disk that happens to require two disk writes. The "hot" image backup software happens to backup the state of the disk after the first write, but before the second. If you then do an install, the disk is corrupted as far as that application is concerned. How many of your applications are paranoid enough to survive arbitrary disk corruption gracefully?
• Size versus speed. Look at the curve of how fast disks are getting bigger. Then look at the curve of how fast disk transfer speeds are getting faster. As Jim Gray [microsoft.com] says, disks are starting to behave more like serial devices. If you've got a 200GB disk to image and you want to keep your backup window down to an hour, you're out of luck.
• Lack of versioning. Most disk image backups don't offer versioning, certainly not at the file level. Yet that is perhaps the most common need for a backup -- I just messed up this file and would like to get yesterday's version back, preferably in a few seconds by just pointing and clicking.
• Decreased testing. If you're using a versioned form of file backup, you probably get to test it on a fairly regular basis, as people restore accidental file deletions and the like. How often will you get to test your image backup this month? Then how much confidence can you have that the restore process will work when you really need it?

Image backups certainly have their place for people who can understand their limitations. However, a good, automatic, versioning file backup is almost certainly a higher priority for most computer users. And under some circumstances, they might also want to go with RAID for home computers [backupcritic.com].

the shared secret

The facility today supports symmetric cryptography, based on a shared secret. The secret is established out-of-band of the network mirror facility today. User identification, authentication and session encryption are all based on leveraging the pre-established shared secret.
----------- Confucious say: "The shared secret is no longer a secret."

Have this been invented?

And what about Ghost for You [feyrer.de]. this does netbackup with onlye one 1.44" disk.

rsync does a fine job for backups.

I run rsync on a backup server, and save the files without compression on removeable disks.

It makes it alot easier to find a file, cause it exists in the same location, uncompressed.

The huge advantage though, is that rsync only transfers those files that have changed. Which means that backups are very quick.

I also mount samba shares on the backup server, and do rsync backups of "My Documents" folders for the windows boxes. Works great there too!

Even better, the My Documents folders are available as (read only) Samba shares on the backup box, and the users can find their own files in the backups.

I have been doing this for years, and it works great!

Not scalable.

rsync is not scalable to large numbers of files. We set up a backuppc machine awhile ago, tried to rsync the entire backup set over to another machine... It was a miserable failure. Even if we didn't check for hardlinks, (which we have to, backuppc uses tons of hardlinks,) the rsync process completely saturated a gig of RAM before it even started syncing.

Now, rsync would have been fine if we'd unmounted the filesystem and done it on the raw partition. But there's a couple of problems with that:

It's not live. Not a big deal for us, since it's a backup machine to begin with, but still...

rsync doesn't do that. A couple of people have submitted patches to allow a flag for rsync to copy block devices as if they were files. They were tiny patches, but they were rejected out of a fear of users doing stupid things with them. I guess the usual Rsync Way is to duplicate the filesystem, so that devices are copied with mknod, not dd.

Re:BSD is 10 years too old

BSD is alot older than 10 years. It's probably 20+ years old.

Re:DOS of the backup server

RTFA: It responds to heavy load by making a log (journal?) of the blocks that need backing up, and then does them when the load is lesser. If you do it on swap, then you're insane and deserve whatever you get :)

This is a good idea, even if its niche is small, but I'm interested in how it handles the encryption. If it doesn't allow key re-generation on the fly, HMACs, certificates (or at least PSKs) and other things we expect from modern (SSH, IPSec/IKE, etc) systems then it's not going to be very useful. And unless I missed something it's going to be difficult to tunnel through a system that does do these things.

Personally I use SSH to tunnel everything possible, especially from Windows where IPSec is a joke, and the thought of sending all of my disk writes over a security system that is any less secure is a worry. Just imagine the problems if a man in the middle (or just a sniffer) catches plaintext: they know what you're doing, they know the contents of what you're doing, and highly likely they know what to do to exploit what you're doing. It's a very good thing that system entropy under nix is stored in the kernel, not on disk :)

Re:der Mouse?

Perhaps you could word that better?

It does seem wrong to allow for an anonymous developer as NetBSD has, Mike Parker sounds much better than der Mouse.

That they allow this is their choice however, that he is bitter about Theo de Raadt and his OpenBSD project does not warrent that kind of behaviour true, but it is up to NetBSD to choose what they view as proper behavoir within their developer circle.

They made that choice with Theo years ago, maybe with time they will choose to reign Mike in. Mike doesn't seem to be counted amoung the core developers, so it's not exactly the same as the situation with Theo however.

Perhaps this reflects poorly on the level of maturity of the developers on the project, but it is the choice of the developers.

And you cannot base your feelings about someone's programming based on their behaviour, or everyone would hate OpenBSD for Theo being a dick.