It’s been a little over two months since I accomplished this and wanted to share it as I hadn’t found it anywhere over the net. It took me a while but finally it’s here! Enjoy! 
To get everyone in context I’ll say that I had some Xen VMs which used LVM volumes as disk images living in one machine and I suddenly needed them on another one. After searching for some tried and tested solution I found none. So the only option I could see was to do some kind of dump (probably dd involved with bzip2) and bring them over to the other machine. Be it by network, DVD or external disk. Haven’t done it before, and it sure would have taken some effort on my part. Downsides: dangerous precise movements required and nothing new learned. This was a no-op 
Luckily an alternative came to my mind. I could use AoE [1] (ATA over Ethernet not Axis of Evil nor Age of Empires!!!) and some LVM wizardry. Upsides: dangerous precise movements required and cool stuff learned. This was the way to go! 
Just for the sake of clarity some definitions and requirements first. All the stuff below was done on two Debian Etch 4.0 boxes. Both of them having Xen for virtualization and LVM2 for storage administration. You’ll need an AoE enabled kernel (Debian’s kernels are from 2.6.11+). You’ll also need the aoetools package in the destination machine at least. Let’s call ORIG and DEST the machines and VG = Volume Group, PV = Physical Volume and lv = Logical Volume. If that’s mumbojumbo to you then keep away from the next commands!
Now to the recipe:
Enable AoE server on ORIG
root@orig-host# vbladed 0 0 eth1 /dev/md1
That should export any block device you want over eth. In my case a whole RAID device (md1) was exported through eth1 with major 0 and minor 0. Consider that Ethernet protocol is layer 2 and is non-routable. Another thing is that AoE provides no security inherently, so I enabled it manually for this and disabled it afterwards. If you are going to use it otherwise make sure you design a security scheme to go with it.
Enable AoE support on DEST
root@dest-host# modprobe aoe
Then search for AoE devices and make sure it found some
root@dest-host# aoe-discover
root@dest-host# aoe-stat
That should output something like “e0.0 250.000GB eth0 up”. At this point you should have a new block device available on the client box named /dev/etherd/e0.0. You can do anything to that block device as if it was a locally attached device. Cool!
Now we have to disable the ORIG-VG and make it unknown to the ORIG LVM system
root@orig-host# vgchange -an ORIG-VG
root@orig-host# vgexport ORIG-VG
Tip from experience: it’s better to have a dedicated VG for storing VMs images so we don’t have any downtime in the real box. I had the root filesystem on the same VG and had to switch to runlevel 1 just in case :???:
Search for new PVs on DEST and add them to LVM
root@dest-host# pvscan
Merge the two volume groups into one
root@dest-host# vgmerge DEST-VG ORIG-VG
Move LVs from ORIG-PV to DEST-PV. This is the step that’s gonna take forever or not depending on your network speed and the amount of data to move. I don’t have my exact figures but I recall it taking a couple of hours.
root@dest-host# pvmove -n some-lv /dev/etherd/e0.0 /dev/md1
Note this time /dev/md1 is the destination RAID device which belongs to DEST-VG, not the one on ORIG box stated before.
Once it finishes we want to make things go back to normal. So we disable the VG so we can do the split (James Brown style baby!) Then we re-enable the VG
root@dest-host# vgchange -an DEST-VG
root@dest-host# vgsplit DEST-VG ORIG-VG /dev/etherd/e0.0
root@dest-host# vgchange -ay DEST-VG
Now we have to disable the ORIG-VG and make it unknown to the DEST LVM system
root@dest-host# vgexport ORIG-VG
Back to the ORIG box we re-add the “lost” VG and re-activate the VG
root@orig-host# vgimport ORIG-VG
root@orig-host# vgchange -ay ORIG-VG
And that should be it! I’ll stop writing now since it’s a long post already. Hope somebody finds this useful and/or inspiring. Good luck!
[1] http://www.howtoforge.com/ata_over_ethernet_debian_etch
