Month: January 2017

After migrating my physical server to AWS recently I needed to shrink the boot volume a bit. The original server’s drive was ~1TB, so that’s the size my boot EBS volume was after the migration, but since I only have about 125GB of used space I wanted to reduce the overall volume size to about 150GB. Not surprisingly AWS doesn’t provide a native way to do this so I had to get creative. I found most of the steps on the AWS Developer Forum and have adapted them to my needs, along with adding a few. And just like with the physical server-to-cloud migration we’ll do here what many say can’t be done….

Step 1 – Create Snapshot of Volume
Using the AWS Console or AWS CLI create a snapshot of the volume you want to reduce, or an AMI of the instance. This will protect you in case something goes off the rails, making it quick and easy to recover.

aws ec2 create-snapshot --volume-id vol-1234567890abcdef0 --description "Snapshot of my root volume."

Step 2 – Shrink Volume
On the server in Disk Management, right-click the volume and select Shrink Volume. Select the desired size and let it run. Depending on a variety of factors this could take a while (several minutes to an hour or so) so be patient.

Step 3 – Stop Server and Detach Volume
When the volume shrink completes, stop (power off) the server. Preferably from within Windows, or use the AWS console or AWS CLI to do so. Then, detach the volume from the Windows server:

aws ec2 detach-volume --volume-id vol-1234567890abcdef0

Step 4 – Start Ubuntu EC2 Instance, Attach Volumes
Select the appropriate Ubuntu AMI (version and region) and launch it. Either through the web console or AWS CLI.

aws ec2 run instances --image-id-<AMI ID> --key-name <key> --count 1 --instance-type m1.large --security-group-ids sg-<SecGroup> --placement AvailabilityZone=<AZ>

Create a new EBS volume the size you want, making it at least the same size or larger than the shrunk volume size. Attach the volume you want to clone to the Ubuntu instance and choose a mount point. For this document we will use “sdo”. “o” for Original (Note “sdo” in the AWS interface gets remapped to “xvdo” in ubuntu). Attach the new volume you want to clone to to the Ubuntu instance and choose a mount point. We will use “sdn”. “n” for New. (Note “sdn” in the AWS interface gets remapped to “xvdn” in Ubuntu).

aws ec2 create-volume --size 150 --region <region%gt; --availability-zone <AZ> --volume-type gp2
aws ec2 attach-volume --volume-id vol-1234567890abcdef0 --instance-id i-01474ef662b89480 --device /dev/sdo
aws ec2 attach-volume --volume-id vol-1234567890abcdef1 --instance-id i-01474ef662b89480 --device /dev/sdn

Step 5 – Connect to Ubuntu, Run Commands
Connect to the Ubuntu instance and elevate yourself to SU with sudo su.

View and Save partition information for the “original” disk:

fdisk -l -u /dev/xvdo

Setup partitions for the “new” disk:

fdisk /dev/xvdn

At the prompt (Command (m for help):
“n” to create a new partition
“p” to make it a primary partition
Select a partition number (match the original setup)
Enter the first sector (match the original setup)
Enter the last sector (for first partition match the original, for a second, use the default last sector)
“t” to set the partition type to 7 (HPFS/NTFS/exFAT) on all partitions
Repeat the above process for all needed partitions
“a” to set the boot flag on partition 1
“p” to review the setup
“w” to write changes to disk and exit

Run the following to verify settings on both “old” and “new” drives:

fdisk -l -u /dev/xvdo
fdisk -l -u /dev/xvdn

Copy the MBR (Master Boot Record). The MBR is on the first sector of the disk, and is split into three parts: Boot Code (446 bytes), Partition Table (64 bytes), Boot Code Signature = 55aa (2 bytes). We only want the boot code, and will copy it with the the “dd” command to do a direct bit copy from disk to disk:

dd if=/dev/xvdo of=/dev/xvdn bs=446 count=1

Clone the NTFS file system one partition at a time (/dev/xvdo1, /dev/svdo2):

ntfsclone --overwrite /dev/xvdn1 /dev/xvdo1
ntfsclone --overwrite /dev/xvdn2 /dev/xvdo2

Step 6 – Detach from Ubuntu, Attach to Windows
Detach both volumes from Ubuntu instance and attach new volume to Windows instance as device /dev/sda1:

aws ec2 detach-volume --volume-id vol-1234567890abcdef0
aws ec2 detach-volume --volume-id vol-1234567890abcdef1
aws ec2 attach-volume --volume-id vol-1234567890abcdef1 --instance-id i-01474ef662b8948a --device /dev/sda1

Step 7 – Verify and Cleanup
Start the Windows instance:

aws ec2 start-instances --instance-ids i-1234567890abcdef0

Note: it may take several minutes to start the instance, so don’t be impatient or alarmed….. Once the instance starts, logon to the Windows instance, then run chkdsk to validate the drive and correct any errors:

chkdsk c: /f

Terminate Ubuntu instance:

aws ec2 terminate-instances --instance-ids i-1234567890abcdef0

Finally, as good measure make an AMI of your instance or Snapshot of the volume.

Doing What They Say Can’t be Done
I’ve had to do this task a few times but because they were separated by a significant amount of time both the methods changed slightly and my memory of the exact steps faded. These are the new-and-improved way to convert a bare metal Windows server (I’m doing 2008 R2, but it will work with 2012 R2, etc.) into an EC2 AMI. It took me several days and multiple attempts (mostly due to the time it took to copy the 120 GB image to AWS) and some trial and error, but if you follow these steps you should have success moving your server to the cloud.

Although various commercial tools exist to perform cloud migrations I used all free tools for this physical-to-cloud migration.

Review these prerequisites for VM Import to help ensure success. This AWS document is a good reference for other necessities and steps as well. According to AWS a physical server-to-AMI conversion cannot be done, but we’ll trick them a bit by converting to a Hyper-V VM from physical, then to an AMI, finally launching an EC2 instance.

Step 1 – Prepare Your Server
Prior to migration you should do a little house keeping to minimize potential issues and decrease the overall time the migration will take. Some suggestions are first, clean up your drive(s) by removing any unnecessary files and directories, this will make the virtual disk smaller and reduce time to copy files to AWS. Next, make sure at least one NIC has DHCP enabled (one of the things that will cause your import to fail). I also took the opportunity to make sure all apps and patches were up-to-date as well. I chose not to remove my server from the AD domain at this point – only after a successful import of the server into EC2.

Step 2 – Create VHD (Virtual Hard Disk) from Physical Server
This is simple with the free Sysinternals tool Disk2vhd. Download and run it. Select the volume(s) you want to turn into a VHD and the name of the destination VHD file (by default it uses the NetBIOS name of the server). Make sure to uncheck “use Vhdx” option as AWS will only allow you to import a VHD file and not a VHDX file. It is recommended that you save the VHD file to a drive other than one you are imaging, but since I didn’t have another drive at the time I wasn’t able to do that and the conversion worked fine. The server I am working on currently is named Alymere, so you’ll see that name throughout.

Disk2vhd – convert server to virtual hard disk.

Step 3 – Import VHD to Hyper-V
Use Hyper-V Manager to import the VHD exported in the previous step. I had another server (at another location) which I was able to install Hyper-V on to perform these steps, but I suppose you could do this on the origin server if it’s the only one you have. Maybe I’ll try it and update later….. Start your newly imported VM to make sure it boots and works as a VM, then shut it down. One critical step is to remove any CD or DVD drives from the VM as these too will cause the EC2/AMI import to fail.

Step 4 – Export VM to VHD
Again, using Hyper-V Manager export the VM. This will create a few directories and possibly additional files, but the only one you need is the new VHD file – in my case this is Alymere.vhd (although it’s the same name as the physical to virtual VHD file it is a different file).

Step 5 – Break up the VHD
If you’ve ever tried to copy a large file over the Internet you know it can be fraught with problems. So, for multiple reasons I used 7-zip to break the file into 100MB chunks. I did it as 7z format with a compression level of “normal.” Although it took about four hours for the compression I saved about eight times that much time when uploading to AWS. My ~120GB VHD file compressed to 41GB of zipped files.

Step 6 – Copy/Upload File(s) to EBS
Since I will have to extract my hundreds of 100MB files back to the original VHD I copied them to an EBS volume on one of my existing EC2 servers over a VPN connected to my VPC using robocopy. One of the reasons for breaking the original file into many smaller ones is that if there’s a problem with the copy (as is common over the Internet) I won’t lose much progress – yes this can be done by using the /z (restart) switch with robocopy – which I would highly recommend, but I’ve had better experiencing breaking large files into smaller ones. Another reason I did this was that the office where the server resides has terrible upload speeds, so I copied the files to an external drive and had my business partner take it to another office (I’m in a different state). It still took 2-3 days to copy the files from both locations to AWS, but was considerably faster doing it the way we did – copying zipped files from two locations to EC2 simultaneously.

Step 7 – Reassemble VHD
Once the files were all copied to my EBS volume on my EC2 server I used 7-zip to extract the files to the original VHD file. As mentioned previously this whole process (zip, copy, unzip) took several days, but using the methods described I feel it was the most efficient way possible given the circumstances. If you have low upload bandwidth or huge files it may make sense to use the AWS Import/Export service, which I’ve used with great success previously.

Step 8 – Copy/Upload to S3
In order to use AWS’s VM import/export the VHD file(s) have to reside in S3. Some tools (like my beloved CloudBerry) cannot copy files of this size (120 GB), so I used the AWS CLI. Namely, aws s3 cp:

aws s3 cp E:TempAlymereALYMERE.VHD s3://<bucket name>/Alymere/

The AWS CLI displays copy progress, including speed.

aws s3 cp – Progress and completion.

Step 9 – Import VHD to AMI
This step requires a little extra work. Follow these steps to create an AMI role and policy necessary to access S3, along with the necessary .json files tailored to your needs. With that done run the command (adapted for your use):

aws ec2 import-image --description "Alymere" --disk-containers file://C:AWSJSONcontainers.json

Depending on the size of the VM this will take from a few minutes to an hour or more. My 120GB VHD took about 90-120 minutes. Executing the export-image command will produce various output including a task ID, which can be used to check the progress of the import:

aws ec2 describe-import-image-tasks --import-task-ids import-ami-<task ID>

I ran this several times and could see it progressing.

Upon completion the message, “AWS Windows completed…” is displayed, along with the AMI ID.

Step 10 – Launch EC2 Instance
Finally, an EC2 instance can be launched from this newly created AMI.

aws ec2 run-instances --image-id ami-<task ID>

Step 11 – Post Launch
At this point I logged onto my newly created VM to verify all my applications and data were intact. Since they were I removed my “old” physical server from my domain and joined this one. Success!

Troubleshooting VM Import/Export
A variety of issues can cause problems, i.e. failure. I would suggest reading the prerequisites for VM Import and “Troubleshooting VM Import/Export” AWS pages before beginning to make sure you can avoid issues, then be able to troubleshoot if necessary.

Good luck and happy importing!