Friday, December 4, 2009

Creating an EBS-backed AMI from an S3-backed AMI

The recent introduction of Boot From EBS for EC2 opens up a lot of new possibilities.  But there are some bootstrapping issues to deal with.  There aren't many EBS-backed AMI's available yet and, given the rather complex process involved in porting them, it may take a while for them to show up.  This article will walk through the process of converting a popular S3-based AMI to an EBS-backed AMI.  I don't guarantee that this is the best process and I certainly wouldn't recommend that anyone use the resulting EBS-backed AMI for anything other than testing and further development, but it puts a stake in the ground regarding a potential process.  I'm sure I will hear about the shortcomings and possible improvements!

As a starting point, I'm going to use one of Eric Hammond's excellent Ubuntu AMI's.  In particular, I'm going to use:

ami-eef61587 alestic-64/ubuntu-9.04-jaunty-base-64-20091011.manifest.xml

This same basic process should work for other Linux-based AMI's.  The first thing I need to do is fire up an new instance of this AMI.  In addition, I'm going to create a new 10GB EBS volume that will serve as the prototype for my EBS-based AMI.  Here's how I do that using boto:

>>> import boto
>>> c = boto.connect_ec2()
>>> c.get_all_images(['ami-eef61587'])
[Image:ami-eef61587]
>>> img = _[0]
>>> img.run(key_name='cloudright', security_groups=['test1'], instance_type='m1.large')
Reservation:r-0369c96b
>>> inst = _.instances[0]
>>> inst.update()
u'pending'
>>> inst.update()
u'running'
>>> inst.placement
u'us-east-1b'
>>> v = c.create_volume(10, inst.placement)
>>> v.attach(inst.id, '/dev/sdh')
u'attaching'
>>> inst.public_dns_name
u'ec2-67-202-30-28.compute-1.amazonaws.com'

So, at this point I have a new EC2 instance up and running using the S3-based AMI and a new 10GB EBS volume attached to that instance.  Now, I need to login to that new instance and do a bit of work.

$ ssh -i ~/.ssh/cloudright root@ec2-67-202-30-28.compute-1.amazonaws.com
...
root@domU-12-31-39-02-31-51:~# apt-get update
...
root@domU-12-31-39-02-31-51:~# apt-get -y upgrade
...
root@domU-12-31-39-02-31-51:~# apt-get -y install cpipe
...
root@domU-12-31-39-02-31-51:~# mkfs -t ext3 /dev/sdh
mke2fs 1.41.4 (27-Jan-2009)
/dev/sdh is entire device, not just one partition!
Proceed anyway? (y,n) y
...
root@domU-12-31-39-02-31-51:~# mkdir /ebs
root@domU-12-31-39-02-31-51:~# mount -t ext3 /dev/sdh /ebs
root@domU-12-31-39-02-31-51:~# tar cpS / | cpipe -vt -b 1024 | gzip -c | tar zxpS -C /ebs
....
root@domU-12-31-39-02-31-51:~# umount /ebs

So, basically I have ssh'ed into the new instance, run apt-get update and apt-get upgrade to install all of the latest patches, formatted the EBS volume as an EXT3 filesystem, mounted that filesystem as /ebs and then copied the entire contents of the current root volume over to the EBS volume.  Then I unmount the EBS volume.  Now, let's go back to my Python session running on my local machine.

>>> v.detach()
u'detaching'
>>> v.create_snapshot('Initial snapshot for EBS-backed 64-bit Ubuntu 9.04 AMI.')
Snapshot:snap-023ca66b
>>> from boto.ec2.blockdevicemapping import EBSBlockDeviceType, BlockDeviceMapping
>>> ebs = EBSBlockDeviceType()
>>> ebs.snapshot_id = 'snap-023ca66b'
>>> block_map = BlockDeviceMapping()
>>> block_map['/dev/sda1'] = ebs
>>> c.register_image('MG-Ubuntu-9.04-EBS-20091204', 'Testing the creation of EBS-backed Ubuntu AMI.',
architecture='x86_64', kernel_id=img.kernel_id,
ramdisk_id=img.ramdisk_id,
root_device_name='/dev/sda1', block_device_map=block_map)
u'ami-f002e099'

So, here we are detaching the EBS volume and then creating a snapshot of that volume.  Then, we need to import some data structures that will allow us to register a new EBS-based image.  The first data structure is the EBSBlockDeviceType.  There are a number of available fields in that object but the only one we need to worry about is the snapshot_id.  This tells EC2 that when someone wants to start up a new instance of our AMI, EC2 needs to start by creating a new EBS volume from this snapshot.  The second data structure is the BlockDeviceMapping.  It is actually a subclass of a Python dictionary and behaves as you would expect.  We need to add an entry that maps the device name of our root volume (in this case /dev/sda1) to the EBS snapshot.  Finally, we call register_image to create the new AMI.  We pass in a name, a description, the architecture, the kernel and ramdisk (we are just referring to the same ones used by the original S3-backed AMI), the name of our root device (/dev/sda1) and the block device mapping we just created.  EC2 returns with a new AMI id and we can then use that to start new EC2 instances.  Just to verify, let's start up a new instance based on our EBS-backed AMI:

>>> c.run_instances('ami-f002e099', key_name='cloudright', security_groups=['test1'], instance_type='m1.large')
Reservation:r-f175d599
>>> inst2 = _.instances[0]
>>> inst2.update()
u'pending'
>>> inst2.update()
u'running'
>>> inst2.public_dns_name
u'ec2-75-101-218-5.compute-1.amazonaws.com'

Now let's SSH into our new EBS-based instance and make sure everything is okay.

jobs:~ mitch$ ssh -i ~/.ssh/cloudright.pem root@ec2-75-101-218-5.compute-1.amazonaws.com
...
root@domU-12-31-39-06-E1-62:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10321208    837672   8959248   9% /
tmpfs                  3935948         0   3935948   0% /lib/init/rw
varrun                 3935948        40   3935908   1% /var/run
varlock                3935948         0   3935948   0% /var/lock
udev                   3935948        72   3935876   1% /dev
tmpfs                  3935948         0   3935948   0% /dev/shm
root@domU-12-31-39-06-E1-62:~# 


I have made this AMI public and available in the us-east-1 region.  Feel free to fire it up and play around with it but be aware that none of the careful testing that accompanies Eric's or Canonical's AMI releases has happened here so it is for illustrative purposes only.

54 comments:

  1. Mitch , this blog post is wonderful. It would become a classic for me! Tks. Domingo Aguilera ( @batok at twitter )

    ReplyDelete
  2. Why are you using gzip here?

    # tar cpS / | cpipe -vt -b 1024 | gzip -c | tar zxpS -C /ebs

    ReplyDelete
  3. Ah, it's not necessary in this case. I was also playing around with copying the data across regions between two different EC2 instances, tunneling over SSH. That seemed too complicated for the initial version of the article so I went back to a single region but left the gzip command in there. I does work as is but it's not really necessary or that beneficial.

    ReplyDelete
  4. At least for me, the 'c.register_image' doesn't wrap, and I can't scroll to see all of it. I can only see up to "architecture='x86_64',". I was very confused how block_map ended up doing anything.

    But thanks for this blog, very useful.

    ReplyDelete
  5. Hmm. It's wrapping for me in Safari but not in FF. Drat. Let me futz with it and see if I can correct it. Definitely would be hard to understand without that command, it ties the whole thing together.

    ReplyDelete
  6. Is that better? If anyone has any good suggestions for incorporating source code into Blogger, please let me know. I find it very painful.

    ReplyDelete
  7. Thank you, Mitch!
    Very timely and usefull post!
    Dmitry

    ReplyDelete
  8. Thank you very much for the public EBS Ubuntu image. This saved me a lot of time. Brilliant work to get this upand running this fast.

    ReplyDelete
  9. Glad you found it useful but bear in mind that very little testing has been done on this. By all means, give it a try but I wouldn't move your production over there just yet 8^)

    Mitch

    ReplyDelete
  10. Thanks for the tutorial, Mitch. Amazon's docs are rather lacking at the moment.

    The only thing I did different was to use rsync rather than the tar pipes.

    rsync -avx --exclude /ebs / /ebs

    -David

    ReplyDelete
  11. on the >>> from boto.ec2.blockdevicemapping import EBSBlockDeviceType, BlockDeviceMapping

    I'm getting:

    Traceback (most recent call last):
    File " stdin ", line 1, in module
    ImportError: No module named blockdevicemapping

    excuse the missing end signs around stdin and module(blogger is being a bugger)

    Any ideas what I can do with this?

    I'm trying to use your method to create a 32-bit EBS backed AMI

    ReplyDelete
  12. You need to be running the very latest boto code, basically the HEAD in subversion. Specifically, r1397.

    ReplyDelete
  13. Thanks for this excellent post Mitch.

    I attempted to create an EBS backed 32 bit AMI from ami-caf615a3, using boto rev 1400 but hit a problem - errors in the tar extract:

    root@domU-12-31-39-03-44-48:~# tar cpS / | tar xpS -C /ebs
    tar: Removing leading `/' from member names
    tar: /dev/log: socket ignored
    tar: /proc/irq/479: file changed as we read it
    tar: /proc/irq/439: file changed as we read it
    tar: /proc/irq/319/smp_affinity: file changed as we read it
    tar: /proc/irq/319: file changed as we read it
    tar: /proc/irq/223: file changed as we read it
    tar: /proc/irq/145: file changed as we read it
    tar: /proc/irq/33: file changed as we read it
    tar: /proc/irq: file changed as we read it
    tar: /proc/sys/net/ipv4/route/flush: Cannot open: Permission denied
    tar: /proc/sys/net/ipv6/route/flush: Cannot open: Permission denied
    tar: /proc/kcore: Cannot open: Operation not permitted
    tar: /proc/filesystems: file changed as we read it
    tar: /proc/2/task/2/exe: Cannot readlink: No such file or directory
    tar: /proc/2/exe: Cannot readlink: No such file or directory
    tar: /proc/3/task/3/exe: Cannot readlink: No such file or directory
    tar: /proc/3/exe: Cannot readlink: No such file or directory
    . . . . .
    tar: /sys/devices/platform/pcspkr: file changed as we read it
    tar: /sys/devices/platform/uevent: Cannot open: Permission denied
    tar: /sys/devices/platform: file changed as we read it
    tar: Removing leading `/' from hard link targets
    tar: /var/run/cups/cups.sock: socket ignored
    tar: /var/run/gdm_socket: socket ignored
    tar: /var/run/dbus/system_bus_socket: socket ignored
    tar: /var/run/avahi-daemon/socket: socket ignored
    tar: Error exit delayed from previous errors
    root@domU-12-31-39-03-44-48:~#

    After this, register_image returned "No root snapshot specified in device mapping.".

    I wonder if you could suggest where I'm going wrong?

    Michael

    ReplyDelete
  14. mumrah also ran into problems with the tar command and, instead, used rsync which seems like a great alternative. You might want to give that a try. I'll try to dig a bit and find out why tar fails in some situations. The bottom line is you want to use some sort of file-based copy mechanism that creates an exact copy of the contents of the root volume as possible. My first attempts used dd for a block-level copy but that writes to all blocks in the EBS volume and makes the first snapshot much bigger and take longer than necessary.

    The error you are seeing from register_image is because you are not passing in the BlockDeviceMapping object correctly. Either you omitted it or got the name of the parameter wrong or it has the wrong values. Can you cut and paste your invocation into a comment?

    ReplyDelete
  15. I'm also trying to convert an existing 32-bit Alestic-derived AMI.
    What flags did you use for rsync? When I get there, I'm about to try:
    -aH --specials

    ReplyDelete
  16. The comment above from mumrah says "sync -avx --exclude /ebs / /ebs". I haven't tried the rsync method but it seems like a good approach.

    ReplyDelete
  17. Anyone tried creating an EBS-backed AMI using the XFS filesystem?

    ReplyDelete
  18. I haven't tried that although I am interested in doing that. That's an example of the "holy crap" kind of changes that BootFromEBS brings about. In the past, I always attached volumes, formatted as XFS, for MySQL and things like that. Now, it's tempting to have those on the root volume but it really means the root volume now needs to be XFS. That should be fine, but I just haven't done it before.

    ReplyDelete
  19. In the EC2 forum, Andries from AWS proposed this mega-command:

    yes | mkfs -t ext2 /dev/sdh && mkdir -p /mnt/tmp{1,2} && mount -t ext3 /dev/sdh /mnt/tmp1 && mount -t ext3 my-root-fs.img /mnt/tmp2 -o loop,ro && tar -cSf - -C /mnt/tmp2 | tar xvf - -C /mnt/tmp1 && umount /mnt/tmp2 && umount /mnt/tmp1 && rmdir /mnt/tmp{1,2}

    to handle the whole process.

    ReplyDelete
  20. Success on making a derivative image from 32-bit Ubuntu 9.10 Karmic ami-1515f67c release.

    For those of you checking out boto from source for the first time:

    % svn checkout http://boto.googlecode.com/svn/trunk/ boto
    % cd boto/trunk
    % sudo python setup.py install

    After that, the BlockDeviceMapping should work.

    ---

    When is the time to associate ephemeral storage? Does that have to be done as part of image registration?

    ReplyDelete
  21. That can be done via RegisterImage or it can be done via RunInstances but once the instance is launched, it can't be changed. Even if you stop the instance. ModifyInstanceAttribute does not currently support modifying the mapping of ephemeral drives although AWS says they plan to support it eventually.

    ReplyDelete
  22. Thanks for sharing how to do this. I wrapped the same basic process into 2 scripts that just use the command line api, http://bit.ly/4Vr3Vg

    ReplyDelete
  23. Pardon my ignorance, but I have a question. I am running PythonWin on Windows XP. I've downloaded the most recent version of BOTO from the Download section. How do I update to include r1397? Is there a special command in CMD PROMPT I can use? Sorry, a bit of a noob here, just trying to learn all this...

    P.S. Boto works fine launching AMIs and attaching EBS, just missing r1397...

    Please help.

    ReplyDelete
  24. This is a great article.

    You do say this might not ready for production yet. I'm going to try and run it in a production type environment with some prototypes of us that are used in the wild so I should get some results of it.

    I will keep you updated here and on my blog at http://shapingclouds.com.

    And again: thanks for this article!

    ReplyDelete
  25. Hi Scott -

    Until I finally get that 1.9 release out, the only way to get the latest code is to check it out of the subversion repository. There are good subversion clients for Windows, like TortoiseSVN. There is a wiki page (http://code.google.com/p/boto/wiki/RunningFromSubversion) that describes this but from a Linux POV rather than Windows.

    I do plan on getting a 1.9 release out there RSN. If it's not urgent, it might be easier to wait for that.

    ReplyDelete
  26. Mitch - that sounds fine. I'll need it for testing in a week or two...so I'll wait until then.

    Thanks for putting all of this together, this is really great and I appreciate all the help you provided to everyone asking questions!

    ReplyDelete
  27. Mitch,

    Thanks again for great post! It worked for me. The only problem was with tar command - it run forever (I saw it copy >50G and terminate it).

    But David's suggestion worked perfectly fine:
    rsync -avx --exclude /ebs / /ebs (thank you David!)

    Could you outline how do we change our AMIs ? For instance I have now "ubuntu 9.10 karmic" and I want to install couple new packages (and/or modify my ubuntu some other way) - do I need to make another snapshot and register new AMI each time I change single byte?

    I don't want manage running instances separately, instead I'd like to be able to kind of start "main" instance, do couple changes, so it affect AMI, and then start my 10-15 puppies from there :)

    thank you,
    Dmitry

    ReplyDelete
  28. I've tried doing this with a CentOS AMI but it doesn't boot, this is what the log shows:

    ...
    Loading mbcache.ko module
    Loading jbd.ko module
    Loading ext3.ko module
    Creating root device.
    Mounting root filesystem.
    VFS: Can't find ext3 filesystem on dev sda1.

    mount: error mounting /dev/root on /sysroot as ext3: Invalid argument
    Setting up other filesystems.
    Setting up new root fs
    setuproot: moving /dev failed: No such file or directory
    no fstab.sys, mounting internal defaults
    setuproot: error mounting /proc: No such file or directory
    setuproot: error mounting /sys: No such file or directory
    Switching to new root and running init.
    unmounting old /dev
    unmounting old /proc
    unmounting old /sys
    switchroot: mount failed: No such file or directory
    Booting has failed.

    ----

    Does anyone know how to fix this?

    ReplyDelete
  29. Yep - confirmed that the tar command recursively includes the target volume if used as indicated in the article. If you add the parameter --one-file-system then only the root "/" volume is used. For my test example, this worked fine since I did not want to include anything but /. The command I used was:

    tar cpS --one-file-system / | cpipe -vt -b 1024 | tar xpS -C /destination/volume

    Of course rsync is also a good way to go.

    ReplyDelete
  30. I think I'll update the command in the article since a lot of people don't take the time to go through the comments. The command as published did work for my on Ubuntu 9.04 (that's how I created the Public AMI) but there obviously are some problems with it.

    Thanks!

    ReplyDelete
  31. I tried following your instructions with a CentOS 5 ami. Things looked good until I tried to SSH to the new instance. This is the console output:

    http://pastebin.com/m104190e0

    Any ideas? Thanks.

    ReplyDelete
  32. Hi Greg -

    Sorry for the slow response, it's been kind of crazy this week.

    When you did the register_image, what kernel and ramdisk did you use? Did you make sure to use the same ones that were used in the original S3-backed AMI?

    Mitch

    ReplyDelete
  33. I am also using a Centos AMI (not sure if that's relevant) and ending up in the same place Greg is.

    One question I have: should the /ebs version of the filesystem have files in the /dev directory? Mine is empty.

    I did not use a ramdisk because my original AMI did not seem to do so. Is it possible that these instructions only work correctly for AMIs backed by a ramdisk?

    This thread suggests that there is some relationship between this error message and the lack of a ramdisk or a /dev :

    http://kerneltrap.org/node/6854

    I'll try copying the console manually and see what happens.

    ReplyDelete
  34. So I got it working after copying some but not all of the files from /dev into the EBS version of the machine instance. In particular, /dev/null /dev/console and /dev/tty* .

    As another tip to readers: I thought it would be "easier" to do this stuff by hand rather than by boto. But I sent all day cutting and pasting volume IDs, snapshot IDs, image IDs, instance IDs and domain names as I tried different variations. It would have been a lot faster to write a Python script that does the whole job end-to-end.

    Another tip: one must be very careful about "fstab" files that rely on sdb or sdc. EBS-backed instances do not have sdb and sdc by default. You must create them manually during ec2-register. Here's what my ec2-register looks like:

    ec2-register -n "App Server" -d "My App Server" -a x86_64 --root-device-name '/dev/sda1' --block-device-mapping /dev/sda1=snap-8d0260e4 --block-device-mapping /dev/sdb=ephemeral0 --block-device-mapping /dev/sdc=ephemeral1 --kernel aki-9800e5f1

    ReplyDelete
  35. So, did you use the "tar" approach or the "rsync" approach?

    ReplyDelete
  36. Honestly, I think I had some problems and ended up doing both, so I can't be confident.

    Now I'm stuck trying to set up /dev/sdb and /dev/sdc using boto.

    You saw the command line above that I'm trying to emulate with boto.

    Here's the XML it outputs (with -v)

    {ns1:RegisterImage}
    {ns1:name}Ayogo RudeServer - test 11{/ns1:name}
    {ns1:description}Ayogo application server image, based on rudeserver-3, which is based on rudeserver-2, which is based on rudeserver-1, which is based on rightscale AMIs{/ns1:description}
    {ns1:architecture}x86_64{/ns1:architecture}
    {ns1:kernelId}aki-9800e5f1{/ns1:kernelId}
    {ns1:rootDeviceName}/dev/sda1{/ns1:rootDeviceName}
    {ns1:blockDeviceMapping}
    {ns1:item}
    {ns1:deviceName}/dev/sda1{/ns1:deviceName}
    {ns1:ebs}
    {ns1:snapshotId}snap-8d0260e4{/ns1:snapshotId}
    {/ns1:ebs}
    {/ns1:item}
    {ns1:item}
    {ns1:deviceName}/dev/sdb{/ns1:deviceName}
    {ns1:virtualName}ephemeral0{/ns1:virtualName}
    {/ns1:item}
    {ns1:item}
    {ns1:deviceName}/dev/sdc{/ns1:deviceName}
    {ns1:virtualName}ephemeral1{/ns1:virtualName}
    {/ns1:item}
    {/ns1:blockDeviceMapping}
    {/ns1:RegisterImage}

    I hacked boto a bit but the closest I have gotten so far is:

    GET /?AWSAccessKeyId=XXX&Action=RegisterImage&Architecture=x86_64&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fsdb&BlockDeviceMapping.1.Ebs.DeleteOnTermination=false&BlockDeviceMapping.1.VirtualName=ephemeral0&BlockDeviceMapping.2.DeviceName=%2Fdev%2Fsda1&BlockDeviceMapping.2.Ebs.DeleteOnTermination=false&BlockDeviceMapping.2.Ebs.SnapshotId=snap-466f0d2f&BlockDeviceMapping.3.DeviceName=%2Fdev%2Fsdc&BlockDeviceMapping.3.Ebs.DeleteOnTermination=false&BlockDeviceMapping.3.VirtualName=ephemeral1&...

    The command seems to work, but the resulting AMI doesn't have the right mappings and the ec2-run results in an instance that terminates before it starts. I'll be annoyed (with Amazon) if I'm being charged an hour for each of these test instances.

    ReplyDelete
  37. This fails for me at the register_image step using boto 1.9b:

    >>> c.register_image('xxx', 'xxx', kernel_id=img.kernel_id, ramdisk_id=img.ramdisk_id, root_device_name='/dev/sda1', block_device_map=block_map)
    Traceback (most recent call last):
    File "", line 1, in
    File "/usr/lib/python2.6/site-packages/boto-1.9b-py2.6.egg/boto/ec2/connection.py", line 227, in register_image
    block_device_map.build_list_params(params)
    File "/usr/lib/python2.6/site-packages/boto-1.9b-py2.6.egg/boto/ec2/blockdevicemapping.py", line 78, in build_list_params
    pre = '%sBlockDeviceMapping.%d' % (pre, i)
    UnboundLocalError: local variable 'pre' referenced before assignment

    ReplyDelete
  38. Everything good in the end, I just "manually" registered the new AMI with ec2-register and after an initial failed boot due to an empty /dev directory, Paul's post solved my problems.

    Thanks everyone!

    ReplyDelete
  39. Yes, this was a bug introduced by me in r1404, after I wrote the article but before I bundled the 1.9a or 1.9b releases. It has been fixed in r1428. I'll bundle up another release soon or you can run the subversion code for now.

    ReplyDelete
  40. After playing with this, I figured out a real easy way to do this. Just create a 10GB EBS volume using elasticfox, attach it to /dev/sdb1 on the machine you want to migrate. Shut down all services on the machine you are copying (i.e. database, apache, etc.). Use the dd command to copy the machine to the EBS volume like so:

    dd bs=65536 if=/dev/sda1 of=/dev/sdb1

    fsck /dev/sdb1 (after dd is complete)

    mkdir /a;mount /dev/sdb1 /a

    remove /mnt entry in /a/etc/fstab file

    unmount /dev/sdb1

    create a snapshot of the ebs volume (elasticfox)

    register the snapshot image:

    ec2-register -s SNAPID -a arch -d "name of template" -n "template comments"

    (make sure you've set up EC2_PRIVATE_KEY and EC2_CERT environmentals befoe you run this command)

    Then boot up your new ebs backed image (also using elasticfox).

    If you need a larger volume (other than the 10GB default), you'll need to do this:

    ec2-run-instances AMI -n 1 -g securitygroup -t m1.small -k KEY -block-device-mapping /dev/sda1:50 (or whatever size you want)

    Once the machine is up and running, you'll need to login and do this to grow the volume:

    resize2fs /dev/sda1

    ReplyDelete
  41. I noticed that the all bundle-vol command issues the following commands:

    Executing: mknod /mnt/img-mnt/dev/null c 1 3
    Executing: mknod /mnt/img-mnt/dev/zero c 1 5
    Executing: mknod /mnt/img-mnt/dev/tty c 5 0
    Executing: mknod /mnt/img-mnt/dev/console c 5 1
    Executing: ln -s null /mnt/img-mnt/dev/X0R

    So I would infer that those are the minimum special files that you'll need to boot a new instance.

    ReplyDelete
  42. Answered my own question: You can create XFS-formatted boot-from-EBS volumes. Here's how:

    http://www.shlomoswidler.com/2010/01/creating-consistent-snapshots-of-live.html

    ReplyDelete
  43. Reguarding this error posted above:

    ...
    Loading mbcache.ko module
    Loading jbd.ko module
    Loading ext3.ko module
    Creating root device.
    Mounting root filesystem.
    VFS: Can't find ext3 filesystem on dev sda1.

    mount: error mounting /dev/root on /sysroot as ext3: Invalid argument
    Setting up other filesystems.
    Setting up new root fs
    setuproot: moving /dev failed: No such file or directory
    no fstab.sys, mounting internal defaults
    setuproot: error mounting /proc: No such file or directory
    setuproot: error mounting /sys: No such file or directory
    Switching to new root and running init.
    unmounting old /dev
    unmounting old /proc
    unmounting old /sys
    switchroot: mount failed: No such file or directory
    Booting has failed.

    ----

    I got this same exact error when trying to bundle an EBS image as XFS. Works just fine with ext3.

    Thanks for the post!!! Great help

    ReplyDelete
  44. Great post and thanks for doing this work.

    I had success using the dd method posted inside here, it worked great, on CentOS first try.

    I am booting the ebs with a CentOS beta kernel from Rightscale. Only problem is the 4gb seg fixup messages being spit out by the kernel constantly.. no good.

    Must be this Xen business the uname of the booted kernel ends in .fc8xen

    Gonna try another kernel here and see what happens.. I've seen lots of posts about this (in the Rails world mostly) but not many concrete solutions to that...

    ReplyDelete
  45. Howto create Amazon EC2 EBS-backed AMI of your favorite x86_64 Linux

    What you need

    Computer with any OS and virtualization (I used Sun Virtualbox) software installed
    Your favorite Linux(I used Centos 5.5 x86_64) installation CD or DVD
    Account on Amazon EC2 service
    Amazon EC2 User Guide (get it on http://aws.amazon.com/documentation/ec2/)
    Amazon EC2 AMI tools (google -> download amazon ami tools) ; I prefered to use rpm version
    Amazon EC2 API tools (google -> download amazon api tools)

    Usefull links

    http://www.philchen.com/2009/02/14/how-to-create-an-amazon-elastic-compute-cloud-ec2-machine-image-ami
    The way

    Create virtual machine(VM) and install Centos on it:

    * use SATA virtual hard disks - reasons: VM will operate faster ; your VM will use /dev/sdX devices like the VM on Amazon EC2
    * create two virtual hard disks for you VM
    * first hard disk(for OS) must have only two partitions: /dev/sda1 for root and /dev/sda2 for swap
    * second virtual hard disk (size >= 11G) and mount it in /mnt


    You need some credentials to work with Amazon Services:

    * get X.509 certificate and private key and put them into /root/.ec2 directory
    * get EC2 user ID(AWS account number)


    Read User Guide to know where to get the credentials

    Install Amazon EC2 AMI tools on your Centos
    the best way:
    # yum localinstall ec2-ami-tools.noarch.rpm

    Install Amazon EC2 API tools on your Centos
    simply decompress the archive
    install java jdk also:
    # yum install java-1.6.0-openjdk

    Set environment variables in .bashrc

    export EC2_HOME=/root/ec2-api-tools-1.3-51254
    export PATH=$PATH:$EC2_HOME/bin
    export EC2_PRIVATE_KEY=~/.ec2/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem
    export EC2_CERT=~/.ec2/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem
    export JAVA_HOME="/usr"

    run:
    # source .bashrc

    Bundle Volume

    # ec2-bundle-vol -c .ec2/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem -k .ec2/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem -u 7433-5125-3958 -d /mnt -r x86_64 -s 10240 --generate-fstab --no-inherit

    the result of ec2-bundle-vol command is 10G image file in /mnt and many image.* files

    Only image file is in interest to us beacause it is image of our /dev/sda1(root partition) of Amazon VM

    compress /mnt/image for example to /mnt/image.tar.bz2

    start an instance of any Linux VM on Amazon EC2(S3-backed or EBS-backed) and copy file image.tar.bz2 to it for example using scp

    create 10G volume on Amazon

    attach the volume to the running Linux VM on Amazon EC2 for example as /dev/sdh (no filesystem creation or mounting needed)

    decompress image.tar.bz2 in any place of running Linux VM on Amazon EC2

    on running Linux VM on Amazon EC2 run
    # dd bs=512M if=image of=/dev/sdh
    wait for command complet

    detach your 10G volume

    create snapshort of this 10G volume

    register the snapshort as ami(run this command on your Centos VM that runs on Virtualbox)
    # ec2-register -n "Any name you like" -d "Any description yuo like" -root-device-name /dev/sda1 -b /dev/sda1=snap-abcd1234:10:false -a x86_64

    !!! The critical option is -a x86_64 because if ommited your ami will be registered as 32bit AMI

    create an instace of you favorite x86_64 Linux using your own AMI

    you can logon to the instance using root password of you VM from Virtualbox(the passwords is identical)

    ReplyDelete
  46. @david - guess that command doesnt work...
    I used rsync -avx // /ebs
    I copied the contents of the current directory on to the ebs.


    mumrah said...

    Thanks for the tutorial, Mitch. Amazon's docs are rather lacking at the moment.

    The only thing I did different was to use rsync rather than the tar pipes.

    rsync -avx --exclude /ebs / /ebs

    -David

    ReplyDelete
  47. Hi guys,

    Does anyone know if there is a way to register the snapshot image through AWS Management Console?

    Thanks,
    empi

    ReplyDelete
  48. Any idea how us window server needy can do the same?

    ReplyDelete
  49. I follow the process, when I try:

    root@domU-12-31-39-02-31-51:~# sudo tar cpS / | cpipe -vt -b 1024 | gzip -c | tar zxpS -C /ebs

    I got the following cpipe message, but takes very long, more than 30 min still hasn't finish:

    thru: 13.027ms at 76.8MB/s ( 20.9MB/s avg) 117.2GB
    thru: 13.186ms at 75.8MB/s ( 20.9MB/s avg) 117.2GB
    thru: 73.351ms at 13.6MB/s ( 20.9MB/s avg) 117.2GB
    thru: 12.603ms at 79.3MB/s ( 20.9MB/s avg) 117.2GB
    thru: 72.915ms at 13.7MB/s ( 20.9MB/s avg) 117.2GB
    thru: 12.978ms at 77.1MB/s ( 20.9MB/s avg) 117.2GB
    thru: 72.213ms at 13.8MB/s ( 20.9MB/s avg) 117.2GB
    thru: 16.507ms at 60.6MB/s ( 20.9MB/s avg) 117.2GB
    thru: 47.859ms at 20.9MB/s ( 20.9MB/s avg) 117.2GB

    Also what does that 117.2GB mean? My volume is only 10G, is that a problem?

    ReplyDelete
  50. Here's a complete procedure on how to build your own S3- and EBS-backed AMIs using vanilla CentOS installation discs and run the kernel that comes with the discs: http://amazonaws.michael--martinez.com

    ReplyDelete
  51. I had the problem below when booting the AMI. Fixed it by reverting back to booting /dev/sda1 explicitly in menu.lst (also changed /etc/fstab).

    > Your problem is that you did not copy the filesystem label when you
    > created an image of the root filesystem. The current 10.04 and forward
    > images boot via pv-grub specifying 'root=LABEL=uec-rootfs'. This tells
    > the ramdisk to find a volume with a label of "uec-rootfs" and use it as
    > the root device.

    http://osdir.com/ml/ec2ubuntu/2011-04/msg00053.html

    ReplyDelete
  52. I have a very basic question, this procedure can be used to convert a instance-store backed ami to EBS backed ami? I am new to AWS.

    Thanks

    ReplyDelete
  53. ebs.delete_on_termination = True

    This will delete the root device on termination otherwise you need to manually delete the volume after terminating the instance.

    ReplyDelete