tag:blogger.com,1999:blog-1231480044619721857.post2510929286637560794..comments2022-12-02T05:07:00.365-05:00Comments on Elastician: Creating an EBS-backed AMI from an S3-backed AMIMitch Garnaathttp://www.blogger.com/profile/02589240083555476561noreply@blogger.comBlogger54125tag:blogger.com,1999:blog-1231480044619721857.post-41266543933803162552012-10-08T20:46:00.530-04:002012-10-08T20:46:00.530-04:00Thanks!Thanks!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-50569523741002256842012-01-16T04:02:22.992-05:002012-01-16T04:02:22.992-05:00ebs.delete_on_termination = True
This will delete...ebs.delete_on_termination = True<br /><br />This will delete the root device on termination otherwise you need to manually delete the volume after terminating the instance.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-42661288892039163772011-08-12T01:07:13.199-04:002011-08-12T01:07:13.199-04:00I have a very basic question, this procedure can b...I have a very basic question, this procedure can be used to convert a instance-store backed ami to EBS backed ami? I am new to AWS.<br /><br />ThanksAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-25783718549152000522011-06-22T03:56:10.660-04:002011-06-22T03:56:10.660-04:00I had the problem below when booting the AMI. Fix...I had the problem below when booting the AMI. Fixed it by reverting back to booting /dev/sda1 explicitly in menu.lst (also changed /etc/fstab).<br /><br />> Your problem is that you did not copy the filesystem label when you<br />> created an image of the root filesystem. The current 10.04 and forward<br />> images boot via pv-grub specifying 'root=LABEL=uec-rootfs'. This tells<br />> the ramdisk to find a volume with a label of "uec-rootfs" and use it as<br />> the root device.<br /><br />http://osdir.com/ml/ec2ubuntu/2011-04/msg00053.htmlAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-20992749646055633052011-05-23T16:53:23.862-04:002011-05-23T16:53:23.862-04:00Here's a complete procedure on how to build yo...Here's a complete procedure on how to build your own S3- and EBS-backed AMIs using vanilla CentOS installation discs and run the kernel that comes with the discs: http://amazonaws.michael--martinez.comM. Martinezhttp://www.michael--martinez.comnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-14648570393760696742011-04-06T11:56:02.720-04:002011-04-06T11:56:02.720-04:00I follow the process, when I try:
root@domU-12-31...I follow the process, when I try:<br /><br />root@domU-12-31-39-02-31-51:~# sudo tar cpS / | cpipe -vt -b 1024 | gzip -c | tar zxpS -C /ebs<br /><br />I got the following cpipe message, but takes very long, more than 30 min still hasn't finish:<br /><br />thru: 13.027ms at 76.8MB/s ( 20.9MB/s avg) 117.2GB<br />thru: 13.186ms at 75.8MB/s ( 20.9MB/s avg) 117.2GB<br />thru: 73.351ms at 13.6MB/s ( 20.9MB/s avg) 117.2GB<br />thru: 12.603ms at 79.3MB/s ( 20.9MB/s avg) 117.2GB<br />thru: 72.915ms at 13.7MB/s ( 20.9MB/s avg) 117.2GB<br />thru: 12.978ms at 77.1MB/s ( 20.9MB/s avg) 117.2GB<br />thru: 72.213ms at 13.8MB/s ( 20.9MB/s avg) 117.2GB<br />thru: 16.507ms at 60.6MB/s ( 20.9MB/s avg) 117.2GB<br />thru: 47.859ms at 20.9MB/s ( 20.9MB/s avg) 117.2GB<br /><br />Also what does that 117.2GB mean? My volume is only 10G, is that a problem?Unknownhttps://www.blogger.com/profile/12168467219625665653noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-14714247749790192992011-02-23T16:15:38.884-05:002011-02-23T16:15:38.884-05:00Any idea how us window server needy can do the sam...Any idea how us window server needy can do the same?John Samhttps://www.blogger.com/profile/05943372152909190001noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-86731888050482502072010-08-17T12:32:31.451-04:002010-08-17T12:32:31.451-04:00Hi guys,
Does anyone know if there is a way to re...Hi guys,<br /><br />Does anyone know if there is a way to register the snapshot image through AWS Management Console? <br /><br />Thanks,<br />empiAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-2580528450646102702010-07-07T16:22:59.652-04:002010-07-07T16:22:59.652-04:00@david - guess that command doesnt work...
I used ...@david - guess that command doesnt work...<br />I used rsync -avx // /ebs <br />I copied the contents of the current directory on to the ebs. <br /><br /><br />mumrah said...<br /><br /> Thanks for the tutorial, Mitch. Amazon's docs are rather lacking at the moment.<br /><br /> The only thing I did different was to use rsync rather than the tar pipes.<br /><br /> rsync -avx --exclude /ebs / /ebs<br /><br /> -DavidAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-45555618732129124222010-06-17T12:40:25.748-04:002010-06-17T12:40:25.748-04:00Howto create Amazon EC2 EBS-backed AMI of your fav...Howto create Amazon EC2 EBS-backed AMI of your favorite x86_64 Linux<br /><br />What you need<br /><br />Computer with any OS and virtualization (I used Sun Virtualbox) software installed<br />Your favorite Linux(I used Centos 5.5 x86_64) installation CD or DVD<br />Account on Amazon EC2 service<br />Amazon EC2 User Guide (get it on http://aws.amazon.com/documentation/ec2/)<br />Amazon EC2 AMI tools (google -> download amazon ami tools) ; I prefered to use rpm version<br />Amazon EC2 API tools (google -> download amazon api tools)<br /><br />Usefull links<br /><br />http://www.philchen.com/2009/02/14/how-to-create-an-amazon-elastic-compute-cloud-ec2-machine-image-ami<br />The way<br /><br />Create virtual machine(VM) and install Centos on it:<br /><br /> * use SATA virtual hard disks - reasons: VM will operate faster ; your VM will use /dev/sdX devices like the VM on Amazon EC2<br /> * create two virtual hard disks for you VM<br /> * first hard disk(for OS) must have only two partitions: /dev/sda1 for root and /dev/sda2 for swap<br /> * second virtual hard disk (size >= 11G) and mount it in /mnt<br /><br /><br />You need some credentials to work with Amazon Services:<br /><br /> * get X.509 certificate and private key and put them into /root/.ec2 directory<br /> * get EC2 user ID(AWS account number)<br /><br /><br />Read User Guide to know where to get the credentials<br /><br />Install Amazon EC2 AMI tools on your Centos<br />the best way:<br /># yum localinstall ec2-ami-tools.noarch.rpm<br /><br />Install Amazon EC2 API tools on your Centos<br />simply decompress the archive<br />install java jdk also:<br /># yum install java-1.6.0-openjdk<br /><br />Set environment variables in .bashrc<br /><br />export EC2_HOME=/root/ec2-api-tools-1.3-51254<br />export PATH=$PATH:$EC2_HOME/bin<br />export EC2_PRIVATE_KEY=~/.ec2/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem<br />export EC2_CERT=~/.ec2/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem<br />export JAVA_HOME="/usr"<br /><br />run:<br /># source .bashrc<br /><br />Bundle Volume<br /><br /># ec2-bundle-vol -c .ec2/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem -k .ec2/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem -u 7433-5125-3958 -d /mnt -r x86_64 -s 10240 --generate-fstab --no-inherit<br /><br />the result of ec2-bundle-vol command is 10G image file in /mnt and many image.* files<br /><br />Only image file is in interest to us beacause it is image of our /dev/sda1(root partition) of Amazon VM<br /><br />compress /mnt/image for example to /mnt/image.tar.bz2<br /><br />start an instance of any Linux VM on Amazon EC2(S3-backed or EBS-backed) and copy file image.tar.bz2 to it for example using scp<br /><br />create 10G volume on Amazon<br /><br />attach the volume to the running Linux VM on Amazon EC2 for example as /dev/sdh (no filesystem creation or mounting needed)<br /><br />decompress image.tar.bz2 in any place of running Linux VM on Amazon EC2<br /><br />on running Linux VM on Amazon EC2 run<br /># dd bs=512M if=image of=/dev/sdh<br />wait for command complet<br /><br />detach your 10G volume<br /><br />create snapshort of this 10G volume<br /><br />register the snapshort as ami(run this command on your Centos VM that runs on Virtualbox)<br /># ec2-register -n "Any name you like" -d "Any description yuo like" -root-device-name /dev/sda1 -b /dev/sda1=snap-abcd1234:10:false -a x86_64<br /><br />!!! The critical option is -a x86_64 because if ommited your ami will be registered as 32bit AMI<br /><br />create an instace of you favorite x86_64 Linux using your own AMI<br /><br />you can logon to the instance using root password of you VM from Virtualbox(the passwords is identical)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-27221395568685117282010-03-12T12:25:39.232-05:002010-03-12T12:25:39.232-05:00Great post and thanks for doing this work.
I had ...Great post and thanks for doing this work.<br /><br />I had success using the dd method posted inside here, it worked great, on CentOS first try.<br /><br />I am booting the ebs with a CentOS beta kernel from Rightscale. Only problem is the 4gb seg fixup messages being spit out by the kernel constantly.. no good.<br /><br />Must be this Xen business the uname of the booted kernel ends in .fc8xen<br /><br />Gonna try another kernel here and see what happens.. I've seen lots of posts about this (in the Rails world mostly) but not many concrete solutions to that...Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-52932255476516380012010-02-17T20:38:30.838-05:002010-02-17T20:38:30.838-05:00Reguarding this error posted above:
...
Loading m...Reguarding this error posted above:<br /><br />...<br />Loading mbcache.ko module<br />Loading jbd.ko module<br />Loading ext3.ko module<br />Creating root device.<br />Mounting root filesystem.<br />VFS: Can't find ext3 filesystem on dev sda1.<br /><br />mount: error mounting /dev/root on /sysroot as ext3: Invalid argument<br />Setting up other filesystems.<br />Setting up new root fs<br />setuproot: moving /dev failed: No such file or directory<br />no fstab.sys, mounting internal defaults<br />setuproot: error mounting /proc: No such file or directory<br />setuproot: error mounting /sys: No such file or directory<br />Switching to new root and running init.<br />unmounting old /dev<br />unmounting old /proc<br />unmounting old /sys<br />switchroot: mount failed: No such file or directory<br />Booting has failed.<br /><br />----<br /><br />I got this same exact error when trying to bundle an EBS image as XFS. Works just fine with ext3. <br /><br />Thanks for the post!!! Great helpAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-1816318848134728092010-01-17T16:28:49.160-05:002010-01-17T16:28:49.160-05:00Answered my own question: You can create XFS-forma...Answered my own question: You can create XFS-formatted boot-from-EBS volumes. Here's how:<br /><br />http://www.shlomoswidler.com/2010/01/creating-consistent-snapshots-of-live.htmlShlomohttps://www.blogger.com/profile/10469902663120418195noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-82358139990651795262010-01-04T13:06:37.988-05:002010-01-04T13:06:37.988-05:00I noticed that the all bundle-vol command issues t...I noticed that the all bundle-vol command issues the following commands:<br /><br />Executing: mknod /mnt/img-mnt/dev/null c 1 3 <br />Executing: mknod /mnt/img-mnt/dev/zero c 1 5 <br />Executing: mknod /mnt/img-mnt/dev/tty c 5 0 <br />Executing: mknod /mnt/img-mnt/dev/console c 5 1 <br />Executing: ln -s null /mnt/img-mnt/dev/X0R <br /><br />So I would infer that those are the minimum special files that you'll need to boot a new instance.Paul Prescodhttps://www.blogger.com/profile/15412258048017521995noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-65881643370913945672010-01-01T17:17:35.436-05:002010-01-01T17:17:35.436-05:00After playing with this, I figured out a real easy...After playing with this, I figured out a real easy way to do this. Just create a 10GB EBS volume using elasticfox, attach it to /dev/sdb1 on the machine you want to migrate. Shut down all services on the machine you are copying (i.e. database, apache, etc.). Use the dd command to copy the machine to the EBS volume like so:<br /><br />dd bs=65536 if=/dev/sda1 of=/dev/sdb1<br /><br />fsck /dev/sdb1 (after dd is complete)<br /><br />mkdir /a;mount /dev/sdb1 /a<br /><br />remove /mnt entry in /a/etc/fstab file<br /><br />unmount /dev/sdb1<br /><br />create a snapshot of the ebs volume (elasticfox)<br /><br />register the snapshot image:<br /><br />ec2-register -s SNAPID -a arch -d "name of template" -n "template comments"<br /><br />(make sure you've set up EC2_PRIVATE_KEY and EC2_CERT environmentals befoe you run this command)<br /><br />Then boot up your new ebs backed image (also using elasticfox).<br /><br />If you need a larger volume (other than the 10GB default), you'll need to do this:<br /><br />ec2-run-instances AMI -n 1 -g securitygroup -t m1.small -k KEY -block-device-mapping /dev/sda1:50 (or whatever size you want)<br /><br />Once the machine is up and running, you'll need to login and do this to grow the volume:<br /><br />resize2fs /dev/sda1Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-60327860487025726852009-12-28T09:00:59.207-05:002009-12-28T09:00:59.207-05:00Yes, this was a bug introduced by me in r1404, aft...Yes, this was a bug introduced by me in r1404, after I wrote the article but before I bundled the 1.9a or 1.9b releases. It has been fixed in r1428. I'll bundle up another release soon or you can run the subversion code for now.Mitch Garnaathttps://www.blogger.com/profile/02589240083555476561noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-32976055964007262582009-12-28T08:55:15.094-05:002009-12-28T08:55:15.094-05:00Everything good in the end, I just "manually&...Everything good in the end, I just "manually" registered the new AMI with ec2-register and after an initial failed boot due to an empty /dev directory, Paul's post solved my problems.<br /><br />Thanks everyone!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-32973364629167469982009-12-28T06:21:52.189-05:002009-12-28T06:21:52.189-05:00This fails for me at the register_image step using...This fails for me at the register_image step using boto 1.9b:<br /><br />>>> c.register_image('xxx', 'xxx', kernel_id=img.kernel_id, ramdisk_id=img.ramdisk_id, root_device_name='/dev/sda1', block_device_map=block_map)<br />Traceback (most recent call last):<br /> File "", line 1, in <br /> File "/usr/lib/python2.6/site-packages/boto-1.9b-py2.6.egg/boto/ec2/connection.py", line 227, in register_image<br /> block_device_map.build_list_params(params)<br /> File "/usr/lib/python2.6/site-packages/boto-1.9b-py2.6.egg/boto/ec2/blockdevicemapping.py", line 78, in build_list_params<br /> pre = '%sBlockDeviceMapping.%d' % (pre, i)<br />UnboundLocalError: local variable 'pre' referenced before assignmentAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-60731338248367623342009-12-13T17:06:13.336-05:002009-12-13T17:06:13.336-05:00Honestly, I think I had some problems and ended up...Honestly, I think I had some problems and ended up doing both, so I can't be confident.<br /><br />Now I'm stuck trying to set up /dev/sdb and /dev/sdc using boto.<br /><br />You saw the command line above that I'm trying to emulate with boto.<br /><br />Here's the XML it outputs (with -v)<br /><br /> {ns1:RegisterImage}<br /> {ns1:name}Ayogo RudeServer - test 11{/ns1:name}<br /> {ns1:description}Ayogo application server image, based on rudeserver-3, which is based on rudeserver-2, which is based on rudeserver-1, which is based on rightscale AMIs{/ns1:description}<br /> {ns1:architecture}x86_64{/ns1:architecture}<br /> {ns1:kernelId}aki-9800e5f1{/ns1:kernelId}<br /> {ns1:rootDeviceName}/dev/sda1{/ns1:rootDeviceName}<br /> {ns1:blockDeviceMapping}<br /> {ns1:item}<br /> {ns1:deviceName}/dev/sda1{/ns1:deviceName}<br /> {ns1:ebs}<br /> {ns1:snapshotId}snap-8d0260e4{/ns1:snapshotId}<br /> {/ns1:ebs}<br /> {/ns1:item}<br /> {ns1:item}<br /> {ns1:deviceName}/dev/sdb{/ns1:deviceName}<br /> {ns1:virtualName}ephemeral0{/ns1:virtualName}<br /> {/ns1:item}<br /> {ns1:item}<br /> {ns1:deviceName}/dev/sdc{/ns1:deviceName}<br /> {ns1:virtualName}ephemeral1{/ns1:virtualName}<br /> {/ns1:item}<br /> {/ns1:blockDeviceMapping}<br /> {/ns1:RegisterImage}<br /><br />I hacked boto a bit but the closest I have gotten so far is:<br /><br />GET /?AWSAccessKeyId=XXX&Action=RegisterImage&Architecture=x86_64&BlockDeviceMapping.1.DeviceName=%2Fdev%2Fsdb&BlockDeviceMapping.1.Ebs.DeleteOnTermination=false&BlockDeviceMapping.1.VirtualName=ephemeral0&BlockDeviceMapping.2.DeviceName=%2Fdev%2Fsda1&BlockDeviceMapping.2.Ebs.DeleteOnTermination=false&BlockDeviceMapping.2.Ebs.SnapshotId=snap-466f0d2f&BlockDeviceMapping.3.DeviceName=%2Fdev%2Fsdc&BlockDeviceMapping.3.Ebs.DeleteOnTermination=false&BlockDeviceMapping.3.VirtualName=ephemeral1&...<br /><br />The command seems to work, but the resulting AMI doesn't have the right mappings and the ec2-run results in an instance that terminates before it starts. I'll be annoyed (with Amazon) if I'm being charged an hour for each of these test instances.Paul Prescodnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-31605924101935921582009-12-13T14:09:35.598-05:002009-12-13T14:09:35.598-05:00So, did you use the "tar" approach or th...So, did you use the "tar" approach or the "rsync" approach?Mitch Garnaathttps://www.blogger.com/profile/02589240083555476561noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-51563842544049678872009-12-13T12:07:41.036-05:002009-12-13T12:07:41.036-05:00So I got it working after copying some but not all...So I got it working after copying some but not all of the files from /dev into the EBS version of the machine instance. In particular, /dev/null /dev/console and /dev/tty* .<br /><br />As another tip to readers: I thought it would be "easier" to do this stuff by hand rather than by boto. But I sent all day cutting and pasting volume IDs, snapshot IDs, image IDs, instance IDs and domain names as I tried different variations. It would have been a lot faster to write a Python script that does the whole job end-to-end.<br /><br />Another tip: one must be very careful about "fstab" files that rely on sdb or sdc. EBS-backed instances do not have sdb and sdc by default. You must create them manually during ec2-register. Here's what my ec2-register looks like:<br /><br />ec2-register -n "App Server" -d "My App Server" -a x86_64 --root-device-name '/dev/sda1' --block-device-mapping /dev/sda1=snap-8d0260e4 --block-device-mapping /dev/sdb=ephemeral0 --block-device-mapping /dev/sdc=ephemeral1 --kernel aki-9800e5f1Paul Prescodnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-28153538015220422212009-12-12T19:11:48.537-05:002009-12-12T19:11:48.537-05:00I am also using a Centos AMI (not sure if that'...I am also using a Centos AMI (not sure if that's relevant) and ending up in the same place Greg is.<br /><br />One question I have: should the /ebs version of the filesystem have files in the /dev directory? Mine is empty.<br /><br />I did not use a ramdisk because my original AMI did not seem to do so. Is it possible that these instructions only work correctly for AMIs backed by a ramdisk?<br /><br />This thread suggests that there is some relationship between this error message and the lack of a ramdisk or a /dev :<br /><br />http://kerneltrap.org/node/6854<br /><br />I'll try copying the console manually and see what happens.Paulnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-38182502813431415772009-12-12T08:22:38.653-05:002009-12-12T08:22:38.653-05:00Hi Greg -
Sorry for the slow response, it's b...Hi Greg -<br /><br />Sorry for the slow response, it's been kind of crazy this week.<br /><br />When you did the register_image, what kernel and ramdisk did you use? Did you make sure to use the same ones that were used in the original S3-backed AMI?<br /><br />MitchUnknownhttps://www.blogger.com/profile/08349877802381481792noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-49337559255898018062009-12-10T15:49:16.814-05:002009-12-10T15:49:16.814-05:00I tried following your instructions with a CentOS ...I tried following your instructions with a CentOS 5 ami. Things looked good until I tried to SSH to the new instance. This is the console output:<br /><br />http://pastebin.com/m104190e0<br /><br />Any ideas? Thanks.Greg Martynhttps://www.blogger.com/profile/00426568700306342105noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-15308425797908316572009-12-08T10:39:00.215-05:002009-12-08T10:39:00.215-05:00I think I'll update the command in the article...I think I'll update the command in the article since a lot of people don't take the time to go through the comments. The command as published did work for my on Ubuntu 9.04 (that's how I created the Public AMI) but there obviously are some problems with it.<br /><br />Thanks!Mitch Garnaathttps://www.blogger.com/profile/02589240083555476561noreply@blogger.com