In Lori MacVittie's latest blog, "The Devil Is In The Details" she not only bestows upon me the honor of my own Theorem (yeah, in your face Pythagorus) she also gives a number of great examples of some of the necessary dimensions of scale beyond just the number of servers.
But besides things like networking and bandwidth, there is a softer side of scale that is equally important: people. You need a certain critical mass of support, billing, operations, development teams, security, sales, developer support, evangelists, etc. to create a viable service offering and economies of scale apply to these dimensions just as in hardware.
There may be niche markets where small providers can provide some unique value-add (specialized security procedures, vertical focus, non-standard technology stacks, etc.) but in general I think the dominance of scale is inevitable. As a developer I love the flexibility and programmability of cloud computing services but ultimately the trump card for businesses is cost and the best way to drive cost down is via scale.
Over the next five years, I think the majority of cloud computing will happen on public clouds and that the public cloud landscape will consist mainly of a relatively small number of big players who will be able to scale their services, both the hard side and the soft side, to achieve the economies of scale required in the marketplace.
Monday, February 15, 2010
Tuesday, February 9, 2010
Using S3 Versioning and MFA to CMA*
* - CMA = Cover My Ass
Amazon's Simple Storage Service (S3) is a great way to safely store loads of data in the cloud. It's highly available, simple to use and provides good data durability by automatically copying your data across multiple regions and/or zones. With over 80 billion objects stored (at last published count) I'm clearly not alone in thinking it's a good thing.
The only problem I've had with S3 over the years is the queazy feeling I get when I think about some nefarious individual getting hold of my AWS AccessKey/SecretKey. Since all S3 capabilities are accessed via a REST API and since that credential pair is used to authenticate all requests with S3, a bad guy/girl with my credentials (or a temporarily stupid version of me) could potentially delete all of the content I have stored in S3. That represents the "Worst Case Scenario" of S3 usage and I've spent a considerable amount of time and effort trying to find ways to mitigate this risk.
Using multiple AWS accounts can help. The Import/Export feature is another way to mitigate your exposure. But what I've always wanted was a WORM (Write Once Read Many) bucket. Well, not always, but at least since May 6, 2007. That would give me confidence that the data I store in S3 could not be accidentally or maliciously deleted. This kind of feature would also provide some interesting functionality for certain types of compliance and regulatory solutions.
Starting today, AWS has released a couple of really useful new features in S3: Versioning and MFADelete. Together, these features provide just about everything I wanted when I asked for a WORM bucket. So, how do they work?
Versioning
Versioning allows you to have multiple copies of the same object. Each version has a unique version ID and the versions are kept in ascending order by the date the version was created. Each bucket can be configured to either enable or disable versioning (only by the bucket owner) and the basic behavior is shown below in the table. The behavior of a Versioned bucket differs based on whether it is being accessed by a Version-Aware (VA) client or NonVersion-Aware (NVA) client.
The above table is just a summary. You should see the S3 documentation for full details but even this summary clearly shows the benefits of versioning. If I enable versioning on a bucket, the chance of accidentally deleting content is greatly reduced. I would have to be using a version-aware delete tool and explicitly referencing individual version ID's to permanently delete them.
So, accidental deletion of content is less of a risk with versioning but how about the other risk? If a bad guy/girl gets my AccessKey/SecretKey, they can still delete all of my content as long as they know how to use the versioning feature of S3. To address this threat, S3 has implemented a new feature called MFADelete.
MFADelete
MFADelete uses the Multi-Factor Authentication device you are already using to protect AWS Portal and Console access. What? You aren't using the MFA device? Well, you should go sign up for one right now. It's well worth the money, especially if you are storing important content in S3.
Like Versioning, MFADelete can be enabled on a bucket-by-bucket basis and only by the owner of the bucket. But, rather than just trusting that the person with the AccessKey/SecretKey is the owner, MFADelete uses the MFA device to provide an additional factor of authentication. To enable MFADelete, you send a special PUT request to S3 with an XML body that looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Status>Enabled</Status>
<MfaDelete>Enabled</MfaDelete>
</VersioningConfiguration>
In addition to this XML body, you also need to send a special HTTP header in the request, like this:
x-amz-mfa: <serial number of MFA device> <token from MFA device>
Once this request has been sent, all delete operations on the bucket and all requests to change the MFADelete status for the bucket will also require the special HTTP header with the MFA information. So, that means that even if the bad guy/girl gets your AccessKey/SecretKey combo they still won't be able to delete anything from your MFADelete-enabled bucket without the MFA device, as well.
It's not exactly the WORM bucket I was originally hoping for but it's a huge improvement and greatly reduces the risk of accidental or malicious deletion of data from S3. I got my pony!
The code in the boto subversion repo has already been updated to work with the new Versioning and MFADelete features. A new release will be out in the near future. I have included a link below to a unit test script that shows most of the basic operations and should give you a good start on incorporating these great new features into your application. The script prompts for you for the serial number of your MFA device once and then prompts for a new MFA code each time on is required. You can only perform one operation with each code so you will have to wait for the device to cycle to the next code between each operation.
Example Code
Amazon's Simple Storage Service (S3) is a great way to safely store loads of data in the cloud. It's highly available, simple to use and provides good data durability by automatically copying your data across multiple regions and/or zones. With over 80 billion objects stored (at last published count) I'm clearly not alone in thinking it's a good thing.
The only problem I've had with S3 over the years is the queazy feeling I get when I think about some nefarious individual getting hold of my AWS AccessKey/SecretKey. Since all S3 capabilities are accessed via a REST API and since that credential pair is used to authenticate all requests with S3, a bad guy/girl with my credentials (or a temporarily stupid version of me) could potentially delete all of the content I have stored in S3. That represents the "Worst Case Scenario" of S3 usage and I've spent a considerable amount of time and effort trying to find ways to mitigate this risk.
Using multiple AWS accounts can help. The Import/Export feature is another way to mitigate your exposure. But what I've always wanted was a WORM (Write Once Read Many) bucket. Well, not always, but at least since May 6, 2007. That would give me confidence that the data I store in S3 could not be accidentally or maliciously deleted. This kind of feature would also provide some interesting functionality for certain types of compliance and regulatory solutions.
Starting today, AWS has released a couple of really useful new features in S3: Versioning and MFADelete. Together, these features provide just about everything I wanted when I asked for a WORM bucket. So, how do they work?
Versioning
Versioning allows you to have multiple copies of the same object. Each version has a unique version ID and the versions are kept in ascending order by the date the version was created. Each bucket can be configured to either enable or disable versioning (only by the bucket owner) and the basic behavior is shown below in the table. The behavior of a Versioned bucket differs based on whether it is being accessed by a Version-Aware (VA) client or NonVersion-Aware (NVA) client.
| Operation | Unversioned Bucket | Versioned Bucket - NVA Client | Versioned Bucket - VA Client |
|---|---|---|---|
| GET | Retrieves the object or a 404 if the object is not found | Retrieves the latest version or a 404 if a Delete Marker is found | Retrieves the version specified by provided version ID |
| PUT | Stores the content in the bucket, overwriting any existing content | Stores content as new version | Stores content as new version |
| DELETE | Irrevocably deletes the content | Stores a DeleteMarker as latest version of object. | Permanently deletes version specified by provided version ID |
The above table is just a summary. You should see the S3 documentation for full details but even this summary clearly shows the benefits of versioning. If I enable versioning on a bucket, the chance of accidentally deleting content is greatly reduced. I would have to be using a version-aware delete tool and explicitly referencing individual version ID's to permanently delete them.
So, accidental deletion of content is less of a risk with versioning but how about the other risk? If a bad guy/girl gets my AccessKey/SecretKey, they can still delete all of my content as long as they know how to use the versioning feature of S3. To address this threat, S3 has implemented a new feature called MFADelete.
MFADelete
MFADelete uses the Multi-Factor Authentication device you are already using to protect AWS Portal and Console access. What? You aren't using the MFA device? Well, you should go sign up for one right now. It's well worth the money, especially if you are storing important content in S3.
Like Versioning, MFADelete can be enabled on a bucket-by-bucket basis and only by the owner of the bucket. But, rather than just trusting that the person with the AccessKey/SecretKey is the owner, MFADelete uses the MFA device to provide an additional factor of authentication. To enable MFADelete, you send a special PUT request to S3 with an XML body that looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Status>Enabled</Status>
<MfaDelete>Enabled</MfaDelete>
</VersioningConfiguration>
In addition to this XML body, you also need to send a special HTTP header in the request, like this:
x-amz-mfa: <serial number of MFA device> <token from MFA device>
Once this request has been sent, all delete operations on the bucket and all requests to change the MFADelete status for the bucket will also require the special HTTP header with the MFA information. So, that means that even if the bad guy/girl gets your AccessKey/SecretKey combo they still won't be able to delete anything from your MFADelete-enabled bucket without the MFA device, as well.
It's not exactly the WORM bucket I was originally hoping for but it's a huge improvement and greatly reduces the risk of accidental or malicious deletion of data from S3. I got my pony!
The code in the boto subversion repo has already been updated to work with the new Versioning and MFADelete features. A new release will be out in the near future. I have included a link below to a unit test script that shows most of the basic operations and should give you a good start on incorporating these great new features into your application. The script prompts for you for the serial number of your MFA device once and then prompts for a new MFA code each time on is required. You can only perform one operation with each code so you will have to wait for the device to cycle to the next code between each operation.
Example Code
Monday, December 21, 2009
Boto 1.9a released
Hi -
I have just uploaded a new version of boto to the downloads section at http://boto.googlecode.com/. Version 1.9a is a significant and long overdue release that includes, among other things:
Mitch
I have just uploaded a new version of boto to the downloads section at http://boto.googlecode.com/. Version 1.9a is a significant and long overdue release that includes, among other things:
- Support for Virtual Private Cloud (VPC)
- Support for Relational Data Service (RDS)
- Support for Shared EBS Snapshots
- Support for Boot From EBS
- Support for Spot Instances
- CloudFront private and streaming Distributions
- Use of POST in data-heavy requests in ec2 and sdb modules
- Support for new us-west-1 region
- Fixes for more than 25 issues
Mitch
Wednesday, December 16, 2009
Private and Streaming Distributions in CloudFront
Boto has supported the CloudFront content delivery service since it's initial launch in November of 2008. CloudFront has recently launched a couple of great new features:
- Distributing Private Content
- Streaming Distributions
While adding support for these features to boto, I also took the opportunity to (hopefully) improve the overall boto support for CloudFront. In this article, I'll take a quick tour of the new CloudFront features and in the process cover the improved support for CloudFront in boto.
First, a little refresher. The main abstraction in CloudFront is a Distribution and in CloudFront all Distributions are backed by an S3 bucket, referred to as the Origin. Until recently, all content distributed by CloudFront had to be public content because there was no mechanism to control access to the content.
To create a new Distribution for public content, let's assume that we already have an S3 bucket called my-origin that we want to use as the Origin:
>>> import boto
>>> c = boto.connect_cloudfront()
>>> d = c.create_distribution(origin='my-origin.s3.amazonaws.com', enabled=True, caller_reference='My Distribution')
>>> d.domain_name
d33unmref5340o.cloudfront.net
So, d now points to my new CloudFront Distribution, backed by my S3 bucket called my-origin. Boto makes it easy to add content objects to my new Distribution. For example, let's assume that I have a JPEG image on my local computer that I want to place in my new Distribution:
>>> fp = open('/home/mitch/mycoolimage.jpg')
>>> obj = d.add_object('mycoolimage.jpg', fp)
>>>
Not only does the add_object method copy the content to the correct S3 bucket, it also makes sure the S3 ACL is set correctly for the type of Distribution. In this case, since it is a public Distribution the content object will be publicly readable.
You can also list all objects currently in the Distribution (or rather it's underlying bucket) by calling the get_objects method and you can also get the CloudFront URL for any object by using it's url method:
>>> d.get_objects()
[] >>> obj.url() http://d33unmref5340o.cloudfront.net/mycoolimage.jpg Don't Cross the Streams
The recently announced streaming feature of CloudFront will be of interest to anyone that needs to server audio or video. The nice thing about streaming is that only the content that the user actually watches or listens to is downloaded so if you have users with short attention spans, you can potentially save a lot of bandwidth costs. Plus, the streaming protocols support the ability to serve different quality media based on the user's available bandwidth.
To take advantage of these cool features, all you have to do is store streamable media files (e.g. FLV, MP3, MP4) in your origin bucket and then CloudFront will make those files available via RTMP, RTMPT, RTMPE or RTMPTE protocol using Adobe's Flash Media Server (see the CloudFront Developer's Guide for details).
The process for creating a new Streaming Distribution is almost identical to the above process.
>>> sd = c.create_streaming_distribution('my-origin.s3.amazonaws.com', True, 'My Streaming Distribution')
>>> fp = open('/home/mitch/embarrassingvideo.flv')
>>> strmobj = sd.add_object('embarrassingvideo.flv', fp)
>>> strmobj.url()
u'rtmp://sj6oeasqgt12x.cloudfront.net/cfx/st/embarrassingvideo.flv'
Note that the url method still returns the correct URL to embed in your media player to access the streaming content.
My Own Private Idaho
Another new feature in CloudFront is the ability to distribute private content across the CloudFront content delivery network. This is really a two-part process:
- Secure the content in S3 so only you and CloudFront have access to it
- Create signed URL's pointing to the secure content that can be distributed to whoever you want to be able to access the content
I'm only going to cover the first part of the process here. The CloudFront Developer's Guide provides detailed instructions for creating the signed URL's. Eventually, I'd like to be able to create the signed URL's directly in boto but doing so requires some non-standard Python libraries to handle the RSA-SHA1 signing and that is something I try to avoid in boto.
Let's say that we want to take the public Distribution I created above and turn it into a private Distribution. The first thing we need to do is create an Origin Access Identity (OAI). The OAI is a kind of virtual AWS account. By granting the OAI (and only the OAI) read access to your private content it allows you to keep the content private but allow the CloudFront service to access it.
Let's create a new Origin Access Identity and associate it with our Distribution:
>>> oai = c.create_origin_access_identity('my_oai', 'An OAI for testing')
>>> d.update(origin_access_identity=oai)
If there is an Origin Access Identity associated with a Distribution then the add_object method will ensure that the ACL for any objects added to the distribution is set so that the OAI has READ access to the object. In addition, by default it will also configure the ACL so that all other grants are removed so only the owner and the OAI have access. You can override this behavior by passing replace=False to the add_object call.
Finally, boto makes it easy to add trusted signers to your private Distribution. A trusted signer is another AWS account that has been authorized to create signed URL's for your private Distribution. To enable another AWS account, you need that accounts AWS Account ID (see this for an explanation about the Account ID).
>>> from boto.cloudfront.signers import TrustedSigners
>>> ts = TrustedSigners()
>>> ts.append('084307701560')
>>> d.update(trusted_signers=ts)
As I said earlier, I'm not going to go into the process of actually creating the signed URL's in this blog post. The CloudFront docs do a good job of explaining this and until I come up with a way to support the signing process in boto, I don't really have anything to add.
Thursday, December 10, 2009
Comprehensive List of AWS Endpoints
Note: AWS has now started their own list of API endpoints here. You may want to begin using that list as the definitive reference.
Another Note: I am now collecting and publishing this information as JSON data. I am generating the HTML below from this JSON data.
Guy Rosen (@guyro on Twitter) recently asked about a comprehensive list of AWS service endpoints. This information is notoriously difficult to find and seems to be spread across many different documents, release notes, etc. Fortunately, I had most of this information already gathered together in the boto source code so I pulled that together and hunted down the stragglers and put this list together.
If you have any more information to provide or have corrections, etc. please comment below. I'll try to keep this up to date over time.
Auto Scaling
Another Note: I am now collecting and publishing this information as JSON data. I am generating the HTML below from this JSON data.
Guy Rosen (@guyro on Twitter) recently asked about a comprehensive list of AWS service endpoints. This information is notoriously difficult to find and seems to be spread across many different documents, release notes, etc. Fortunately, I had most of this information already gathered together in the boto source code so I pulled that together and hunted down the stragglers and put this list together.
If you have any more information to provide or have corrections, etc. please comment below. I'll try to keep this up to date over time.
Auto Scaling
- us-east-1: autoscaling.us-east-1.amazonaws.com
- us-west-1: autoscaling.us-west-1.amazonaws.com
- us-west-2: autoscaling.us-west-2.amazonaws.com
- sa-east-1: autoscaling.sa-east-1.amazonaws.com
- eu-west-1: autoscaling.eu-west-1.amazonaws.com
- ap-southeast-1: autoscaling.ap-southeast-1.amazonaws.com
- ap-southeast-2: autoscaling.ap-southeast-2.amazonaws.com
- ap-northeast-1: autoscaling.ap-northeast-1.amazonaws.com
- us-east-1: cloudformation.us-east-1.amazonaws.com
- us-west-1: cloudformation.us-west-1.amazonaws.com
- us-west-2: cloudformation.us-west-2.amazonaws.com
- sa-east-1: cloudformation.sa-east-1.amazonaws.com
- eu-west-1: cloudformation.eu-west-1.amazonaws.com
- ap-southeast-1: cloudformation.ap-southeast-1.amazonaws.com
- ap-southeast-2: cloudformation.ap-southeast-2.amazonaws.com
- ap-northeast-1: cloudformation.ap-northeast-1.amazonaws.com
- universal: cloudfront.amazonaws.com
- us-east-1: cloudsearch.us-east-1.amazonaws.com
- us-east-1: monitoring.us-east-1.amazonaws.com
- us-west-1: monitoring.us-west-1.amazonaws.com
- us-west-2: monitoring.us-west-2.amazonaws.com
- sa-east-1: monitoring.sa-east-1.amazonaws.com
- eu-west-1: monitoring.eu-west-1.amazonaws.com
- ap-southeast-1: monitoring.ap-southeast-1.amazonaws.com
- ap-southeast-2: monitoring.ap-southeast-2.amazonaws.com
- ap-northeast-1: monitoring.ap-northeast-1.amazonaws.com
- universal: ls.amazonaws.com
- us-east-1: dynamodb.us-east-1.amazonaws.com
- us-west-1: dynamodb.us-west-1.amazonaws.com
- us-west-2: dynamodb.us-west-2.amazonaws.com
- ap-northeast-1: dynamodb.ap-northeast-1.amazonaws.com
- ap-southeast-1: dynamodb.ap-southeast-1.amazonaws.com
- ap-southeast-2: dynamodb.ap-southeast-2.amazonaws.com
- eu-west-1: dynamodb.eu-west-1.amazonaws.com
- us-east-1: elasticache.us-east-1.amazonaws.com
- us-west-1: elasticache.us-west-1.amazonaws.com
- us-west-2: elasticache.us-west-2.amazonaws.com
- sa-east-1: elasticache.sa-east-1.amazonaws.com
- eu-west-1: elasticache.eu-west-1.amazonaws.com
- ap-southeast-1: elasticache.ap-southeast-1.amazonaws.com
- ap-northeast-1: elasticache.ap-northeast-1.amazonaws.com
- us-east-1: elasticbeanstalk.us-east-1.amazonaws.com
- us-west-1: elasticbeanstalk.us-west-1.amazonaws.com
- us-west-2: elasticbeanstalk.us-west-2.amazonaws.com
- ap-northeast-1: elasticbeanstalk.ap-northeast-1.amazonaws.com
- ap-southeast-1: elasticbeanstalk.ap-southeast-1.amazonaws.com
- ap-southeast-2: elasticbeanstalk.ap-southeast-2.amazonaws.com
- eu-west-1: elasticbeanstalk.eu-west-1.amazonaws.com
- us-east-1: ec2.us-east-1.amazonaws.com
- us-west-1: ec2.us-west-1.amazonaws.com
- us-west-2: ec2.us-west-2.amazonaws.com
- sa-east-1: ec2.sa-east-1.amazonaws.com
- eu-west-1: ec2.eu-west-1.amazonaws.com
- ap-southeast-1: ec2.ap-southeast-1.amazonaws.com
- ap-southeast-2: ec2.ap-southeast-2.amazonaws.com
- ap-northeast-1: ec2.ap-northeast-1.amazonaws.com
- us-east-1: elasticloadbalancing.us-east-1.amazonaws.com
- us-west-1: elasticloadbalancing.us-west-1.amazonaws.com
- us-west-2: elasticloadbalancing.us-west-2.amazonaws.com
- sa-east-1: elasticloadbalancing.sa-east-1.amazonaws.com
- eu-west-1: elasticloadbalancing.eu-west-1.amazonaws.com
- ap-southeast-1: elasticloadbalancing.ap-southeast-1.amazonaws.com
- ap-southeast-2: elasticloadbalancing.ap-southeast-2.amazonaws.com
- ap-northeast-1: elasticloadbalancing.ap-northeast-1.amazonaws.com
- us-east-1: elasticmapreduce.us-east-1.amazonaws.com
- us-west-1: elasticmapreduce.us-west-1.amazonaws.com
- us-west-2: elasticmapreduce.us-west-2.amazonaws.com
- sa-east-1: elasticmapreduce.sa-east-1.amazonaws.com
- eu-west-1: elasticmapreduce.eu-west-1.amazonaws.com
- ap-southeast-1: elasticmapreduce.ap-southeast-1.amazonaws.com
- ap-southeast-2: elasticmapreduce.ap-southeast-2.amazonaws.com
- ap-northeast-1: elasticmapreduce.ap-northeast-1.amazonaws.com
- sandbox: authorize.payments-sandbox.amazon.com/cobranded-ui/actions/start
- production: authorize.payments.amazon.com/cobranded-ui/actions/start
- sandbox: fps.sandbox.amazonaws.com
- production: fps.amazonaws.com
- us-east-1: glacier.us-east-1.amazonaws.com
- us-west-1: glacier.us-west-1.amazonaws.com
- us-west-2: glacier.us-west-2.amazonaws.com
- eu-west-1: glacier.eu-west-1.amazonaws.com
- ap-northeast-1: glacier.ap-northeast-1.amazonaws.com
- universal: iam.amazonaws.com
- universal: importexport.amazonaws.com
- universal: mechanicalturk.amazonaws.com
- us-east-1: rds.us-east-1.amazonaws.com
- us-west-1: rds.us-west-1.amazonaws.com
- us-west-2: rds.us-west-2.amazonaws.com
- sa-east-1: rds.sa-east-1.amazonaws.com
- eu-west-1: rds.eu-west-1.amazonaws.com
- ap-southeast-1: rds.ap-southeast-1.amazonaws.com
- ap-southeast-2: rds.ap-southeast-2.amazonaws.com
- ap-northeast-1: rds.ap-northeast-1.amazonaws.com
- universal: route53.amazonaws.com
- universal: sts.amazonaws.com
- us-east-1: email.us-east-1.amazonaws.com
- us-east-1: sns.us-east-1.amazonaws.com
- us-west-1: sns.us-west-1.amazonaws.com
- us-west-2: sns.us-west-2.amazonaws.com
- sa-east-1: sns.sa-east-1.amazonaws.com
- eu-west-1: sns.eu-west-1.amazonaws.com
- ap-southeast-1: sns.ap-southeast-1.amazonaws.com
- ap-southeast-2: sns.ap-southeast-2.amazonaws.com
- ap-northeast-1: sns.ap-northeast-1.amazonaws.com
- us-east-1: sqs.us-east-1.amazonaws.com
- us-west-1: sqs.us-west-1.amazonaws.com
- us-west-2: sqs.us-west-2.amazonaws.com
- sa-east-1: sqs.sa-east-1.amazonaws.com
- eu-west-1: sqs.eu-west-1.amazonaws.com
- ap-southeast-1: sqs.ap-southeast-1.amazonaws.com
- ap-southeast-2: sqs.ap-southeast-2.amazonaws.com
- ap-northeast-1: sqs.ap-northeast-1.amazonaws.com
- : s3.amazonaws.com
- us-west-1: s3-us-west-1.amazonaws.com
- us-west-2: s3-us-west-2.amazonaws.com
- sa-east-1: s3.sa-east-1.amazonaws.com
- eu-west-1: s3-eu-west-1.amazonaws.com
- ap-southeast-1: s3-ap-southeast-1.amazonaws.com
- ap-southeast-2: s3-ap-southeast-2.amazonaws.com
- ap-northeast-1: s3-ap-northeast-1.amazonaws.com
- us-east-1: swf.us-east-1.amazonaws.com
- us-east-1: sdb.amazonaws.com
- us-west-1: sdb.us-west-1.amazonaws.com
- us-west-2: sdb.us-west-2.amazonaws.com
- sa-east-1: sdb.sa-east-1.amazonaws.com
- eu-west-1: sdb.eu-west-1.amazonaws.com
- ap-southeast-1: sdb.ap-southeast-1.amazonaws.com
- ap-southeast-2: sdb.ap-southeast-2.amazonaws.com
- ap-northeast-1: sdb.ap-northeast-1.amazonaws.com
- us-east-1: storagegateway.us-east-1.amazonaws.com
- us-west-1: storagegateway.us-west-1.amazonaws.com
- us-west-2: storagegateway.us-west-2.amazonaws.com
- sa-east-1: storagegateway.sa-east-1.amazonaws.com
- eu-west-1: storagegateway.eu-west-1.amazonaws.com
- ap-southeast-1: storagegateway.ap-southeast-1.amazonaws.com
- ap-southeast-2: storagegateway.ap-southeast-2.amazonaws.com
- ap-northeast-1: storagegateway.ap-northeast-1.amazonaws.com
- us-east-1: ec2.us-east-1.amazonaws.com
- us-west-1: ec2.us-west-1.amazonaws.com
- us-west-2: ec2.us-west-2.amazonaws.com
- sa-east-1: vpc.sa-east-1.amazonaws.com
- eu-west-1: ec2.eu-west-1.amazonaws.com
- ap-southeast-1: ec2.ap-southeast-1.amazonaws.com
- ap-southeast-2: ec2.ap-southeast-2.amazonaws.com
- ap-northeast-1: ec2.ap-northeast-1.amazonaws.com
Friday, December 4, 2009
Creating an EBS-backed AMI from an S3-backed AMI
The recent introduction of Boot From EBS for EC2 opens up a lot of new possibilities. But there are some bootstrapping issues to deal with. There aren't many EBS-backed AMI's available yet and, given the rather complex process involved in porting them, it may take a while for them to show up. This article will walk through the process of converting a popular S3-based AMI to an EBS-backed AMI. I don't guarantee that this is the best process and I certainly wouldn't recommend that anyone use the resulting EBS-backed AMI for anything other than testing and further development, but it puts a stake in the ground regarding a potential process. I'm sure I will hear about the shortcomings and possible improvements!
As a starting point, I'm going to use one of Eric Hammond's excellent Ubuntu AMI's. In particular, I'm going to use:
ami-eef61587 alestic-64/ubuntu-9.04-jaunty-base-64-20091011.manifest.xml
This same basic process should work for other Linux-based AMI's. The first thing I need to do is fire up an new instance of this AMI. In addition, I'm going to create a new 10GB EBS volume that will serve as the prototype for my EBS-based AMI. Here's how I do that using boto:
As a starting point, I'm going to use one of Eric Hammond's excellent Ubuntu AMI's. In particular, I'm going to use:
ami-eef61587 alestic-64/ubuntu-9.04-jaunty-base-64-20091011.manifest.xml
This same basic process should work for other Linux-based AMI's. The first thing I need to do is fire up an new instance of this AMI. In addition, I'm going to create a new 10GB EBS volume that will serve as the prototype for my EBS-based AMI. Here's how I do that using boto:
>>> import boto
>>> c = boto.connect_ec2()
>>> c.get_all_images(['ami-eef61587'])
[Image:ami-eef61587]
>>> img = _[0]
>>> img.run(key_name='cloudright', security_groups=['test1'], instance_type='m1.large')
Reservation:r-0369c96b
>>> inst = _.instances[0]
>>> inst.update()
u'pending'
>>> inst.update()
u'running'
>>> inst.placement
u'us-east-1b'
>>> v = c.create_volume(10, inst.placement)
>>> v.attach(inst.id, '/dev/sdh')
u'attaching'
>>> inst.public_dns_name
u'ec2-67-202-30-28.compute-1.amazonaws.com'
So, at this point I have a new EC2 instance up and running using the S3-based AMI and a new 10GB EBS volume attached to that instance. Now, I need to login to that new instance and do a bit of work.
$ ssh -i ~/.ssh/cloudright root@ec2-67-202-30-28.compute-1.amazonaws.com
...
root@domU-12-31-39-02-31-51:~# apt-get update
...
root@domU-12-31-39-02-31-51:~# apt-get -y upgrade
...
root@domU-12-31-39-02-31-51:~# apt-get -y install cpipe
...
root@domU-12-31-39-02-31-51:~# mkfs -t ext3 /dev/sdh
mke2fs 1.41.4 (27-Jan-2009)
/dev/sdh is entire device, not just one partition!
Proceed anyway? (y,n) y
...
root@domU-12-31-39-02-31-51:~# mkdir /ebs
root@domU-12-31-39-02-31-51:~# mount -t ext3 /dev/sdh /ebs
root@domU-12-31-39-02-31-51:~# tar cpS / | cpipe -vt -b 1024 | gzip -c | tar zxpS -C /ebs
....
root@domU-12-31-39-02-31-51:~# umount /ebs
So, basically I have ssh'ed into the new instance, run apt-get update and apt-get upgrade to install all of the latest patches, formatted the EBS volume as an EXT3 filesystem, mounted that filesystem as /ebs and then copied the entire contents of the current root volume over to the EBS volume. Then I unmount the EBS volume. Now, let's go back to my Python session running on my local machine.
>>> v.detach()
u'detaching'
>>> v.create_snapshot('Initial snapshot for EBS-backed 64-bit Ubuntu 9.04 AMI.')
Snapshot:snap-023ca66b
>>> from boto.ec2.blockdevicemapping import EBSBlockDeviceType, BlockDeviceMapping
>>> ebs = EBSBlockDeviceType()
>>> ebs.snapshot_id = 'snap-023ca66b'
>>> block_map = BlockDeviceMapping()
>>> block_map['/dev/sda1'] = ebs
>>> c.register_image('MG-Ubuntu-9.04-EBS-20091204', 'Testing the creation of EBS-backed Ubuntu AMI.',
architecture='x86_64', kernel_id=img.kernel_id,
ramdisk_id=img.ramdisk_id,
root_device_name='/dev/sda1', block_device_map=block_map)
u'ami-f002e099'
So, here we are detaching the EBS volume and then creating a snapshot of that volume. Then, we need to import some data structures that will allow us to register a new EBS-based image. The first data structure is the EBSBlockDeviceType. There are a number of available fields in that object but the only one we need to worry about is the snapshot_id. This tells EC2 that when someone wants to start up a new instance of our AMI, EC2 needs to start by creating a new EBS volume from this snapshot. The second data structure is the BlockDeviceMapping. It is actually a subclass of a Python dictionary and behaves as you would expect. We need to add an entry that maps the device name of our root volume (in this case /dev/sda1) to the EBS snapshot. Finally, we call register_image to create the new AMI. We pass in a name, a description, the architecture, the kernel and ramdisk (we are just referring to the same ones used by the original S3-backed AMI), the name of our root device (/dev/sda1) and the block device mapping we just created. EC2 returns with a new AMI id and we can then use that to start new EC2 instances. Just to verify, let's start up a new instance based on our EBS-backed AMI:
>>> c.run_instances('ami-f002e099', key_name='cloudright', security_groups=['test1'], instance_type='m1.large')
Reservation:r-f175d599
>>> inst2 = _.instances[0]
>>> inst2.update()
u'pending'
>>> inst2.update()
u'running'
>>> inst2.public_dns_name
u'ec2-75-101-218-5.compute-1.amazonaws.com'
Now let's SSH into our new EBS-based instance and make sure everything is okay.
jobs:~ mitch$ ssh -i ~/.ssh/cloudright.pem root@ec2-75-101-218-5.compute-1.amazonaws.com
...
root@domU-12-31-39-06-E1-62:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 10321208 837672 8959248 9% /
tmpfs 3935948 0 3935948 0% /lib/init/rw
varrun 3935948 40 3935908 1% /var/run
varlock 3935948 0 3935948 0% /var/lock
udev 3935948 72 3935876 1% /dev
tmpfs 3935948 0 3935948 0% /dev/shm
root@domU-12-31-39-06-E1-62:~#
I have made this AMI public and available in the us-east-1 region. Feel free to fire it up and play around with it but be aware that none of the careful testing that accompanies Eric's or Canonical's AMI releases has happened here so it is for illustrative purposes only.
Tuesday, December 1, 2009
API Maturity in Cloud Services
I think one point people often overlook when discussing common cloud API's is the overall maturity of API's for different types of cloud services. Trying to achieve commonality on API's that have not yet reached maturity can be a frustrating and time-consuming effort. So, how stable are the API's for common cloud computing services? This article focuses on a couple of different measures of API maturity to try to answer that question.
API Churn
One measure of API maturity is churn. In other words, how much is the API changing. Mature API's should show relatively small amounts of churn while immature, evolving API's will show increased levels of churn. The chart below shows one measurement of churn, namely the growth in the API. For HTTP-based API's, this can easily be measured by the number of different types of requests supported by the API.
This chart is showing us the number of API requests supported by each of the services at launch compared to the number of API requests currently supported by the services. The EC2 API has grown from 14 requests at launch (08/24/2006) compared to 71 requests today. This includes requests for the CloudWatch, AutoScaling and Elastic Load Balancing services which are essentially part of EC2. Even if you exclude those services, though, the total is now 47 API requests for EC2 today, a 3X growth. In comparison, S3 went from 8 API requests to 13 and SQS actually reduced their API requests from 12 to 11.
During this same time period, EC2 has published 17 API version while SQS has published only 4 and S3 is still running the same API version published at launch, even though some new capabilities have been added.
API Consistency
Another way to measure maturity is to compare similar API's offered by different vendors. Mature API's should show a great deal of similarity among the different API's while immature API's will tend to show many differences. The reasoning behind this is that a mature API should be based on a well-defined set of abstractions and therefore the type and number of API calls dealing with those abstractions should be similar across vendors. In an immature API, the abstractions have not yet been fully fleshed out and you will see more variation among similar API's from different vendors.
First, let's compare Amazon's Simple Queue Service with Microsoft's Azure Queue Service:
This graph is comparing the number of API requests that are common between the different vendors (shown in blue) versus the number of API requests that are unique in a particular vendor's API (shown in red). An API request is considered common if both API's have similar requests with similar actions and side effects. Large blue bars and small red bars indicate more consistent API's whereas large red bars and small blue bars indicate less inconsistent API's. The graph shows a fairly high degree of consistency between the two different queue service API's.
Now let's compare Amazon's Simple Storage Service with Rackspace's CloudFiles and Microsoft's Azure Blob Service:
Again, we see a fairly high degree of consistency across all three API's. The major difference is related to the Azure Blob service's handling of large Blob's. It places a limit of 64 MB on individual pages within a Blob and allows multiple pages to be associated with a particular page. The other services have no equivalent abstraction, hence the differences.
Now let's compare Amazon's EC2 service with Rackspace's CloudServers:
Here we see the opposite of the graphs for the queue and storage services. The large red bars indicate a large number of API requests that are unique to a particular vendor's API and a relatively small number of API requests that are common across both vendors. In fact, aside from the basic operations of listing instances, starting/stopping instances, and listing/creating and deleting available images, it was difficult to even compare the two API's. The handling of IP addresses, networking, firewalls, security credentials, block storage, etc. were unique to each vendor.
So, does that mean that common cloud API's are impossible? Of course not. However, I do believe that achieving meaningful levels of functionality across multiple vendors API's (especially around servers) is building on a shifting and incomplete set of abstractions at this point. Solidifying those abstractions is key to achieving useful interoperability.
Subscribe to:
Posts (Atom)