Sunday, December 5, 2010

S3 MultiPart Upload in boto

Amazon recently introduced MultiPart Upload to S3.  This new feature lets you upload large files in multiple parts rather than in one big chunk.  This provides two main benefits:

  • You can get resumable uploads and don't have to worry about high-stakes uploading of a 5GB file which might fail after 4.9GB.  Instead, you can upload in parts and know that the all of the parts that have successfully uploaded are there patiently waiting for the rest of the bytes to make it to S3.
  • You can parallelize your upload operation.  So, not only can you break your 5GB file into 1000 5MB chunks, you can run 20 uploader processes and get much better overall throughput to S3.
It took a few weeks but we have just added full support for MultiPart Upload to the boto library.  This post gives a very quick intro to the new functionality to help get you started.

Below is a transcript from an interactive IPython session that exercises the new features.  Below that is a line by line commentary of what's going on.



  1. Self-explanatory, I hope 8^)
  2. We create a connection to the S3 service and assign it to the variable c.
  3. We lookup an existing bucket in S3 and assign that to the variable b.
  4. We initiate a MultiPart Upload to bucket b.  We pass in the key_name.  This key_name will be the name of the object in S3 once all of the parts are uploaded.  This creates a new instance of a MultiPartUpload object and assigns it to the variable mp.
  5. You might want to do a bit of exploration of the new object.  In particular, it has an attribute called id which is the upload transaction ID assigned by S3.  This transaction ID must accompany all subsequent requests related to this MultiPart Upload.
  6. I open a local file.  In this case, I had a 17MB PDF file.  I split that into 5MB chunks using the split command ("split -b5m test.pdf").  This creates 3 5MB chunks and one smaller chunk with the leftovers.  You can use larger chunk sizes if you want but 5MB is the minimum size (except for the last, of course).
  7. I upload this chunk to S3 using the upload_part_from_file method of the MultiPartUpload object.
  8. Close the filepointer
  9. Open the file for the second chunk.
  10. Upload it.
  11. Close it.
  12. Open the file for the third chunk.
  13. Upload it.
  14. Close it.
  15. Open the file for the fourth and final chunk (the small one).
  16. Upload it.
  17. Close it.
  18. I can now examine all of the parts that are currently uploaded to S3 related to this key_name.  As you can see, I can use the MultiPartUpload object as an iterator and, when so doing, the generator object handles any pagination of results from S3 automatically.  Each object in the list is an instance of the Part class and, as you can see, have attributes such as part_number, size, etag.
  19. Now that the last part has been uploaded I can complete the MultiPart Upload transaction by calling the complete_upload method of the MultiPartUpload object.  If, on the other hand, I wanted to cancel the operation I could call cancel_upload and all of the parts that had been uploaded would be deleted in S3.
This provides a simple example.  However, to really benefit fully from the MultiPart Upload functionality, you should consider trying to introduce some concurrency into the mix.  Either fire off separate threads or subprocesses to upload different parts in parallel.  The actual order the parts are uploaded doesn't matter as long as they are numbered sequentially.

Update

To find all of the current MultiPart Upload transactions for a given bucket, you can do this:

Wednesday, September 15, 2010

Using Identity & Access Management (IAM) Service in boto

The recently announced Identity and Access Management service from AWS provides a whole bunch of useful and long-requested functionality.  The boto library provides full support for IAM and this article provides a quick intro to some basic IAM capabilities and how to access them via boto.

IAM introduces a new concept to AWS; users.  Prior to IAM, you created an account and that account had the necessary credentials for accessing various AWS services (via the access key and secret key or X.509 cert associated with your account) and also acted as a billing entity (you get a bill from AWS).   Conflating these two concepts causes problems, especially if you want to use AWS within businesses and enterprises.  In those environments, the people who use the services and the people who manage and pay for the services are very distinct.  With IAM, AWS has introduced the notion of a user who has the necessary credentials to use AWS services but accounting and billing are handled by the controlling AWS account.  This distinction between accounts and users is actually fundamental to IAM and important to understand.

Based on that description, it's clear that IAM can be used as a user provisioning system for AWS.  Using the API, you can provision new AWS users, create credentials for the user (both for the AWS Console web site as well as for the API), create X.509 certs with the user or associate existing certs and even manage the Multi-Factor Authentication (MFA) devices associated with the user.  In addition, you can create groups, add and remove users from those groups and associate policies with groups to control which services and resources member of a group have access to.  And all of the users are created under the control of a single master account which ultimately owns all resources created by all users and gets the AWS bill for all users in one monthly statement.

So, clearly if you are a business (large or small) and want to automate the process of user management and have visibility into the resources and costs across your entire organization, IAM is great.  But, even if you are an individual developer, IAM provides some important features that have been conspicuously absent from AWS up till now.

If you read my previous posts about managing your AWS credentials (part1 and part2) you will probably remember some of the hoops we had to jump through to find a way to safely manage AWS credentials on EC2 instances.  And even with all of that hoop-jumping, we couldn't really come up with a perfect solution.  But with IAM's ability to create users with very limited capabilities, we finally have an elegant way to solve the problem.

I'm going to show a few code examples that illustrate how to accomplish some simple but useful things in IAM using boto.  Before we delve into those examples, though, I want to talk a little bit about the iam module in boto because it uses a different approach than other boto modules.  Depending on the reaction, this approach may be expanded to other modules in the future.

Using boto, you make requests to services and they send responses back to you.  For AWS, the responses are XML documents that contain the information you requested.  The standard approach to handling these responses in boto has been to write a small Python class for each possible response type.  The class is then responsible for parsing the XML and extracting the pertinent values and storing them as attributes on the Python object.  Users then interact with the Python objects and never see the XML.  This approach works well but the downside is that it requires a lot of small, hand-coded Python objects to be written which takes time.

For the iam module, I wrote a generic response handler that parses the XML and turns it into native Python data structure.  So, if the following XML is returned from the service:



The the generic response parser will return the following Python data structure:



As you can see, the Python data structure is deeply nested.  To make it easier to get to the stuff you want, I've added a little magic to allow you to directly access any key, regardless of the depth, by simply accessing it as an attribute.  So, if you did something like this:



I'd love feedback on this approach.  Feel free to comment to this post or post to the boto users Google group.  Now, on to the examples.

Create An Admin Group

This example shows how to create a group that is authorized to access all actions supported by IAM.  This would allow you to defer user/group management to another person or group of people.



Create a Group for EC2 / S3 Users

This example shows how to create a group and user that has full access to all EC2 functionality and S3 functionality but nothing else.



Create a Group for Read Only Access to SimpleDB Domain

This example illustrates how you can use IAM to solve some of those credential problems we discussed earlier.  Assume that you have a SimpleDB domain that contains important information needed by an application running on EC2 instances.  To query the domain, you need to have AWS credentials on the EC2 instances but you really don't want to put your main AWS credentials on there because a bad guy could do all kinds of damage with those credentials.  IAM, to the rescue!  We can create a group that has read-only access to the specific domain it needs to access and is authorized to use only the GetAttribute and Select requests from SimpleDB.  Even if a bad guy gets those credentials, they really can't do any damage.  Here's how to set that up in IAM.

Friday, July 2, 2010

And Now For Something Completely Different...

As some of you may know, I've spent the past three years or so focused on AWS-related consulting through my own little company, CloudRight.  It's been fun and exciting and I feel that I've really had a front row seat for the amazing growth and excitement around cloud computing.  But consulting has it's downsides, too.  After a while the pace of new projects started to lose it's lustre and I found myself pining for the fjords, or at least for a bit more focus in my professional life.

So, I'm excited to say that I have joined the development team at Eucalyptus.  I like their technology, I like their positioning in the marketplace, I like their commitment to open source but mainly I just really like the team.  Everyone there is not only great at what they do, they are also great people and in my experience that's the recipe for a great company.   I'm absolutely thrilled to be a part of it.

My main focus at Eucalyptus will be in the area of tools.  Basically trying to make sure that all of the capabilities of the core system are easily and consistently accessible to users and administrators.  The current Euca2ools command line utilities are a great start but we all feel there is an opportunity to do a lot more.

This is also great news for boto.  Euca2ools are built on top of boto so, for the first time, boto will actually be a part of my day job rather than something I try to squeeze in between gigs and after hours.  That should mean more frequent and consistent releases and better quality overall.

And now, it's time for the traditional "new job" fish slapping dance...

Sunday, June 13, 2010

Using Reduced Redundancy Storage (RRS) in S3

This is just a quick blog post to provide a few examples of using the new Reduced Redundancy Storage (RRS) feature of S3 in boto.  This new storage class in S3 gives you the option to tradeoff redundancy for cost.  The normal S3 service (and corresponding pricing) is based on a 12-nines 11 nines (yes, that's 99.999999999% - Thanks to Jeff Barr for correction in comments below) level of durability.  In order to achieve this extremely highly level of reliability, the S3 service must incorporate a high-level of redundancy.  In other words, it keeps many copies of your data in many different locations so that even if multiple locations encounter failures, your data will still be safe.

That's a great feature but not everyone needs that level of redundancy.  If you already have copies of your data locally and are just using S3 as a convenient place to store data that is actively being accessed by services within the AWS infrastructure, RRS may be for you.  It provides a much lower level of durability (99.99%) at a significantly lower cost.  If that fits the bill for you, the next three code snippets will provide you with the basics you need to start using RRS in boto.

Create a New S3 Key Using the RRS Storage Class


Convert An Existing S3 Key from Standard Storage Class to RRS


Create a Copy of an Existing S3 Key Using RRS

Friday, June 4, 2010

AWS By The Numbers

I recently gave a short talk about Amazon Web Services at GlueCon 2010.  It was part of a panel discussion called "Major Platform Providers" and included similar short talks from others about Azure, Force.com and vCloud.  It's very hard (i.e. impossible) to give a meaningful technical overview of AWS in 10 minutes so I struggled a bit trying to decide what to talk about.  In the end, I decided to try to come up with some quantitative data to describe Amazon Web Services.  My goal was to try to show that AWS is:

  • A first mover - AWS introduced their first web services in 2005
  • A broad offering - 13 services currently available
  • Popular - details of how I measure that described below
  • Prolific - the pace of innovation from AWS is impressive

After the conference, I was going to post my slides but I realized they didn't really work that well on their own so I decided instead to turn the slides into a blog post.  That gives me the opportunity to explain the data and resulting graphs in more detail and also allows me to provide the graphs in a more interactive form.

Data?  What data?

The first challenge in trying to do a data-heavy talk about AWS is actually finding some data.  Most of the data that I would really like to have (e.g. # users,  # requests, etc.) is not available.  So, I needed to find some publicly available data that could provide some useful insight.  Here's what I came up with:

  • Forum data - I scraped the AWS developer forums and grabbed lots of useful info.  I use things like forum views, number of messages and threads, etc. to act as a proxy for service popularity.  It's not perfect by any means, but it's the best I could come up with.
  • AWS press releases - I analyzed press releases from 2005 to the present day and use that to populate a spreadsheet of significant service and feature releases.
  • API WSDL's - I parsed the WSDL for each of the services to gather data about API complexity.
With that background, let's get on to the data.

Service Introduction and Popularity

This first graph uses data scraped from the forums.  Each line in the graph represents one service and the Y axis is the total number of messages in that services forum for the given month.  The idea is that the volume of messages on a forum should have some relationship to the number of people using the service and, therefore, the popularity of the service.  Following the timeline across also shows the date of introduction for each of the services.

Note: If you have trouble loading the following graph, try going directly to the Google Docs spreadsheet which I have shared.




The following graph shows another, simpler view of the forum data.  This view plots the average number of views on the forum for each service normalized.



API Complexity

Another piece of publicly available data for AWS is the WSDL for each service.  The WSDL is an XML document that describes the operations supported by the service and the data types used by the operations.  The following graph shows the API Complexity (measured as the number of operations) for each of the services.



Velocity

Finally, I wanted to try to measure the pace of innovation by AWS.  To do this, I used the spreadsheet I created that tracked all significant service and feature announcements by AWS.  I then counted the number of events per quarter for AWS and used that to compute an agile-style velocity.



Summary

Hopefully these graphs are interesting and help to prove the points that I outlined at the beginning of the talk.  I actually have a lot more data available from the forum scrapping and may try to mine that in different ways later.

While this data was all about AWS, I think the bigger point is that the level of interest and innovation in Amazon's services is really just an indicator of a trend across the cloud computing market.

Sunday, May 23, 2010

Boto and Google Storage

You probably noticed, in the blitz of announcements from the recent I/O conference that Google now has a storage service very similar to Amazon's S3 service.  The Google Storage (GS) service provides a REST API that is compatible with many existing tools and libraries.

In addition to the API, Google also announced some tools to make it easier for people to get started using the Google Storage service.  The main tool is called gsutil and it provides a command line interface to both Google Storage and S3.  It allows you to reference files in GS or S3 or even on your file system using URL-style identifiers.  You can then use these identifiers to copy content to/from the storage services and your local file system, between locations within a storage service or even between the services.  Cool!

What was even cooler to me personally was that gsutil leverages boto for API-level communication with S3 and GS.  In addition, Google engineers have extended boto with a higher-level abstraction of storage services that implements the URL-style identifiers.  The command line tools are then built on top of this layer.

As an open source developer, it is very satisfying when other developers use your code to do something interesting and this is certainly no exception.  In addition, I want to thank Mike Schwartz from Google for reaching out to me prior to the Google Storage session and giving me a heads up on what they were going to announce.  Since that time Mike and I have been collaborating to try to figure out the best way to support the use of boto in the Google Storage utilities.  For example, the storage abstraction layer developed by Google to extend boto is generally useful and could be extended to other storage services.

In summary, I view this as a very positive step in the boto project.  I look forward to working with Google to make boto more useful for them and for the community of boto users.  And as always, feedback from the boto community is not only welcome but essential.

Tuesday, April 20, 2010

Failure as a Feature

One need only peruse the EC2 forums a bit to realize that EC2 instances fail.  Shock.  Horror.  Servers failing?  What kind of crappy service is this, anyway.  The truth, of course, is that all servers can and eventually will fail.  EC2 instances, Rackspace CloudServers, GoGrid servers, Terremark virtual machines, even that trusty Sun box sitting in your colo.  They all can fail and therefore they all will fail eventually.

What's wonderful and transformative about running your applications in public clouds like EC2 and CloudServers, etc. is not that the servers never fail but that when they do fail you can actually do something about it.  Quickly.  And programmatically.  From an operations point of view, the killer feature of the cloud is the API.  Using the API's, I can not only detect that there is a problem with a server but I can actually correct it.  As easily as I can start a server, I can stop one and replace it with a new one.

Now, to do this effectively I really need to think about my application and my deployment differently.  When you have physical servers in a colo failure of a server is, well, failure.  It's something to be dreaded.  Something that you worry about.  Something that usually requires money and trips to the data center to fix.

But for apps deployed on the cloud, failure is a feature.  Seriously.  Knowing that any server can fail at any time and knowing that I can detect that and correct that programmatically actually allows me to design better apps.  More reliable apps.  More resilient and robust apps.  Apps that are designed to keep running with nary a blip when an individual server goes belly up.

Trust me.  Failure is a feature.  Embrace it.  If you don't understand that, you don't understand the cloud.