Wednesday, September 15, 2010

Using Identity & Access Management (IAM) Service in boto

The recently announced Identity and Access Management service from AWS provides a whole bunch of useful and long-requested functionality.  The boto library provides full support for IAM and this article provides a quick intro to some basic IAM capabilities and how to access them via boto.

IAM introduces a new concept to AWS; users.  Prior to IAM, you created an account and that account had the necessary credentials for accessing various AWS services (via the access key and secret key or X.509 cert associated with your account) and also acted as a billing entity (you get a bill from AWS).   Conflating these two concepts causes problems, especially if you want to use AWS within businesses and enterprises.  In those environments, the people who use the services and the people who manage and pay for the services are very distinct.  With IAM, AWS has introduced the notion of a user who has the necessary credentials to use AWS services but accounting and billing are handled by the controlling AWS account.  This distinction between accounts and users is actually fundamental to IAM and important to understand.

Based on that description, it's clear that IAM can be used as a user provisioning system for AWS.  Using the API, you can provision new AWS users, create credentials for the user (both for the AWS Console web site as well as for the API), create X.509 certs with the user or associate existing certs and even manage the Multi-Factor Authentication (MFA) devices associated with the user.  In addition, you can create groups, add and remove users from those groups and associate policies with groups to control which services and resources member of a group have access to.  And all of the users are created under the control of a single master account which ultimately owns all resources created by all users and gets the AWS bill for all users in one monthly statement.

So, clearly if you are a business (large or small) and want to automate the process of user management and have visibility into the resources and costs across your entire organization, IAM is great.  But, even if you are an individual developer, IAM provides some important features that have been conspicuously absent from AWS up till now.

If you read my previous posts about managing your AWS credentials (part1 and part2) you will probably remember some of the hoops we had to jump through to find a way to safely manage AWS credentials on EC2 instances.  And even with all of that hoop-jumping, we couldn't really come up with a perfect solution.  But with IAM's ability to create users with very limited capabilities, we finally have an elegant way to solve the problem.

I'm going to show a few code examples that illustrate how to accomplish some simple but useful things in IAM using boto.  Before we delve into those examples, though, I want to talk a little bit about the iam module in boto because it uses a different approach than other boto modules.  Depending on the reaction, this approach may be expanded to other modules in the future.

Using boto, you make requests to services and they send responses back to you.  For AWS, the responses are XML documents that contain the information you requested.  The standard approach to handling these responses in boto has been to write a small Python class for each possible response type.  The class is then responsible for parsing the XML and extracting the pertinent values and storing them as attributes on the Python object.  Users then interact with the Python objects and never see the XML.  This approach works well but the downside is that it requires a lot of small, hand-coded Python objects to be written which takes time.

For the iam module, I wrote a generic response handler that parses the XML and turns it into native Python data structure.  So, if the following XML is returned from the service:

The the generic response parser will return the following Python data structure:

As you can see, the Python data structure is deeply nested.  To make it easier to get to the stuff you want, I've added a little magic to allow you to directly access any key, regardless of the depth, by simply accessing it as an attribute.  So, if you did something like this:

I'd love feedback on this approach.  Feel free to comment to this post or post to the boto users Google group.  Now, on to the examples.

Create An Admin Group

This example shows how to create a group that is authorized to access all actions supported by IAM.  This would allow you to defer user/group management to another person or group of people.

Create a Group for EC2 / S3 Users

This example shows how to create a group and user that has full access to all EC2 functionality and S3 functionality but nothing else.

Create a Group for Read Only Access to SimpleDB Domain

This example illustrates how you can use IAM to solve some of those credential problems we discussed earlier.  Assume that you have a SimpleDB domain that contains important information needed by an application running on EC2 instances.  To query the domain, you need to have AWS credentials on the EC2 instances but you really don't want to put your main AWS credentials on there because a bad guy could do all kinds of damage with those credentials.  IAM, to the rescue!  We can create a group that has read-only access to the specific domain it needs to access and is authorized to use only the GetAttribute and Select requests from SimpleDB.  Even if a bad guy gets those credentials, they really can't do any damage.  Here's how to set that up in IAM.


  1. It is very good news that boto supports iam so early. Tks.

    BTW a use case scenario is delegating accesss to a specific bucket to a user created with iam. In such case , how the json would be. Can you tell us an example on that? Tks in advance Mitch.

  2. I'll try to create a few more gists on github demonstrating other examples. I'll add a link in the comments when I've done that.

  3. Mitch,

    I have this snippet to grant full access tu iam users to an specific bucket.

  4. "pip install boto" gives a boto 2.02b that isn't...the 2.02b that has IAM. So I grabbed the one from github, and I got a lot of debugging info just by doing a simple boto.connect_iam(). Did you add IAM without changing the minor version info? Curious why pip is confused.

  5. I haven't created a new release incorporating IAM, it's only in the github repo right now. I will create a 2.0b3 release sometime today.

    As to the debug info, I inadvertently checked in a version of the iam module with debugging enabled. I have corrected that. Thanks.

  6. Mitch - get_all_access_keys doesn't appear to be responding with any list (it's a heavily nested dict, no lists). The upshot of this is that if I give an account 2 keys, only 1 is listed (always the most recent created).

  7. Okay, let me take a look at that. Probably just a bug on my part.

  8. According to the API documentation:

    The ListAccessKeys request does not return a list of AccessKeys. It returns only a single access key. I think this is worth a post to the AWS forums.

  9. I was wrong about that. Please see:

    for details. This should be fixed now in github master.

  10. Mitch: tried to email you after finding a couple other things, but it bounced. I'm not sure what the rest of Batok's code was, but I found if I tried to give full access to only *one* S3 bucket, and nothing else, I couldn't do a simple .get_bucket() on the bucket with the account. Turns out this is an issue with a lot of tools, and the workaround is to add another policy with "Action":["s3:ListAllMyBuckets"] - but, I don't want the account to be able to list all buckets. I stole this fix from

    The cloudberry rep suggested it was a problem with their tool...then, they fixed it. I looked and couldn't really find why Bucket.get_key/Bucket.get_all_keys caused this; when I did a get_bucket(bucket,validate=False) I still got an access denied when trying to do some things, but I'm able to copy files to the bucket with that account. I can't though list the keys (which I can create...).

  11. addendum: if I set public-read on a bucket, things work. For now, everything I need this for is ok with public-read...though, in a phase coming up, that won't be the case. Maybe I'll spend more than a few minutes and see if I can find what in the boto code causes this and offer a patch myself, if you haven't already by then.

  12. Mitch - what's the chance, regarding issue #465 I mention above and opened, that boto could respond with an empty list instead of an error? I understand it's probably just because amazon is responsible - because they're not giving you a list to respond with. However, should boto catch this particular exception and respond with an empty list? Or is it an issue I should bring up with Amazon directly, so that they don't give an error in this situation (which would keep you from passing the error in boto)?

  13. I'll try to take a look at that tonight or tomorrow. Been kind of busy but it does sound like I'm doing something wrong and I'd like to get it taken care of. I'll update the issue with any progress.