Showing posts with label S3. Show all posts
Showing posts with label S3. Show all posts

Tuesday, February 22, 2011

Accessing the Internet Archive with boto

A recent tweet from Pete Skomoroch twigged me to the fact that the Internet Archive provides an S3-like API.   Cool!  The Internet Archive is a great resource which provides, in their words:
...a digital library of Internet sites and other cultural artifacts in digital form. Like a paper library, we provide free access to researchers, historians, scholars, and the general public.
Since boto supports S3 I wondered if it would be possible to access the Internet Archive's API with boto.  Turns out, it's quite simple.  To make it even simpler, I've added a new "connect_ia" method.  Before you can use this, you need to get API credentials from the Internet Archive but fortunately that's really easy.  Just sign up for an account (if you don't already have one) and then go to this link to generate the API keys.

Once you have your credentials, the easiest thing to do is to add the credentials to your boto config file.  They need to go in the Credentials section like this:



Then, you can create a connection to the Internet Archive like this:



I've only tested this a bit so if you run into any problems with it, post a message to boto-users or create an issue.

Sunday, May 23, 2010

Boto and Google Storage

You probably noticed, in the blitz of announcements from the recent I/O conference that Google now has a storage service very similar to Amazon's S3 service.  The Google Storage (GS) service provides a REST API that is compatible with many existing tools and libraries.

In addition to the API, Google also announced some tools to make it easier for people to get started using the Google Storage service.  The main tool is called gsutil and it provides a command line interface to both Google Storage and S3.  It allows you to reference files in GS or S3 or even on your file system using URL-style identifiers.  You can then use these identifiers to copy content to/from the storage services and your local file system, between locations within a storage service or even between the services.  Cool!

What was even cooler to me personally was that gsutil leverages boto for API-level communication with S3 and GS.  In addition, Google engineers have extended boto with a higher-level abstraction of storage services that implements the URL-style identifiers.  The command line tools are then built on top of this layer.

As an open source developer, it is very satisfying when other developers use your code to do something interesting and this is certainly no exception.  In addition, I want to thank Mike Schwartz from Google for reaching out to me prior to the Google Storage session and giving me a heads up on what they were going to announce.  Since that time Mike and I have been collaborating to try to figure out the best way to support the use of boto in the Google Storage utilities.  For example, the storage abstraction layer developed by Google to extend boto is generally useful and could be extended to other storage services.

In summary, I view this as a very positive step in the boto project.  I look forward to working with Google to make boto more useful for them and for the community of boto users.  And as always, feedback from the boto community is not only welcome but essential.

Tuesday, February 9, 2010

Using S3 Versioning and MFA to CMA*

* - CMA = Cover My Ass


Amazon's Simple Storage Service (S3) is a great way to safely store loads of data in the cloud.  It's highly available, simple to use and provides good data durability by automatically copying your data across multiple regions and/or zones.  With over 80 billion objects stored (at last published count) I'm clearly not alone in thinking it's a good thing.

The only problem I've had with S3 over the years is the queazy feeling I get when I think about some nefarious individual getting hold of my AWS AccessKey/SecretKey.  Since all S3 capabilities are accessed via a REST API and since that credential pair is used to authenticate all requests with S3, a bad guy/girl with my credentials (or a temporarily stupid version of me) could potentially delete all of the content I have stored in S3.  That represents the "Worst Case Scenario" of S3 usage and I've spent a considerable amount of time and effort trying to find ways to mitigate this risk.

Using multiple AWS accounts can help.  The Import/Export feature is another way to mitigate your exposure.  But what I've always wanted was a WORM (Write Once Read Many) bucket.  Well, not always, but at least since May 6, 2007.  That would give me confidence that the data I store in S3 could not be accidentally or maliciously deleted.  This kind of feature would also provide some interesting functionality for certain types of compliance and regulatory solutions.

Starting today, AWS has released a couple of really useful new features in S3: Versioning and MFADelete.  Together, these features provide just about everything I wanted when I asked for a WORM bucket.  So, how do they work?

Versioning

Versioning allows you to have multiple copies of the same object. Each version has a unique version ID and the versions are kept in ascending order by the date the version was created. Each bucket can be configured to either enable or disable versioning (only by the bucket owner) and the basic behavior is shown below in the table. The behavior of a Versioned bucket differs based on whether it is being accessed by a Version-Aware (VA) client or NonVersion-Aware (NVA) client.

Operation Unversioned Bucket Versioned Bucket - NVA Client Versioned Bucket - VA Client
GET Retrieves the object or a 404 if the object is not found Retrieves the latest version or a 404 if a Delete Marker is found Retrieves the version specified by provided version ID
PUT Stores the content in the bucket, overwriting any existing content Stores content as new version Stores content as new version
DELETE Irrevocably deletes the content Stores a DeleteMarker as latest version of object. Permanently deletes version specified by provided version ID

The above table is just a summary. You should see the S3 documentation for full details but even this summary clearly shows the benefits of versioning. If I enable versioning on a bucket, the chance of accidentally deleting content is greatly reduced. I would have to be using a version-aware delete tool and explicitly referencing individual version ID's to permanently delete them.

So, accidental deletion of content is less of a risk with versioning but how about the other risk? If a bad guy/girl gets my AccessKey/SecretKey, they can still delete all of my content as long as they know how to use the versioning feature of S3. To address this threat, S3 has implemented a new feature called MFADelete.

MFADelete

MFADelete uses the Multi-Factor Authentication device you are already using to protect AWS Portal and Console access.  What?  You aren't using the MFA device?  Well, you should go sign up for one right now.  It's well worth the money, especially if you are storing important content in S3.

Like Versioning, MFADelete can be enabled on a bucket-by-bucket basis and only by the owner of the bucket.  But, rather than just trusting that the person with the AccessKey/SecretKey is the owner, MFADelete uses the MFA device to provide an additional factor of authentication.  To enable MFADelete, you send a special PUT request to S3 with an XML body that looks like this:


<?xml version="1.0" encoding="UTF-8"?>

  <VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">

    <Status>Enabled</Status>

    <MfaDelete>Enabled</MfaDelete>

</VersioningConfiguration>


In addition to this XML body, you also need to send a special HTTP header in the request, like this:



x-amz-mfa: <serial number of MFA device> <token from MFA device>


Once this request has been sent, all delete operations on the bucket and all requests to change the MFADelete status for the bucket will also require the special HTTP header with the MFA information.  So, that means that even if the bad guy/girl gets your AccessKey/SecretKey combo they still won't be able to delete anything from your MFADelete-enabled bucket without the MFA device, as well.

It's not exactly the WORM bucket I was originally hoping for but it's a huge improvement and greatly reduces the risk of accidental or malicious deletion of data from S3.  I got my pony!

The code in the boto subversion repo has already been updated to work with the new Versioning and MFADelete features.  A new release will be out in the near future.  I have included a link below to a unit test script that shows most of the basic operations and should give you a good start on incorporating these great new features into your application.  The script prompts for you for the serial number of your MFA device once and then prompts for a new MFA code each time on is required.  You can only perform one operation with each code so you will have to wait for the device to cycle to the next code between each operation.

Example Code