Friday, October 30, 2009

Using RDS in Boto

Initial support for RDS has just been added to boto.  The code currently lives in the subversion trunk but a new boto release will be out very soon that will also include the new RDS module.  To get things started, I'll give a short tutorial on using RDS.

The first thing we need to do is create a connection to the RDS service.  This is done in the same way all other service connections are created in boto:


>>> import boto
>>> rds = boto.connect_rds()

Ultimately, we want to create a new DBInstance, basically an EC2 instance that has been pre-configured to run MySQL.  Before we can do that, we need to create a couple of things that are required when creating a new DBInstance.  First, we will need a DBSecurityGroup.  This is very similar to the SecurityGroup used in EC2 but it's considerably more simple because it is focused on only one type of application, MySQL.  Within a DBSecurityGroup I can authorize access either by a CIDR block or by specifying an existing EC2 SecurityGroup.  Since I'm going to be accessing my DBInstance from an EC2 instance, I'm just going to authorize the EC2 SecurityGroup that my instance is running in.  Let's assume it's the group "default":


>>> sg = rds.create_dbsecurity_group('group1', 'My first DB Security group') 
>>> ec2 = boto.connect_ec2()
>>> my_ec2_group = ec2.get_all_security_groups(['default'])[0]
>>> sg.authorize(ec2_group=my_ec2_group)

 Now that we have a DBSecurityGroup created, we now need a DBParameterGroup.  The DBParameterGroup is what's used to manage all of the configuration settings you would normally have in your MySQL config file.  Because you don't have direct access to your DBInstance (unlike a normal EC2 instance) you need to use the DBParameterGroup to retrieve and modify the configuration settings for your DBInstance.  Let's create a new one:


>>>pg = rds.create_parameter_group('paramgrp1', description='My first param group.')

 The ParameterGroup object in boto subclasses dict, so it behaves just like a normal mapping type.  Each key in the ParameterGroup is the name of a config entry and it's value is a Parameter object.  Let's explore one of the Parameters in the ParameterGroup.  Because the set of parameters is quite large, RDS doesn't send all of the default parameter settings to you when you create a new ParameterGroup.  To fetch them from RDS, we need to call get_params:


>>> pg.get_params()
>>> pg.keys()
[u'default_week_format',
 u'lc_time_names',
 u'innodb_autoinc_lock_mode',
 u'collation_server',
<...>
  u'key_buffer_size',
 u'key_cache_block_size',
 u'log-bin']
>>> param = pg['max_allowed_packet']
>>> param.name
u'max_allowed_packet'
>>> param.type
u'integer'
>>> param.allowed_values
u'1024-1073741824'
>>> param.value = -5
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

ValueError: range is 1024-1073741824
>>> param.value = 2048
>>> param.apply()

Because the Parameters have information about the type of the data and allowable ranges, we can do a pretty good job of validating values before sending them back to RDS with the apply method.

Now that we have a DBSecurityGroup and DBParameterGroup created, we can create our DBInstance.


>>> inst = rds.create_dbinstance(id='dbinst1', allocated_storage=10,
instance_class='db.m1.small', master_username='mitch',
master_password='topsecret', param_group='paramgrp1',
security_group='group1')

At this point, RDS will start the process of bringing up a new MySQL instance based on my specifications.  There are lots of other parameters available to tweak.  In addition, you can do things like set the preferred maintenance window and when you would prefer to have snapshots run.  To check on the status of our instance, we can do the following:


>>> rs = rds.get_all_dbinstances()
>>> rs
[DBInstance:dbinst1]
>>> inst = rs[0]
>>> inst.status
>>> u'available'
>>> inst.endpoint
>>> (u'dbinst1.c07mrl4pthxk.us-east-1.rds.amazonaws.com', 3306)

So, at this point our new DBInstance is up and running and we have the endpoint and port number we need to connect to it.  One of the nice things about RDS is that once the instance is running, I can use RDS to perform a lot of the management tasks associated with the server.  I can do snapshots of the server at any time, or I can automate that process.  I can change any of the parameters associated with the server and decide whether I want those changes to take place immediately or to wait until the next maintenance window.  I can also use the modify_dbinstance method to tell RDS to increase the allocated storage on my server or even move my instance up to a larger instance class.

The current RDS code is checked in.  It's still beta quality but we will be releasing a 1.9 version of boto early next week which will include this code as well as support for VPC and a ton of bug fixes.  So, if you get a chance, give the boto RDS module a try and let us know what you think.

18 comments:

  1. How would you do this with a CIDR group?

    ReplyDelete
  2. It would be something like this:

    sg.authorize(cidr_ip='0.0.0.0/0')

    Obviously, you would probably want to limit the CIDR block a bit more than that.

    ReplyDelete
  3. One more questions. Is the allocated_storage in Gb? Just trying to setup an rds to see what we can do with it and our applications.

    ReplyDelete
  4. thanks for the help with the cidr format.

    ReplyDelete
  5. Yes, it's in GB. Valid values are from 5-1024.

    ReplyDelete
  6. I tried to connect to the new rds but it says comes back with the following message:

    MySQL said: Can't connect to MySQL server on 'dbdev.crwrstqvz17c.us-east-1.rds.amazonaws.com'

    I use the CIDR 9#.2##.2##.1##/32 to allow my ip to test out accces to the new rds database. It says that it is ready but I can't connect. Am I missing something? I used the master user I setup as the user and it's password but no luck. Also how do I use the ModifyDBInstance of the API in boto? I will want to add another security group to the db once I'm done testing from my machine here. Thanks again for a great python product to help do this.

    ReplyDelete
  7. Can you telnet to that host/port? That would tell us whether it's a connectivity problem or a MySQL problem.

    ReplyDelete
  8. Nope. It resolves to an ip address but it times out.

    connect to address ###.###.###.###: Operation timed out

    Any thoughts? Thanks again

    ReplyDelete
  9. Can you confirm that the DBInstance is actually in the DBSecurityGroup that you created? Try doing a get_all_dbinstances() and then checking the security_group attribute.

    ReplyDelete
  10. Hi Mitch,
    It looks like it is set to default instead of the one I created. How do I modify this? I tried:
    inst2 = rds.modify_dbinstance(id='dbdev', security_group='linkage') but I get a global name 'backup_retention_period' is not defined error

    ReplyDelete
  11. Sounds like a boto issue. I'm wondering if there is a problem in setting the sec group in the first place, too. Let me try to reproduce here and fix if necessary.

    ReplyDelete
  12. Just checked in a fix to both problems (they were related). Thanks for pointing it out.

    Mitch

    ReplyDelete
  13. Thanks Mitch. I just tried it and I get the following error when I try to modify the instance.
    I created a new connection and then tried:

    inst2 = rds.modify_dbinstance(id='dbdev', security_group='linkage')

    TypeError: modify_dbinstance() got an unexpected keyword argument 'security_group'

    Is this not how I would change the security group?

    Thanks again,

    Dean

    ReplyDelete
  14. You need to pass in a list of security groups or security group names. That was the change I made, The docs were a little unclear. So, try:

    inst2 = rds.modify_dbinstance(id='dbdev', security_group=['linkage'])

    Mitch

    ReplyDelete
  15. Oh, and it's security_groups, not security_group.

    ReplyDelete
  16. Hi Mitch how do you remove an rds?
    I tried:

    rds.delete_dbinstance('dbdev',True,'')

    but it still shows up in the list.

    Thanks again,

    Dean

    ReplyDelete
  17. Oops. I found where I made the mistake. I didn't refresh my get_all_dbinstances

    ReplyDelete