I have just uploaded a new version of boto to the downloads section at http://boto.googlecode.com/. Version 1.9a is a significant and long overdue release that includes, among other things:
- Support for Virtual Private Cloud (VPC)
- Support for Relational Data Service (RDS)
- Support for Shared EBS Snapshots
- Support for Boot From EBS
- Support for Spot Instances
- CloudFront private and streaming Distributions
- Use of POST in data-heavy requests in ec2 and sdb modules
- Support for new us-west-1 region
- Fixes for more than 25 issues
Mitch
Nice work, Mitch. I updated the recent changes from the repository and it fixed a minor bug I was tracking down.
ReplyDeleteHi Mitch just updated to 1.9b and am having problems trying to put a serialized protocol buffer string into simpledb. I get the following error:
ReplyDeleteraise self.ResponseError(response.status, response.reason, body)
boto.exception.SDBResponseError: 403 Forbidden
< Code >SignatureDoesNotMatchThe request signature we calculated does not math the signature you provided....
item = sdb.new_item(item.name)
item["test"] = "some text"
item["nextval"] = "More text"
vlog = vlog_pb2.Vlog()
vlog.test = "some text"
vlog.nextval = "More text"
vStr = vlog.SerializeToString()
item.["pb"] = vStr
item.save()
I get this when I do an item.save()
I can send you more code or the serilized string output if you need me to. I get an error if I do a unicode(vStr,"utf-8") if this helps:
UnicodeDecodeError: 'utf8' can't decode byte 0xa6 in position 89: unexpected code byte
Boto is great for a lot of what we use AWS for. Quick question though in 1.9b do selects return the whole record set or just up to 100? I thought boto handled the pagination of the larger queries. Or is there something more that I need to do.
ReplyDeleteselect = 'SELECT * FROM `MYTABLE` where vdate between "'+dt+'" and "'+dt2+'"'
rs=dev_conn.select('MYTABLE',select)
for item in rs:
print item.name
I seem to only get 100 when there should be 1506 for the date range I used. I get the proper count if I do:
select = 'SELECT count(*) FROM `MYTABLE` where vdate between "'+dt+'" and "'+dt2+'"'
Thanks in advance again
There are really two interfaces into the select query. The one on the SDBConnection object is a low-level one that requires you to handle the results paging yourself. However, if you instead do:
ReplyDelete>>> d = c.lookup('mydomain')
>>> rs = d.select('your query')
and then iterate over "rs" it will handle the paging for you.
Thanks Mitch as always. It worked!
ReplyDeleteWhat is the app you are interacting with above? The one that produces the vStr that is causing problems?
ReplyDeletegoogle's protocol buffers 2.2.0a for python
ReplyDeleteModel is very simple one
package pviewlog;
message Viewlog {
optional string KEY_ID = 1;
optional int32 ORDER_NUM = 2;
optional string USER = 3;
optional string VIEWED_ON = 4;
}
after setting up the pb.py file
import viewlog_pb2
Then I create the protocol buffer file and then serialize it. I don't seem to have the problem writing the file out to s3 but I am with sdb.
Thanks again
ok a friend of mine has come up with some interesting conclusions about protocol buffers and simple db.
ReplyDelete1. Protocol Buffer data is language neutral when in it is in its binary format;
2. The Java and C++ generated objects (and other referenced code) serialize the buffers in this binary format;
3. I was able to serialize a buffer to SimpleDB in form of UTF-8 because I made a mistake; I used a convenience protocol buffer method (part of the ByteString object) to take their 'encoded' stream and turn into into an encoded ByteString and then turned that into a UTF-8; this allows Java programs to deal with protocol buffers as "real" strings, but these strings (unless the encoding is duplicated) are not the intended binary data (so, only a program that knew that encoding could retrieve it) -- and that totally defeats the purpose; and
4. The Python library for protocol buffers (SerializeToString) does return a str, but it is NOT a real string.
The SimpleDB interface (if I am not mistaking) accepts String data type only. If someone wants to save binary data, then they need to encode it as such. There are may published methods to do that, and we can always build our own. But that would defeat the purpose. We want to use protocol buffers because of the advantages it has, and because it is a convenient language neutral data packaging mechanism.