tag:blogger.com,1999:blog-1231480044619721857.post3364479514327737292..comments2022-12-02T05:07:00.365-05:00Comments on Elastician: S3 MultiPart Upload in botoMitch Garnaathttp://www.blogger.com/profile/02589240083555476561noreply@blogger.comBlogger16125tag:blogger.com,1999:blog-1231480044619721857.post-43754700991558909612011-12-29T21:14:54.827-05:002011-12-29T21:14:54.827-05:00Fabian:
This multipart copy is an issue when copy...Fabian:<br /><br />This multipart copy is an issue when copy from S3 bucket to S3 bucket as opposed to uploads from local source to S3. Any ideas on a similar function as FileChunkIO that would take an S3 key without having to download and upload?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-83767369628824674442011-10-20T10:44:29.565-04:002011-10-20T10:44:29.565-04:00Habit?Habit?Mitch Garnaathttps://www.blogger.com/profile/02589240083555476561noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-80776503164896762062011-10-20T10:38:44.279-04:002011-10-20T10:38:44.279-04:00Any reason you're using the deprecated lookup ...Any reason you're using the deprecated lookup method?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-11895372631008646402011-04-18T07:11:11.588-04:002011-04-18T07:11:11.588-04:00Yes, I say your article about parallel uploads and...Yes, I say your article about parallel uploads and tweeted about it. Very cool. I'll check out your FileChunkIO, as well.Mitch Garnaathttps://www.blogger.com/profile/02589240083555476561noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-29569675458836603292011-04-18T06:05:34.280-04:002011-04-18T06:05:34.280-04:00Thanks for implementing that feature! I wrote File...Thanks for implementing that feature! I wrote <a href="https://bitbucket.org/fabian/filechunkio/" rel="nofollow">FileChunkIO</a> that you can avoid splitting the file with. And maybe my code example about <a href="http://www.topfstedt.de/weblog/?p=558" rel="nofollow">parallel S3 multipart uploads using boto</a> is of interest.Fabian Topfstedthttp://topfstedt.denoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-85315163163624074302011-04-08T07:47:19.106-04:002011-04-08T07:47:19.106-04:00There was a bug in boto, too. It was not correctl...There was a bug in boto, too. It was not correctly handling error responses that were sent in the body of 200 responses from the server so it was failing silently on these. That was corrected a few weeks ago in the github master repo.Mitch Garnaathttps://www.blogger.com/profile/02589240083555476561noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-69130868385284233882011-04-08T02:37:59.048-04:002011-04-08T02:37:59.048-04:00Aah, my mistake. My part files were just 1MB. I wa...Aah, my mistake. My part files were just 1MB. I was supposed to split the file in 10MB parts. Now it works.<br /><br />Thank you.Unknownhttps://www.blogger.com/profile/16257094682951015774noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-72035710457116656592011-04-07T11:37:18.988-04:002011-04-07T11:37:18.988-04:00How large are the parts that you are uploading? W...How large are the parts that you are uploading? What happens if you try to list the parts prior to completing the upload?Mitch Garnaathttps://www.blogger.com/profile/02589240083555476561noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-7404646808926990062011-04-07T11:25:14.149-04:002011-04-07T11:25:14.149-04:00Thank you for this post. This is exactly what I ne...Thank you for this post. This is exactly what I need. I'm having some trouble getting the multipart upload working properly though.<br /><br />When the files are incorrectly numbered (something I discovered by accident when copying and pasting), and they all have the same number, I get a file uploaded to my bucket that looks like you might expect (content is that of the last file). But when correctly numbered, nothing shows up in my bucket at all.<br /><br />- - -<br />conn = S3Connection(AWS_ID, AWS_KEY)<br />bucket = conn.lookup('mybucket')<br />mpart = bucket.initiate_multipart_upload('testtmp')<br />fp = open('w_aa', 'r')<br />mpart.upload_part_from_file(fp, 1)<br />fp.close()<br />fp = open('w_ab', 'r')<br />mpart.upload_part_from_file(fp, 2)<br />fp.close()<br />fp = open('w_ac', 'r')<br />mpart.upload_part_from_file(fp, 3)<br />fp.close()<br />fp = open('w_ad', 'r')<br />mpart.upload_part_from_file(fp, 4)<br />fp.close()<br />mpart.complete_upload()<br />- - -<br /><br />Thankful for any help or pointers.Unknownhttps://www.blogger.com/profile/16257094682951015774noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-1013723430477761112011-02-02T16:16:51.946-05:002011-02-02T16:16:51.946-05:00Yes, there will be a 2.0b4 release soon. Can'...Yes, there will be a 2.0b4 release soon. Can't say exactly when but probably within a week. I know I've said that before but this time I really mean it. Really!Mitch Garnaathttps://www.blogger.com/profile/02589240083555476561noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-48770579252895020632011-02-02T12:35:00.847-05:002011-02-02T12:35:00.847-05:00Thanks again for the quick reply. Yes I am a littl...Thanks again for the quick reply. Yes I am a little concerned but our application has a limited set of needs. Some sqs and a few s3 read/write calls. Having another tag would make it easier to use, at least something to report issues against. Is there a 2.0b4 in the near future?Unknownhttps://www.blogger.com/profile/18016858628377169182noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-63403198010826404262011-02-02T08:34:29.806-05:002011-02-02T08:34:29.806-05:00I don't really have a roadmap plan. I've ...I don't really have a roadmap plan. I've been trying to figure out the best way to handle that. It's kind of complicated trying to define the current "stability" of boto when it is comprised of so many different modules at different levels of maturity.<br /><br />The Multipart Upload is a recent addition so I'm sure it's less mature but it really is an incremental addition to existing code. From my POV, the best way to mature software is for it to be used in production but I can also understand your reluctance to use it in production before it is 100% stable. All I can offer is my commitment to respond to any issues as quickly as possible.Mitch Garnaathttps://www.blogger.com/profile/02589240083555476561noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-12071230572905587412011-02-02T00:58:06.162-05:002011-02-02T00:58:06.162-05:00This is announcement is great news, I just hit the...This is announcement is great news, I just hit the need for this feature in boto. Now the hard part, is this stable enough for production? Do you have any roadmap plan?<br /><br />DaveUnknownhttps://www.blogger.com/profile/18016858628377169182noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-53392661490506627812011-01-09T03:28:55.824-05:002011-01-09T03:28:55.824-05:00Sorry it wasn't clear. I posted issue #66 to t...Sorry it wasn't clear. I posted issue #66 to the project page with a very short sample program to https://github.com/boto/boto/issues/#issue/66Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-46552203219360465092011-01-06T20:53:18.998-05:002011-01-06T20:53:18.998-05:00Could you explain a bit more about the problem you...Could you explain a bit more about the problem you were having? I'm not sure I understand the fix you have provided. Also, it may be better to create an issue on the project page to better track this.Mitch Garnaathttps://www.blogger.com/profile/02589240083555476561noreply@blogger.comtag:blogger.com,1999:blog-1231480044619721857.post-73202364021189232942011-01-05T03:35:18.398-05:002011-01-05T03:35:18.398-05:00I believe I have found, and perhaps fixed, a bug r...I believe I have found, and perhaps fixed, a bug related to multipart upload of Unicode (utf8) key names.<br />Scenario:<br /> b = _conn.get_bucket("mybucket")<br /> k = u'/\u00f1'.encode('utf8')<br /> mp = b.initiate_multipart_upload(k)<br /> mp = mp.upload_part_from_file(s,0)<br /> mp = mp.complete_upload()<br /><br />A fix that seems to work for me is to add a test for the endElement method of class MultiPartUpload:<br /><br /> elif name == 'Key':<br /> if type(value) == unicode:<br /> value = value.encode('utf8')Anonymousnoreply@blogger.com