Amazon S3 is excited to announce Multipart Upload which allows faster, more flexible uploads into Amazon S3. Multipart Upload allows you to upload a single object as a set of parts. After all parts of your object are uploaded, Amazon S3 then presents the data as a single object. With this feature you can create parallel uploads, pause and resume an object upload, and begin uploads before you know the total object size. For more information on Multipart Upload, review the Amazon S3 Developer Guide or attend the Amazon S3 Multipart Upload webcast on November 17.
2. What is Amazon S3? Amazon S3 is storage for the Internet. It can be used to store and retrieve any amount of data, at any time, from anywhere on the web. AWS Cloud 3rd Party Applications Durable Scalable Secure Available Your Applications Libraries Cost Effective 2
3.
4. If an upload fails, you must restart the object
5. You must know the object size before starting the upload
6. Uploads are linear, no ability to parallelize1 MB √ 10 MB X √ 100 MB X X √ 1000 MB 3
7. What is Multipart Upload? Multipart Upload Initiate 1 Upload ID Initiate the Multipart Upload Separate the object into multiple parts Upload the parts in any order, one at a time or in parallel Complete the upload 1 5 MB 2 5 MB 2 X 3 5 MB 3 1 GB 1 GB 4 5 MB 5 5 MB X 6 5 MB … … Complete 4 4
8. Parallel uploads - upload multiple parts of an object in parallel to improve throughput Pause – begin uploading parts of an object then stop and wait as long as you like before resuming Resume – resume an upload that was paused or one that only partially succeeded Unknown size uploads - begin uploading data before you know the total size of the object Multipart Upload Benefits 5
9. Stream data from genomic sequencer Reduce network failures by chunking into smaller parts when utilizing poor networks Aggregate individual Amazon S3 and CloudFront log files into daily files Elastic Map Reduce job where parts are returned in random order Potential Use Cases 6
11. List in-progress multipart uploads List parts within in-progress multipart uploads Control access using Bucket Policies Other Interesting API Capabilities 8