question Re: What kind of speed/throughput should I be expecting to AWS S3? in Support Questions

What kind of speed/throughput should I be expecting to AWS S3?

cspencer — Tue, 06 Oct 2015 02:45:42 GMT

Re: What kind of speed/throughput should I be expecting to AWS S3?

drice1 — Tue, 06 Oct 2015 02:49:44 GMT

What kind of speed/throughput are you seeing? Are you just writing to an S3 bucket within an AWS instance? Note sure if you saw this post but this may help a bit http://www.rightscale.com/blog/cloud-industry-insights/network-performance-within-amazon-ec2-and-amazon-s3

Re: What kind of speed/throughput should I be expecting to AWS S3?

drice1 — Tue, 06 Oct 2015 03:16:00 GMT

Ok that helps, here is what I found related to this...hope it helps

Test: Copy 50 GB of data from a Hadoop cluster running on Amazon Elastic Compute Cloud (EC2) in Virginia to an Amazon S3 bucket in Oregon. NOTE the different between S3DistCp and DistCp, but your results may vary.

Method Data Size Copied Total Time

DistCp 50 GB 26 min

S3DistCp 50 GB 19 Min.

See here for more context https://media.amazonwebservices.com/AWS_Amazon_EMR_Best_Practices.pdf

Re: What kind of speed/throughput should I be expecting to AWS S3?

cspencer — Tue, 06 Oct 2015 03:18:28 GMT

thanks @drice@hortonworks.com

Re: What kind of speed/throughput should I be expecting to AWS S3?

Jagatheeshr — Thu, 08 Oct 2015 18:37:03 GMT

@Cassandra

Here are some of the results

S3 vs Native HDFS

Courtesy : Professional Hadoop Solutions.

Re: What kind of speed/throughput should I be expecting to AWS S3?

sdutta — Wed, 18 Nov 2015 01:26:53 GMT

There is a new S3 driver that Gopal has written that is supposed to be as fast as the driver from AWS. If you are implementing in the field, please reach out to him. It is supposed to be part of HDP 2.4 afaik 🙂