Support Questions
Find answers, ask questions, and share your expertise

HDP on AWS using EFS mounts for yarn and hdfs

Solved Go to solution
Highlighted

HDP on AWS using EFS mounts for yarn and hdfs

Explorer

Hello,

Has anyone tired using aws EFS mounts as yarn scratch and hdfs directories?

The data long term will sit in S3.

Please let me know your thoughts.

Thanks,

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: HDP on AWS using EFS mounts for yarn and hdfs

Rising Star

Hi Mike yes, i have seen it once, the efs must be in the same Availability Zone ( if not, you may be charged for data transfer and have some perf degradation .. ) and i think the only downside is that it's network performance , it's less performant that having a local disk. i think it's ok if your use case permit , however if you are planning to use Hbase/Storm where latency is critical , i recommend you to do some benchmarking first.

View solution in original post

1 REPLY 1
Highlighted

Re: HDP on AWS using EFS mounts for yarn and hdfs

Rising Star

Hi Mike yes, i have seen it once, the efs must be in the same Availability Zone ( if not, you may be charged for data transfer and have some perf degradation .. ) and i think the only downside is that it's network performance , it's less performant that having a local disk. i think it's ok if your use case permit , however if you are planning to use Hbase/Storm where latency is critical , i recommend you to do some benchmarking first.

View solution in original post