HDP on AWS using EFS mounts for yarn and hdfs

[#1]

Hello,

Has anyone tired using aws EFS mounts as yarn scratch and hdfs directories?

The data long term will sit in S3.

Please let me know your thoughts.

Thanks,

[#2 · Accepted solution]

Hi Mike yes, i have seen it once, the efs must be in the same Availability Zone ( if not, you may be charged for data transfer and have some perf degradation .. ) and i think the only downside is that it's network performance , it's less performant that having a local disk. i think it's ok if your use case permit , however if you are planning to use Hbase/Storm where latency is critical , i recommend you to do some benchmarking first.