Created 11-30-2017 08:01 PM
Hello,
Has anyone tired using aws EFS mounts as yarn scratch and hdfs directories?
The data long term will sit in S3.
Please let me know your thoughts.
Thanks,
Created 12-01-2017 11:56 AM
Hi Mike yes, i have seen it once, the efs must be in the same Availability Zone ( if not, you may be charged for data transfer and have some perf degradation .. ) and i think the only downside is that it's network performance , it's less performant that having a local disk. i think it's ok if your use case permit , however if you are planning to use Hbase/Storm where latency is critical , i recommend you to do some benchmarking first.
Created 12-01-2017 11:56 AM
Hi Mike yes, i have seen it once, the efs must be in the same Availability Zone ( if not, you may be charged for data transfer and have some perf degradation .. ) and i think the only downside is that it's network performance , it's less performant that having a local disk. i think it's ok if your use case permit , however if you are planning to use Hbase/Storm where latency is critical , i recommend you to do some benchmarking first.