question Re: What is EMRFS? Is it a file system in AWS that is different from S3? Is the sqoop import command different when EMRFS is used or do you still refer to the "target" as S3? in Archives of Support Questions (Read Only)

What is EMRFS? Is it a file system in AWS that is different from S3? Is the sqoop import command different when EMRFS is used or do you still refer to the "target" as S3?

terri_ulrich — Fri, 03 Mar 2017 20:42:34 GMT

Re: What is EMRFS? Is it a file system in AWS that is different from S3? Is the sqoop import command different when EMRFS is used or do you still refer to the "target" as S3?

stevel — Sat, 04 Mar 2017 06:04:07 GMT

EMRFS is an amazon-proprietary replacement for HDFS for cluster storage.

We work on S3A, which is the open source client for reading and writing data in S3: this is not something you can replace HDFS with. In HDP and HDCloud clusters running in EC2, you must use HDFS for the cluster filesystem, with the S3A client to read data from S3 and write it back and the end of a workflow.

We are doing lots of work on S3A performance, much of which is available in HDCloud and HDP2.5.

Note that you can use S3A for remote access to S3 data: between S3 regions and from physical clusters wherever they live. This lets you use S3 as a backup repository of your Hadoop cluster data.

Re: What is EMRFS? Is it a file system in AWS that is different from S3? Is the sqoop import command different when EMRFS is used or do you still refer to the "target" as S3?

terri_ulrich — Mon, 06 Mar 2017 20:07:54 GMT

Thank you very much for the information. It helps a great deal.

Re: What is EMRFS? Is it a file system in AWS that is different from S3? Is the sqoop import command different when EMRFS is used or do you still refer to the "target" as S3?

rajsyrus — Tue, 07 Mar 2017 18:26:40 GMT

This link may help you in understanding,

http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-file-systems.html