HDP 2.2 Configuration required for S3

If you wish to reference a file in S3 from a pig script you might do something like this:

set fs.s3n.awsSecretAccessKey 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx';
set fs.s3n.awsAccessKeyId 'xxxxxxxxxxxxxxxxxxxxx';
A = load 's3n://<bucket>/<path-to-file>' USING TextLoader;

If you're on HDP 2.2.6, you'll likely see this error:

Error: java.io.Exception, no filesystem for scheme: s3n


The following steps resolve this issue:

In core-site.xml add:

<property>
<name>fs.s3n.impl</name>
<value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
<description>The FileSystem for s3n: (Native S3) uris.</description>
</property>

Then add to the MR2 and/or TEZ class path(s):

/usr/hdp/${hdp.version}/hadoop-mapreduce/*

These configs ensure 2 things:

1. That the worker YARN containers spawned by pig have access to the hadoop-aws.jar file
2. That the worker YARN containers know which class implements the file system type identified by "s3n://"

References:

• Apache Mailing Lists
• Topic from Legacy Hortonworks Forums