If you wish to reference a file in S3 from a pig script you might do something like this: set fs.s3n.awsSecretAccessKey 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx';
set fs.s3n.awsAccessKeyId 'xxxxxxxxxxxxxxxxxxxxx';
A = load 's3n://<bucket>/<path-to-file>' USING TextLoader; If you're on HDP 2.2.6, you'll likely see this error: Error: java.io.Exception, no filesystem for scheme: s3n
The following steps resolve this issue: In core-site.xml add: <property>
<name>fs.s3n.impl</name>
<value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
<description>The FileSystem for s3n: (Native S3) uris.</description>
</property> Then add to the MR2 and/or TEZ class path(s): /usr/hdp/${hdp.version}/hadoop-mapreduce/* These configs ensure 2 things: That the worker YARN containers spawned by pig have access to the hadoop-aws.jar file That the worker YARN containers know which class implements the file system type identified by "s3n://" References:
Apache Mailing Lists Topic from Legacy Hortonworks Forums
... View more
