Hi Following are the answers of questions What version of CDH are you using to connect to S3? [5.7.0] What connector of S3 are you using (we recommend using s3a:// over anything else)? [tried with both s3n:// and s3a://] What is your command and what is your received error? > I am using spark-shell val hadoopConf = sc.hadoopConfiguration; hadoopConf.set("fs.s3a.access.key", ); hadoopConf.set("fs.s3a.secret.key", ); hadoopConf.set("fs.s3.access.key", ); hadoopConf.set("fs.s3.secret.key", ); hadoopConf.set("fs.s3n.access.key", ); hadoopConf.set("fs.s3n.secret.key", ); var df = sqlContext.read.parquet("s3a:///) df.show(); Exception : 16/08/25 13:46:24 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2, hadoop03.monetization.dev.ggn1.ops.snapdeal.io): com.cloudera.com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain at com.cloudera.com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117) at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3779) at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1107) at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1070) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:322) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2696) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2733) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2715) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:382) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:385) at parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:162) at parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:145) at org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.(SqlNewHadoopRDD.scala:180) at org.apache.spark.rdd.SqlNewHadoopRDD.compute(SqlNewHadoopRDD.scala:126) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
... View more
