Hi all, I am trying to read the files from s3 bucket (which contain many sub directories). As of now i am giving the phyisical path to read the files. How to read the files without hard coded values. File path : S3 bucket name/Folder/1005/SoB/20180722_zpsx3Gcc7J2MlNnViVp61/JPR_DM2_ORG/ *.gz files "S3 bucket name/Folder/" this path is fixed one and client id(1005) we have to pass as a parameter. Under Sob folder, we are having monthly wise folders and I have to take only latest two months data. Please help me how to read the data without hard-coded. Many thanks for your help. ... View more

Hi All, I am new to Scala coding and trying to access AWS S3 bucket but it is failed. Please find the below error. Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3native.NativeS3FileSystem not found I want to read multiple files(*.gz) from S3 bucket and make a single CSV file (merge all gz files) but unable to read the data and getting exception as shown above. Here is my code : import org.apache.spark.SparkConf import org.apache.spark.sql.SparkSession object ReadS3Files { def main(args: Array[String]) { val spark = SparkSession.builder.master("local[*]").appName("ReadS3Files").getOrCreate() val sc = spark.sparkContext val conf = new SparkConf().setAppName("ReadS3Files").setMaster("local[*]") val sqlContext = spark.sqlContextspark.sparkContext.hadoopConfiguration.set("fs.s3n.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem") spark.sparkContext.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "AccessKey") spark.sparkContext.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", "SecretKey") val df = spark.read.format("csv").option("delimiter",",").load("s3n://bucketname/201808/1034/JPR_DM2_ORG/*.gz") df.count() spark.stop() } } Please help me on this issue. Many thanks for your help. ... View more