question Re: spark 2.1.0 Reading *.gz files from an s3 bucket or dir as a Dataframe or Dataset.. in Archives of Support Questions (Read Only)

spark 2.1.0 Reading *.gz files from an s3 bucket or dir as a Dataframe or Dataset..

bigspark — Fri, 03 Feb 2017 01:59:03 GMT

Just wondering if spark supports Reading *.gz files from an s3 bucket or dir as a Dataframe or Dataset.. I think we can read as RDD but its still not working for me. Any help would be appreciated. Thank you.

iam using s3n://.. but spark says invalid input path exception.

val df = spark.sparkContext.textFile("s3n://..../*.gz)

doesnt work for me 😞

I prefer to the s3 dir of .gz files as a DF or Dataset if possible else atleast RDD please. thank you

Re: spark 2.1.0 Reading *.gz files from an s3 bucket or dir as a Dataframe or Dataset..

tmccuch — Sun, 05 Feb 2017 01:38:39 GMT

@BigDataRocks

I believe you need to escape the wildcard: val df = spark.sparkContext.textFile("s3n://..../\*.gz).

Additionally, the S3N filesystem client, while widely used, is no longer undergoing active maintenance except for emergency security issues. The S3A filesystem client can read all files created by S3N. Accordingly it should be used wherever possible.

Please see: https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md for the s3a classpath dependencies and authentication properties you need to be aware of.

A nice tutorial on this subject can be found here: https://community.hortonworks.com/articles/36339/spark-s3a-filesystem-client-from-hdp-to-access-s3.html

Re: spark 2.1.0 Reading *.gz files from an s3 bucket or dir as a Dataframe or Dataset..

tmccuch — Wed, 08 Feb 2017 22:00:27 GMT

@BigDataRocks

Please let me know if this helped answer your question.

Thanks. Tom

Re: spark 2.1.0 Reading *.gz files from an s3 bucket or dir as a Dataframe or Dataset..

stevel — Tue, 14 Feb 2017 03:21:32 GMT

There's also the documentation here: https://hortonworks.github.io/hdp-aws/s3-spark/