in pyspark addPyfile should be working in the same way as in python directly. So maybe you can provide more details on your issue? The addPyfile is wokring? But the import fails? Do you get an error message? ... View more

@bharat sharma As we see the following error which indicates that you have not placed the hadoop-aws jars in the classpath: py4j.protocol.Py4JJavaError: An error occurred while calling o32.load.: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found<br> . So can you please check and download the aws sdk for java https://aws.amazon.com/sdk-for-java/ Uploaded it to the hadoop directory. please check your "spark.driver.extraClassPath" if it has the "hadoop-aws*.jar" and "aws-java-sdk*.jar" For more details please refer to : https://community.hortonworks.com/articles/25523/hdp-240-and-spark-160-connecting-to-aws-s3-buckets.html https://community.hortonworks.com/articles/36339/spark-s3a-filesystem-client-from-hdp-to-access-s3.html ... View more

OR .collect()[0][1] ... View more

@bharat sharma If the above answer helped addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more