As a followup to this, Apache Spark in HDP 2.5 has all the packaging needed to use the s3a:// connector straight from Spark and the Apache Zeppelin notebook. it also contains a lot of leading edge performance improvements. consult Using Apache Spark with Amazon S3. Although written for HDP-cloud; it applies to HDP-2.5. The main difference is that you need to configure your AWS credentials if you aren't running in EC2 (it will automatically request them from AWS if you are) ... View more