The codec should be associated to the Hadoop configuration. In Scala: val hadoopConfig = new org.apache.hadoop.conf.Configuration() val hdfs = org.apache.hadoop.fs.FileSystem.get (hadoopConfig) val bzCodec = new org.apache.hadoop.io.compress.BZip2Codec() bzCodec.setConf (hadoopConfig) val outputFile = hdfs.create (new org.apache.hadoop.fs.Path (uriDest)) val outputStream = bzCodec.createOutputStream (outputFile) ... View more

As stated in https://hortonworks.com/apache/spark/#section_4, both Spark 1.6.3 and 2.1 should be available ... View more