This can be achieved by setting the following property in spark, sqlContext.setConf("mapreduce.input.fileinputformat.input.dir.recursive","true") Note here that the property is set usign sqlContext instead of sparkContext. And I tested this in spark 1.6.2 , This can be achieved by setting the following property in the spark. sqlContext.setConf("mapreduce.input.fileinputformat.input.dir.recursive","true") Note: i tested it in spark 1.6.2 Do not set this using spark context but use sqlContext to for dataframes created out of hive tables. ... View more