adding : SparkContext.setSystemProperty("hive.metastore.sasl.enabled", "true") SparkContext.setSystemProperty("hive.security.authorization.enabled", "true") SparkContext.setSystemProperty("hive.metastore.kerberos.principal", "principal@domain") SparkContext.setSystemProperty("hive.metastore.execute.setugi", "true") to the code resolved the problem. ... View more

No, if you only have one sink, you would have one file (assuming you don't use header variable buckets). The sink will consume from all three partitions and may deliver those in one batch to one file. -pd ... View more