Issue was with spark authentication set up between spark and spar2 in cdh. default value of spark2 did not match to spark. once changed it resolved the issue spark.authenticate=true ... View more