try to switch to a kafka user : su kafka and retry again ... View more

Thanks for details @Param NC SPARK-20922 is already corrected in HDP 2.6.2 as you can see here : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_release-notes/content/patch_spark.html You can check this list also for any other issue. As I said, Spark 2.2 is GA if you can use HDP 2.6.3 ... View more

@Param NC, I am facing the same issue with trying to start a sparksession on yarn. Did you solve this ? ... View more

Thank you for the response , I agree with you completely , But My question even the partial result it not getting stored correctly . Thanks, Param. ... View more

@yvora Actually I am submitting my spark application through program , I have done the same thing programatically by adding the sparkConfig.setJars(new String[]{"myjar.jar"}); still it is not working . ... View more

@Param NC , Please close this thread by accepting the answer and consider asking new question. ... View more

@Param NC From command I can see that you are using --new-consumer to describe consumer group. When new consumer is used it tries to fetch consumer group info from consumer offset topics which gets created in kafka log directory. Try using --zookeeper instead of --new-consumer, for eg : $ /usr/bin/kafka-consumer-groups.sh --zookeeper <zookeeper-hostname>:2181 --describe --group <consumer-group> ... View more

kafka-consumer-offset-checker.sh has been deprecated in Kafka 0.9.0.0 and it does not support secure clusters. See: https://kafka.apache.org/documentation/ ... View more

I am not able to find spark.hadoop.yarn.* properties. these properties are not listed in any spark documents. please help me where can I find list of spark.hadoop.yarn properties? ... View more