Hi    Continuum ships Anaconda parcel and Cloudera does not have control on which python version it installs.   Please use the OS package management tool to install python 3.5 on the servers in the CDH cluster, once that is done, please follow this doc to set python for your pyspark job:   https://www.cloudera.com/documentation/enterprise/5-8-x/topics/spark_python.html#spark_python__section_ark_lkn_25 ... View more

Okay, since the process owner is cloudera-scm,  one way to fix the issue is adding cloudera-scm user to 'spark' group on all nodes. ... View more

You may need to make sure the process owner of the Spark2 history server (by default it is spark user as well), belongs to the group "spark". So that the spark2 history server process would be able to read all the spark2 event log files.   You can check the process owner with " ps -ef |grep java| grep SPARK2" on the node where spark2 history server runs on. ... View more

This is usually caused by not having proper HADOOP or SPARK CONF on the node. You need to assign spark2 gateway role to this node, and deploy spark2 client configureations, then re-launch spark2-shell. ... View more

There is a complete sample in the Cloudera engineering blog [1], note the requirements mentioned there.   You will need to provide the jaas file with java options, see [2], notice the options used in spark2-submit:  --driver-java-options "-Djava.security.auth.login.config=./spark_jaas.conf"... --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./spark_jaas.conf" along with distributed files via "--files".   You will also need to set the SSL parameters when initlizing the kafka client, see [3].   [1] https://blog.cloudera.com/blog/2017/05/reading-data-securely-from-apache-kafka-to-apache-spark/ [2] https://github.com/markgrover/spark-secure-kafka-app [3] https://github.com/markgrover/spark-secure-kafka-app/blob/master/src/main/java/com/cloudera/spark/examples/DirectKafkaWordCount.scala#L60 ... View more

It is usually caused by a wrong solr url configured somewhere in HUE, you can try to correct it: 1) Check if any one has a safety value configured in HUE -> Configuration -> "Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini"   2) Make sure Solr is selected as a depency service in HUE -> Configuration -> Solr.   3) If above still does not resovle your issue, try to overide solr url with a correct (protocal, host, port) by following :   a) Verify the solr URL, Solr -> Instance -> any one of the Solr instance -> Solr Server -> Solr Server Web UI. Note the url, and we will use it in next step.   b) Use a safety value for solr url in HUE -> Configuration -> search for "Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini", add:   [search] ## URL of the Solr Server solr_url=http://hostname:port/solr  Restart your HUE server. ... View more

Regarding how to make Spark work with Kerberos enabled Kafka, please refer to Cloudera engineering blog:   https://blog.cloudera.com/blog/2017/05/reading-data-securely-from-apache-kafka-to-apache-spark/   There are explainations on prerequisites, solution and sample code. ... View more

It's a spark side configuraion. So you can always specify it via "--conf" option with spark-submit, or you can set the property globally on CM via "Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf", so CM will include such setting for you via spark gateway client configuration. ... View more

#1) Do you have spark gateway installed and client configuration deployed on this host? #2) Do you have Spark selected as one of the dependent service on CM oozie configuration page?     Usgually the client should be able to source the log4j file from /etc/spark/conf/log4j.properties, instead of the NodeManager process dir. ... View more