Please follow the below steps to make changes in spark-env.sh advanced configuration snippet 1. Login to Cloudera manager 2 Choose "SPARK2_ON_YARN-1" on Cluster 3. Choose "Configurations" tab on the displayed page 4. Search "Spark 2 Client Advanced Configuration Snippet (Safety Valve) for spark2-conf/spark-env.sh " in the Search box displayed there. 5. In "Gateway Default Group " change the value to export HADOOP_CONF_DIR=/etc/spark/conf/yarn-conf/*:/etc/hive/conf:/etc/hive/conf. save the configurations and restart the server to make the changes into effect. ... View more

For a basic setup you can have it on the same node (e.g. for testing or low volume usage).    Please see the below considerations when deploying services into nodes: https://www.cloudera.com/documentation/enterprise/release-notes/topics/hardware_requirements_guide.html#concept_qkj_ymn_3bb ... View more

Based on the documentation " When running jobs that require the new Kafka integration, set  SPARK_KAFKA_VERSION=0.10  in the shell before launching spark-submit. Use the appropriate environment variable syntax for your shell"... https://www.cloudera.com/documentation/spark2/latest/topics/spark2_kafka.html#running_jobs ... View more

Hi,   Please check your JDK version and change to a supported one: http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_ig_req_supported_versions.html#concept_pdd_kzf_vp_unique_1 ... View more