@TJSully  Similarly for the Zookeeper process you can search for "zookeeper.log.dir" property in the output: # ps -ef | grep zookeeper | grep zookeeper.log.dir --color ... View more

@TJSully  On the Kafka Host where Kafka Broker is running can you please try running the following command to see the value of "kafka.logs.dir" property. # ps -ef | grep kafka . Example Output in my case: kafka 21831 1 41 22:58 ? 00:00:03 /usr/jdk64/jdk1.8.0_112/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Xloggc:/var/log/kafka/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/var/log/kafka -Dlog4j.configuration=file:/usr/hdp/2.6.5.0-292kafka/bin/../config/log4j.properties .................. kafka.Kafka /usr/hdp/2.6.5.0-292/kafka/config/server.properties   Notice the value of property "kafka.logs.dir" it might be a different value in your case. .   . ... View more

@eswarloges  In one of your previous update you have mentioned that you have added  "the hiveserver2 jdbc url is configured in the spark config." However, looks like the error that you are getting is because the mentioned properties are not found in the spark2-defaults config which is there in your classpath.   So can you please make sure that you have included the correct CLASSPATH which is pointing to correct spark-defaults which has the following properties added as mentioned in "Required properties" Section of the following Doc:  https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/integrating-hive/content/hive_configure_a_spark_hive_connection.html . You must add several Spark properties through spark2-defaults in Ambari to use the Hive Warehouse Connector for accessing data in Hive.  Alternatively, configuration can be provided for each job using --conf.   spark.sql.hive.hiveserver2.jdbc.url The URL for HiveServer2 Interactive spark.datasource.hive.warehouse.metastoreUri The URI for the metastore spark.datasource.hive.warehouse.load.staging.dir The HDFS temp directory for batch writes to Hive, /tmp for example spark.hadoop.hive.llap.daemon.service.hosts The application name for LLAP service spark.hadoop.hive.zookeeper.quorum The ZooKeeper hosts used by LLAP Set the values of these properties as follows: spark.sql.hive.hiveserver2.jdbc.url In Ambari, copy the value from Services > Hive > Summary > HIVESERVER2 INTERACTIVE JDBC URL. spark.datasource.hive.warehouse.metastoreUri Copy the value from hive.metastore.uris. In Hive, at the hive> prompt, enter set hive.metastore.uris and copy the output. For example, thrift://mycluster-1.com:9083. spark.hadoop.hive.llap.daemon.service.hosts Copy value from Advanced hive-interactive-site > hive.llap.daemon.service.hosts. spark.hadoop.hive.zookeeper.quorum Copy the value from Advanced hive-sitehive.zookeeper.quorum. .     .     ... View more

@mike_bronson7  You can try something like this:   # curl -u admin:admin -H "X-Requested-By: ambari" -X GET "http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/alerts?fields=*&Alert/service_name=KAFKA"   . Example Output:   # curl -u admin:admin -H "X-Requested-By: ambari" -X GET "<a href="http://newhwx1.example.com:8080/api/v1/clusters/NewCluster/alerts?fields=*&Alert/service_name=KAFKA" target="_blank">http://newhwx1.example.com:8080/api/v1/clusters/NewCluster/alerts?fields=*&Alert/service_name=KAFKA</a>" { "href" : "<a href="http://newhwx1.example.com:8080/api/v1/clusters/NewCluster/alerts?fields=*&Alert/service_name=KAFKA" target="_blank">http://newhwx1.example.com:8080/api/v1/clusters/NewCluster/alerts?fields=*&Alert/service_name=KAFKA</a>", "items" : [ { "href" : "<a href="http://newhwx1.example.com:8080/api/v1/clusters/NewCluster/alerts/704" target="_blank">http://newhwx1.example.com:8080/api/v1/clusters/NewCluster/alerts/704</a>", "Alert" : { "cluster_name" : "NewCluster", "component_name" : "KAFKA_BROKER", "definition_id" : 401, "definition_name" : "kafka_broker_process", "firmness" : "HARD", "host_name" : "newhwx3.example.com", "id" : 704, "instance" : null, "label" : "Kafka Broker Process", "latest_timestamp" : 1575403190535, "maintenance_state" : "OFF", "occurrences" : 14, "original_timestamp" : 1575402410385, "repeat_tolerance" : 1, "repeat_tolerance_remaining" : 0, "scope" : "HOST", "service_name" : "KAFKA", "state" : "OK", "text" : "TCP OK - 0.000s response on port 6667" } }, { "href" : "<a href="http://newhwx1.example.com:8080/api/v1/clusters/NewCluster/alerts/1201" target="_blank">http://newhwx1.example.com:8080/api/v1/clusters/NewCluster/alerts/1201</a>", "Alert" : { "cluster_name" : "NewCluster", "component_name" : "KAFKA_BROKER", "definition_id" : 401, "definition_name" : "kafka_broker_process", "firmness" : "HARD", "host_name" : "newhwx5.example.com", "id" : 1201, "instance" : null, "label" : "Kafka Broker Process", "latest_timestamp" : 1575403167289, "maintenance_state" : "OFF", "occurrences" : 12, "original_timestamp" : 1575402507311, "repeat_tolerance" : 1, "repeat_tolerance_remaining" : 0, "scope" : "HOST", "service_name" : "KAFKA", "state" : "CRITICAL", "text" : "Connection failed: [Errno 111] Connection refused to newhwx5.example.com:6667" } } ] }         .   ... View more

@asmarz  Please try using "spark-submit" instead of "spark-shell" as mentioned in: https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/running-spark-applications/content/running_sample_spark_2_x_applications.html Example: # /usr/hdp/current/spark2-client/bin/spark-submit --class org.apache.spark.examples.SparkPi \ --master yarn \ --num-executors 1 \ --driver-memory 512m \ --executor-memory 512m \ --executor-cores 1 \ --deploy-mode cluster \ /usr/hdp/current/spark2-client/examples/jars/spark-examples_2.11-2.3.2.3.1.4.0-315.jar 10 ... View more