I am getting this error while running the below code in CDH 6.2   import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ import org.apache.spark.sql.DataFrame import org.apache.spark.sql.Row import org.apache.spark.sql.Column import org.apache.spark.sql.streaming.StreamingQueryException import org.apache.spark.sql.types._ import java.io.PrintStream import org.apache.spark.SparkConf import org.apache.spark.SparkContext import java.io.File import org.apache.spark.sql.{Row, SaveMode, SparkSession} object test3 { def main(args: Array[String]){ val warehouseLocation = new File("spark-warehouse").getAbsolutePath val spark = SparkSession .builder() .appName("Spark Hive Example") .config("spark.sql.warehouse.dir", warehouseLocation) .enableHiveSupport() .getOrCreate() import spark.implicits._ import spark.sql sql("SELECT * FROM default.test").show() }} ... View more

I have created a topic and ingest some sample data using producer . But I am not able to consume those using spark structure streaming using below streaming code but it works fine using only kafka consumer code. /usr/hdp/3.1.0.0-78/kafka/bin/kafka-topics.sh --create --zookeeper hadoopmaster:2181 --replication-factor 1 --partitions 1 --topic test_topic /usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list hadoopmaster:6667 --topic test_topic >abc >xyz >mno spark-shell --packages "org.apache.spark:spark-streaming-kafka_2.11:1.6.3" --packages "org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.2" scala > val df1 = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "hadoopmaster:6667").option("subscribe", "test_topic").load() scala > df1.writeStream.format("console").option("truncate",false).start().awaitTermination() Here no data is coming. ... View more

I have created a topic and ingest some sample data using producer . But I am not able to consume those using spark structure streaming using below streaming code but it works fine using only kafka consumer code. /usr/hdp/3.1.0.0-78/kafka/bin/kafka-topics.sh --create --zookeeper hadoopmaster:2181 --replication-factor 1 --partitions 1 --topic test_topic /usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list hadoopmaster:6667 --topic test_topic >abc >xyz >mno spark-shell --packages "org.apache.spark:spark-streaming-kafka_2.11:1.6.3" --packages "org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.2" scala > val df1 = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "hadoopmaster:6667").option("subscribe", "test_topic").load() scala > df1.writeStream.format("console").option("truncate",false).start().awaitTermination() Here no data is coming. ... View more

[root@hadoopmaster ~]# spark-shell --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.1.0-187.jar scala> import com.hortonworks.hwc.HiveWarehouseSession import com.hortonworks.hwc.HiveWarehouseSession scala> import com.hortonworks.hwc.HiveWarehouseSession._ import com.hortonworks.hwc.HiveWarehouseSession._ scala> val hive = HiveWarehouseSession.session(spark).build() hive: com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl = com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl@5da2966 scala> hive.setDatabase("default") scala> val df = hive.executeQuery("select * from dd1") df: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [col: int] scala> df.show() 19/01/11 15:35:24 ERROR LlapBaseInputFormat: Closing connection due to error shadehive.org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.IllegalArgumentException: No running LLAP daemons! Please check LLAP service status and zookeeper configuration ... View more

I am running the below code to load oracle data to hive but I am getting the error mentioned below. spark-submit --driver-class-path /app/ojdbc5.jar /app/raci.py Driver stacktrace: 18/09/27 16:30:47 INFO DAGScheduler: Job 0 failed: saveAsTable at NativeMethodAccessorImpl.java:0, took 5.097862 s 18/09/27 16:30:47 ERROR FileFormatWriter: Aborting job null. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, hadoopslave2, executor 1): java.lang.ClassNotFoundException: oracle.jdbc.OracleDriver ... View more