We need run Create table and alter table statements to hive from within a spark streaming application. The spark version is 2.1.X and it is HDP 2.6.2. spark.sqlContext.sql("CREATE ...") spark.sqlContext.sql("ALTER ...") The create statement works but alter fails with Spark error "Operation not allowed". We are stuck and not able to proceed further. Is there any way by which we can run Alter table commands to hive with HDP 2.6.2 from within a spark application. From the SparkSession we cannot get hold of HiveContext hence that approach is also not possible. Any help in this matter is much appreciated. ... View more

I am using spark session to save a data frame to hive table. The code is as below. df.write.mode(SaveMode.Append).format("orc").insertInto("table") The data comes to spark from kafka. This can be huge amount of data coming throughout the day. Does , spark dataframe save internally does hive compaction ?. If not what is the best way to do compaction at regular intervals without affecting the table insertions. ... View more

What is the advantage of hive-streaming over hive-jdbc. If we do batch in jdbc what extra advantage hive-streaming has over hive jdbc ... View more

Is it a valid scenario to use hive-streaming inside a spark program. I have seen examples of hive streaming as standalone program and spark streaming for writing to hive. Never seen any program where hive-streaming is used inside a spark application and submitted to cluster. Does hive streaming work inside a spark application or is this a totally wrong usage. Please share your thoughts. ... View more