question pyspark streaming writing data in to hive using foreachbatch method in Support Questions

question pyspark streaming writing data in to hive using foreachbatch method in Support Questions https://community.cloudera.com/t5/Support-Questions/pyspark-streaming-writing-data-in-to-hive-using-foreachbatch/m-p/311744#M224808 Hi everybody,I am trying the following approach to write data in to hive table.   <LI-CODE lang="python">import logging from pyspark.sql import SparkSession from pyspark.sql.functions import explode from pyspark.sql.functions import split from pyspark.streaming.kafka import KafkaUtils import datetime from pyspark.sql.functions import lit,unix_timestamp from os.path import * from pyspark import Row warehouseLocation = abspath("spark-warehouse") spark = SparkSession.builder.appName("spark_streaming").config("spark.sql.warehouse.dir", warehouseLocation).enableHiveSupport().getOrCreate() kafka = "kafka" offsets = "earliest" servers = "server_1:port,server_2:port" security_protocol = "SSL" keystore_location = "keystore" keystore_password = "keystore_password" kafka_topic = "kafka_topic" checkpoint_location ="/checkpoint/location" def hiveInsert(df, batchId): df.createOrReplaceTempView("updates") spark.sql("insert into hive_db.hive_table select value, time_stamp from updates") df = spark.readStream.format(kafka).option("startingoffsets", offsets).option("kafka.bootstrap.servers", servers).option("kafka.security.protocol", security_protocol).option("kafka.ssl.keystore.location", keystore_location).option("kafka.ssl.keystore.password", keystore_password).option("subscribe",kafka_topic).load().selectExpr("CAST(value AS STRING)").select('value').withColumn('time_stamp',lit(datetime.datetime.now().strftime('%Y%m%d%H%M'))) query = df.writeStream.foreachBatch(hiveInsert).start() query.awaitTermination()</LI-CODE>   The above code is not workingAny pointers are of great help! Thu, 18 Feb 2021 17:21:06 GMT mark_hadoop 2021-02-18T17:21:06Z