We are trying to write Spark Structure Stream to Partitioned table. Previously we had accomplished this by using the HWC, but now we are trying to use the toTable() function newly added from Spark 3.1.
The above code worked perfectly, and the data is written to the new_table in the database. Please note that the new_able is created through this process. However, when we were trying to write to a partitioned hive table using the below code segment, it did not write anything to the table, but it ran without any error.