We are trying to write Spark Structure Stream to Partitioned table. Previously we had accomplished this by using the HWC, but now we are trying to use the toTable() function newly added from Spark 3.1.
df.writeStream.format("parquet").option("checkpointLocation", "path/").toTable("database.new_table")
The above code worked perfectly, and the data is written to the new_table in the database. Please note that the new_able is created through this process. However, when we were trying to write to a partitioned hive table using the below code segment, it did not write anything to the table, but it ran without any error.
df.writeStream.format("parquet").option("checkpointLocation", "path/").partitionBy("col_1", "col_2").toTable("database.new_partitioned_table")
The col_1 and col_2 are columns available in the df.
Please let me know if someone knows where I made the mistake or from where we can get additional reference materials about the toTable() function.
