Dears team, I have been using PySpark 3.4.2 with the following syntax: sql_query = "
INSERT OVERWRITE TABLE table_1 PARTITION(partition_date = {YYYYMMDD})
SELECT
table_1.a
, table_1.b
, table_2.c
FROM table_2 change_capture_view
FULL OUTER JOIN (
SELECT * FROM table_1 WHERE WHERE partition_date = {YYYYMMDD_D_1}
) current_view
ON change_capture_view.a <=> current_view.a
WHERE change_capture_view.a IS NULL
" and use spark.sql(sql_query ). And encountered the error: File "/usr/hdp/3.1.0.0-78/spark3/python/lib/pyspark.zip/pyspark/sql/session.py", line 1440, in sql
File "/usr/hdp/3.1.0.0-78/spark3/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
File "/usr/hdp/3.1.0.0-78/spark3/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 175, in deco
pyspark.errors.exceptions.captured.AnalysisException: Cannot overwrite a path that is also being read from. In essence, I am trying to retrieve data from the partitions of previous dates to process and write into the partition of the current date on the same table. Although previously, with the same syntax, I used it on PySpark 2.3.2.3.1.0.0-78 and it worked normally. Can someone help me with this issue? I've tried creating a temporary table from table 1, but still encountered a similar error.
... View more
