Thank you @mszurap for you response . I tried the suggested work around already and it seems like the issue still persists .  I agree the table has lot of partitions but I am pretty sure the code times out before 5 mins .  I have also tried enforcing the hive-site.xml with the updated timeout which also did not help much.  Only thing which worked was adding spark.catalog.recoverPartitions(table) before issuing the drop partition command .  I am really not sure as why recovering the partitions in the catalog eliminated the metastore warning .  Below is the updated code which is working without any warning : spark.sql.catalog.recoverPartitions(orders) spark.sql("alter table orders drop if exists partition(year=2023)") data.write.mode('Overwrite').parquet(hdfsPath) Any help here in understanding the problem will be much appreciated . ... View more