question Re: pyspark - can not create managed table in Support Questions

pyspark - can not create managed table

haze5736 — Tue, 21 Apr 2026 07:55:12 GMT

I'm writing some pyspark code where I have a dataframe that I want to write to a hive table.

I'm using a command like this.

dataframe.write.mode("overwrite").saveAsTable(“bh_test”)

Everything I've read online indicates that this should, by default, create a managed table.

However, no matter what I try, it always creates an external table. Is there a configuration setting somewhere that overrides the default behavior?

Re: pyspark - can not create managed table

jagadeesan — Wed, 15 Jun 2022 15:35:34 GMT

Hi @haze5736, You need to use Hive Warehouse Connector (HWC) software to query Apache Hive managed tables from Apache Spark. Using HWC API you can read and write Apache Hive tables from Apache Spark. For example, to write the managed table.

df.write.format(HiveWarehouseSession().HIVE_WAREHOUSE_CONNECTOR).option("table", &tableName>).option("partition", <partition_spec>).save()

Ref: https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/integrating-hive-and-bi/topics/hive-read-write-operations.html

For more details you can refer the below documentation:
https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/integrating-hive-and-bi/topics/hive_hivewarehouseconnector_for_handling_apache_spark_data.html
https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/integrating-hive-and-bi/topics/hive_submit_a_hivewarehouseconnector_python.html

Re: pyspark - can not create managed table

DianaTorres — Mon, 27 Jun 2022 16:33:49 GMT

@haze5736 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks