pyspark - can not create managed table

Question

I'm writing some pyspark code where I have a dataframe that I want to write to a hive table.

I'm using a command like this.

dataframe.write.mode("overwrite").saveAsTable(“bh_test”)

Everything I've read online indicates that this should, by default, create a managed table.

However, no matter what I try, it always creates an external table. Is there a configuration setting somewhere that overrides the default behavior?

jagadeesan · Accepted Answer

Hi @haze5736, You need to use Hive Warehouse Connector (HWC) software to query Apache Hive managed tables from Apache Spark. Using HWC API you can read and write Apache Hive tables from Apache Spark. For example, to write the managed table.

df.write.format(HiveWarehouseSession().HIVE_WAREHOUSE_CONNECTOR).option("table", &tableName>).option("partition", <partition_spec>).save()

Ref: https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/integrating-hive-and-bi/topics/hive-read-write-operations.html

For more details you can refer the below documentation:
https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/integrating-hive-and-bi/topics/hive_hivewarehouseconnector_for_handling_apache_spark_data.html
https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/integrating-hive-and-bi/topics/hive_submit_a_hivewarehouseconnector_python.html

DianaTorres · Answer

@haze5736 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.