Member since
05-13-2019
3
Posts
0
Kudos Received
0
Solutions
01-07-2020
08:01 AM
Even with HDP 3.1.5 that allow sharing same metastore for Spark and Hive, if you still want to use HWC to connect to Hive manage tables, you'll need LLAP available. This is required because Spark executors speaks directly with LLAP daemons using HWC to be more efficient. See documentations below that explains the mechanism: https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/developing-spark-applications/content/using_spark_hive_warehouse_connector.html https://blog.cloudera.com/hive-warehouse-connector-use-cases/
... View more
10-28-2019
10:40 AM
Since Hadoop 2.8, it is possible to make a directory protected and so all its files cannot be deleted, using : fs.protected.directories property. From documentation: "A comma-separated list of directories which cannot be deleted even by the superuser unless they are empty. This setting can be used to guard important system directories against accidental deletion due to administrator error." It does not exactly answer the question but it is a possibility.
... View more
10-14-2019
02:30 AM
Hello @sduraisankar93, If you are facing this issue, as you said it's because you do not have imported the module. I believe you should check this documentation on how to import HWC and use it : https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/integrating-hive/content/hive_configure_a_spark_hive_connection.html If you are using Zeppelin, please check this : https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/integrating-hive/content/hive_zeppelin_configuration_hivewarehouseconnector.html Note that, in Zeppelin pysark configuration could not work, so a work around is to set (via Ambari) in Zeppelin-env.sh section, this configuration : export SPARK_SUBMIT_OTPIONS="--jars usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-<version>.jar --py-files usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-<version>.zip" Then to start a pyspark shell on your machines, launch this command : pyspark --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-jar --py-files /usr/hdp/current/hive_warehouse_connector/pyspark_hwc-.zip
... View more