Because I ran into this thread when looking how to solve this error and because we found a solution, I thought it might still serve some people if I share what solution we found.
We needed HWC to profile Hive managed + transactional tables from Ataccama (data quality solution). And we found someone who successfully got spark-submit working. We checked their settings and changed the spark-submit as follows:
COMMAND="$SPARK_HOME/bin/$SPARK_SUBMIT \
--files $MYDIR/$LOG4J_FILE_NAME $SPARK_DRIVER_JAVA_OPTS $SPARK_DRIVER_OPTS \
--jars {{ hwc_jar_path }} \
--conf spark.security.credentials.hiveserver2.enabled=false \
--conf "spark.sql.hive.hiveserver2.jdbc.url.principal=hive/_HOST@{{ ad_realm }}" \
--conf spark.dynamicAllocation.enable=false \
--conf spark.hadoop.metastore.catalog.default=hive \
--conf spark.yarn.maxAppAttempts=1 \
--conf spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED \
--conf spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED \
--conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED \
--conf spark.sql.legacy.timeParserPolicy=LEGACY \
--conf spark.sql.legacy.typeCoercion.datetimeToString.enabled=true \
--conf spark.sql.parquet.int96TimestampConversion=true \
--conf spark.sql.extensions=com.hortonworks.spark.sql.rule.Extensions \
--conf spark.sql.extensions=com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension \
--conf spark.kryo.registrator=com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator \
--conf spark.sql.sources.commitProtocolClass=org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol \
--conf spark.datasource.hive.warehouse.read.mode=DIRECT_READER_V2 \
--class $CLASS $JARS $MYLIB $PROPF $LAUNCH $*";
exec $COMMAND
Probably the difference was in the spark.hadoop.metastore.catalog.default=hive setting.
In the above example are some Ansible variables:
hwc_jar_path: "/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/hive-warehouse-connector-assembly-1.0.0.7.1.7.1000-141.jar"
ad_realm is our LDAP realm.
Hope it helps anyone.
