It seems like your Spark workers are pointing to the default/system installation of python rather than your virtual environment. By setting the environment variable, you can tell Spark to use your virtual environment. You can set the below two configs in <spark_home_dir>/conf/spark-env.sh export PYSPARK_PYTHON=<Python_binaries_Path> export PYSPARK_DRIVER_PYTHON=<Python_binaries_Path> ... View more

Hi @dfdf, I am not able to reproduce this issue. I can able to get the table details while running the queries in the Spark3 session. Please can you help us with the exact version of Spark3 and Hive version that you are running on your environment? For example, you can get the value by running spark3-shell --version Please verify are you seeing any errors or alerts related to the Hive service? Also, can you try to run similar queries directly from Hive and see whether are you getting the results?  ... View more

@QiDam Please can you check the Ozone service installed in your CDP Base Cluster and also ensure the Ozone service is up and running? After Ozone service is set up and functional, enable CDE Service afresh and remove the failed CDE service. Also, can you check all PODs are up and running? ... View more

Thanks @adrijand  for sharing your updates, it's highly appreciated. ... View more

Hi @joyabrata  I think you are looking in the Data Lake tab which is a different one, you can go to the Summary tab, then scroll down to FreeIPA session then click Actions and get Get FreeIPA Certificate from the drop-down menu. Hope this will help you.  ... View more

Hi @joyabrata Obtain the FreeIPA certificate of your environment: From the CDP Home Page navigate to Management Console > Environments. Locate and select your environment from the list of available environments. Go to the Summary tab, scroll down to FreeIPA session then click Actions Select Get FreeIPA Certificate from the drop-down menu. The FreeIPA certificate downloads and follows this document  ... View more

Hi @adrijand  Thanks for your detailed explanation here. Yeah indeed, we need all versions to be the same to avoid some classnotfoundexception because of jar conflicts. We encourage you to explore these and provide feedback on your experiences.  ... View more

Hi @adrijand Yeah, it seems some jar conflicts somewhere. You are trying to load Hive 1.1.0 classes before the ones included with Spark, and as such, they might fail to reference a Hive configuration that didn't exist in 1.1.0. like below. : java.lang.NoSuchFieldError: METASTORE_CLIENT_SOCKET_LIFETIME at org.apache.spark.sql.hive.HiveUtils$.formatTimeVarsForHiveClient(HiveUtils.scala:195) at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:286) at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66) But here in the description mentioned you are using CDH 5.15v but in your log snippets it showing Apache Spark (spark-2.3.0-bin-without-hadoop) and Apache Hive (apache-hive-1.1.0-bin) version which is not a pre-built package version that comes along with CDH stack compatibility. Are you trying with building with varying versions of Hive which you would like to connect from a remote airflow docker container?  ... View more