Assistance Required for Deploying Iceberg SerDe in CDP 7.1.9 Cluster

[#1]

Hi Team,

We are planning to deploy the Iceberg Hive runtime JAR (iceberg-hive-runtime.jar) in our CDP Private Cloud Base 7.1.9 cluster (Cloudera Manager 7.11.3) to enable Hive and Spark to work with Iceberg tables.

Iceberg Hive runtime JAR: iceberg-hive-runtime-1.3.1.jar,iceberg-spark-runtime-3.3_2.12-1.3.1.jar

step: 1
Upload the JAR to HDFS

Create a directory in HDFS:
hdfs dfs -mkdir -p /user/share/lib

Upload the JAR:

hdfs dfs -put iceberg-hive-runtime.jar /user/share/lib/
hdfs dfs -chmod 755 /user/share/lib/iceberg-hive-runtime.jar

Configure Hive via Cloudera Manager

Navigate to Hive → Configuration → Auxiliary JARs Directory

Add:

hdfs:///user/share/lib

Step 2:
Configure Spark via Cloudera Manager

Navigate to Spark → Configuration → Spark Client Advanced Configuration Snippet (Safety Valve)

Add:
spark.yarn.jars hdfs:///user/share/lib/iceberg-hive-runtime.jar

Step 3:
Restart Services
Restart Hive, Spark, and YARN.

We would like guidance and confirmation on the steps, any possible impacts, and best practices to ensure a smooth deployment.

• Are the steps outlined above correct and fully supported for CDP 7.1.9?
• Are there any additional recommendations we should be aware of?
• Can you provide guidance on the best way to validate the deployment?

Environment Details:

• CDP Private Cloud Base: 7.1.9
• Cloudera Manager: 7.11.3
• Hadoop Cluster (metadata connection)
• Keytab authentication
• Spark Universal enabled

[#2]

@gurumoorthyk Welcome to our community! To help you get the best possible answer, I have tagged in our Iceberg experts @smruti, @dipankartnt, and @tjangid , who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

@gurumoorthyk Iceberg is not supported with Hive in CDP on-prem 7.1.9. You may enable it using the runtime jars, but some functionalities may not work, and at Cloudera, we do not support it. Iceberg with Hive is supported from 7.3.1(on prem).

The procedure you have listed to load iceberg runtime jar to Hive classpath is correct.

[#4]

@gurumoorthyk adding to Smruthi's update, Iceberg + Hive+ Spark2 is not supported in CDP 7.1.9, it's officially supported from CDP 7.3.1. We don't need to add any jars explicitly in 7.3.1. By default, service[Impala,spark3, Hive] classpaths will have the required iceberg jars.

In CDP 7.1.9, you can use Iceberg with Impala and Spark 3, but with limited functionalities. It does not require any classpath manipulation.

Doc: https://docs.cloudera.com/cdp-public-cloud/cloud/cdp-iceberg/topics/iceberg-in-cdp.html

FYI: [You can check the commands below in your env]

[root@host1 jars]# pwd [You can see impala & spark3 classpaths by default has the iceberg jars]
/opt/cloudera/parcels/
[root@ccycloud-1.nightly-71x-kl.root.comops.site jars]# find ../ -name *iceberg-hive-runtime*
../lib/oozie/oozie-sharelib-yarn/lib/sqoop/iceberg-hive-runtime-1.3.0.7.1.9.1003-449.jar
../lib/hive/lib/atlas-hive-plugin-impl/iceberg-hive-runtime-1.3.0.7.1.9.1003-449.jar
../lib/impala/lib/iceberg-hive-runtime-1.3.0.7.1.9.1003-449.jar ====>impala classpath
../lib/atlas/hook/hive/atlas-hive-plugin-impl/iceberg-hive-runtime-1.3.0.7.1.9.1003-449.jar

../jars/iceberg-hive-runtime-1.3.0.7.1.9.1003-449.jar
[root@host1 jars]# find ../ -name *iceberg-spark*
find ../.. -name *iceberg-spark*
../../SPARK3-3.5.4.3.5.7191000.0-30-1.p0.68499982/lib/spark3/iceberg/iceberg-spark-runtime-3.5_2.12-1.3.0.3.5.7191000.0-30.jar.  =====>SPARK3 Classpath
../../CDH-7.1.9-1.cdh7.1.9.p1003.74501001/lib/oozie/oozie-sharelib-yarn/lib/spark3/iceberg-spark-runtime-3.3_2.12-1.3.0.3.3.7191000.1-138.jar
../../CDH-7.1.9-1.cdh7.1.9.p1003.74501001/jars/iceberg-spark-runtime-3.3_2.12-1.3.0.3.3.7191000.1-138.jar