I'm trying to connect with HDP 3.0 built-in kafka 1.0.1 using consumer/producer API. When I run the kafka consumers on several cluster nodes, I found an issue with incompatililty between kafka client/server. I'm using the correct kafka 1.0.1 client jars, but I found another legacy "kafka-clients-0.8.2.1.jar" added into classpath, which is causing incompatibility issue: 2CIUSERARG -Djava.class.path=/opt/IBM/InformationServer/Server/DSEngine/../DSComponents/bin/ccapi.jar:.........:/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar:...... 2CIENVVAR CLASSPATH=/usr/hdp/3.0.1.0-187/hadoop/conf:/usr/hdp/3.0.1.0-187/hadoop/lib/nimbus-jose-jwt-4.41.1.jar:.......:/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar:......:/usr/hdp/3.0.1.0-187/tez/lib/tez.tar.gz:./AppMaster.jar I found that this jar is being added to java process using script: /usr/hdp/3.0.0.0-1634/hadoop/libexec/shellprofile.d/hadoop-kafka.sh: hadoop_add_classpath "${HADOOP_TOOLS_HOME}/${HADOOP_TOOLS_LIB_JARS_DIR}/kafka-clients-0.8.2.1.jar" The question is: what is this old legacy kafka-clients-0.8.2.1.jar used for? It has been added in HDP v3., for 2.x version it wasn't there ... View more