Hi. Our company has already had Kafka and Zookeeper Instances on Cloudera. But it lacks of some useful functions that Confluent has for handling streaming data (mostly from Kafka Connect) so we want to use Confluent with Kafka instance from Cloudera. But I don’t know where to start. I read some guides on the Confluent homepage (link) but it’s for local installation with its own Zookeeper and Kafka. So I wonder is there any way to “integrate” Confluent with our Kafka/Zookeeper from Cloudera? Does anyone has already done this before and shown me how to do it? We're using Cloudera 6.2.0, I think it comes with Apache Kafka 2.1.0     ... View more

Hi. I want to know Where is the Jar folder for Spark in Cloudera? In my previous company, we just put all the needed jars inside $SPARK_HOME/jar folder (on every node), so that we don't have to worry much about the --jars, --packages,... when running the spark-submit job. Also, it saves lots of disk space and time since we don't need to include every package when building a jar. But in my new company, which uses Cloudera, I don't know where is this jar folder. I found 2 places (maybe not the right one): - /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars - /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars Where should I put the needed jars file on? It seems like all the jars file on the second one are linked to the first one. Also, I found dozens of lib/jar folders everywhere in Cloudera. Or is there any other way to do this with Cloudera? I read some guides about modified Spark configs on the Cloudera manager on the internet.   ... View more