New Contributor
Posts: 1
Registered: ‎09-12-2017
Accepted Solution

Integrate kerberos with kafka in hbase coprocessor

[ Edited ]



I want to integrate kerberos with kafka in hbase coprocessor and I could not autenticate inside the application. I am getting this error:


Caused by: java.lang.IllegalArgumentException: You must pass in secure mode.
... 14 more



This is part of my coprocessor postput code:



SparkConf conf = new SparkConf().setAppName("Coprocessor").setMaster("local[1]");
JavaSparkContext sc = new JavaSparkContext(conf);
sc.getConf().set("spark.yarn.keytab", "/home/user/user.keytab");
sc.getConf().set("spark.yarn.credentials.file", "credential_file");

Properties props = new Properties();
props.put("bootstrap.servers", "");
props.put("", "client-id-coprocessor");
props.put("key.serializer", StringSerializer.class.getName());
props.put("value.serializer", StringSerializer.class.getName());
props.put("", "kafka");

KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);
ProducerRecord<String, String> message = new ProducerRecord<String, String>(KAFKA_TOPIC,"key", "this is a simple message");

Cloudera Employee
Posts: 10
Registered: ‎03-01-2016

Re: Integrate kerberos with kafka in hbase coprocessor

Regarding how to make Spark work with Kerberos enabled Kafka, please refer to Cloudera engineering blog:


There are explainations on prerequisites, solution and sample code.

New Contributor
Posts: 4
Registered: ‎09-20-2017

Re: Integrate kerberos with kafka in hbase coprocessor

Thanks, It help me to solve my problem!
New Contributor
Posts: 1
Registered: ‎08-24-2017

Re: Integrate kerberos with kafka in hbase coprocessor

Hi Flore,


We are blocked due to Co-Processor issue in Kerberos environment. It would be great if you can explain bit detail about the steps you have done for running co-processor in Kerberos Environment.


Below are the few points.

  1. a) Which Keytab you have used, whether CM generated keytab or user keytab generated by you?
  2. b) Path of your jaas.conf and keytab for Kafka?
  3. c) How Kafka Kerberos configuration parameters set?


I am able to execute my coprocessor code in Non Kerberos cluster but in getting error  "org.apache.kafka.common.KafkaException: Jaas configuration not found" while running the code inside the co-processor in Kerberos environment.


Thanks in advance.