Reply
New Contributor
Posts: 1
Registered: ‎09-12-2017
Accepted Solution

Integrate kerberos with kafka in hbase coprocessor

hi:

 

I want to integrate kerberos with kafka in hbase coprocessor and I could not autenticate inside the application. I am getting this error:

 

Caused by: java.lang.IllegalArgumentException: You must pass java.security.auth.login.config in secure mode.
at org.apache.kafka.common.security.kerberos.Login.login(Login.java:289)
at org.apache.kafka.common.security.kerberos.Login.<init>(Login.java:104)
at org.apache.kafka.common.security.kerberos.LoginManager.<init>(LoginManager.java:44)
at org.apache.kafka.common.security.kerberos.LoginManager.acquireLoginManager(LoginManager.java:85)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:55)
... 14 more

 

 

This is part of my coprocessor postput code:

 

 

SparkConf conf = new SparkConf().setAppName("Coprocessor").setMaster("local[1]");
JavaSparkContext sc = new JavaSparkContext(conf);
sc.getConf().set("spark.yarn.principal","user@EXAMPLE.COM");
sc.getConf().set("spark.yarn.keytab", "/home/user/user.keytab");
sc.getConf().set("spark.yarn.credentials.file", "credential_file");

Properties props = new Properties();
props.put("bootstrap.servers", "server.com:9092");
props.put("client.id", "client-id-coprocessor");
props.put("key.serializer", StringSerializer.class.getName());
props.put("value.serializer", StringSerializer.class.getName());
props.put("security.protocol","SASL_PLAINTEXT");
props.put("sasl.kerberos.service.name", "kafka");

KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);
ProducerRecord<String, String> message = new ProducerRecord<String, String>(KAFKA_TOPIC,"key", "this is a simple message");
producer.send(message);
producer.close();

Cloudera Employee
Posts: 8
Registered: ‎03-01-2016

Re: Integrate kerberos with kafka in hbase coprocessor

Regarding how to make Spark work with Kerberos enabled Kafka, please refer to Cloudera engineering blog:

 

https://blog.cloudera.com/blog/2017/05/reading-data-securely-from-apache-kafka-to-apache-spark/

 

There are explainations on prerequisites, solution and sample code.

Highlighted
New Contributor
Posts: 4
Registered: ‎09-20-2017

Re: Integrate kerberos with kafka in hbase coprocessor

Thanks, It help me to solve my problem!
Announcements