Created on 09-12-2017 01:17 PM - edited 09-16-2022 05:14 AM
hi:
I want to integrate kerberos with kafka in hbase coprocessor and I could not autenticate inside the application. I am getting this error:
Caused by: java.lang.IllegalArgumentException: You must pass java.security.auth.login.config in secure mode.
at org.apache.kafka.common.security.kerberos.Login.login(Login.java:289)
at org.apache.kafka.common.security.kerberos.Login.<init>(Login.java:104)
at org.apache.kafka.common.security.kerberos.LoginManager.<init>(LoginManager.java:44)
at org.apache.kafka.common.security.kerberos.LoginManager.acquireLoginManager(LoginManager.java:85)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:55)
... 14 more
This is part of my coprocessor postput code:
SparkConf conf = new SparkConf().setAppName("Coprocessor").setMaster("local[1]");
JavaSparkContext sc = new JavaSparkContext(conf);
sc.getConf().set("spark.yarn.principal","user@EXAMPLE.COM");
sc.getConf().set("spark.yarn.keytab", "/home/user/user.keytab");
sc.getConf().set("spark.yarn.credentials.file", "credential_file");
Properties props = new Properties();
props.put("bootstrap.servers", "server.com:9092");
props.put("client.id", "client-id-coprocessor");
props.put("key.serializer", StringSerializer.class.getName());
props.put("value.serializer", StringSerializer.class.getName());
props.put("security.protocol","SASL_PLAINTEXT");
props.put("sasl.kerberos.service.name", "kafka");
KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);
ProducerRecord<String, String> message = new ProducerRecord<String, String>(KAFKA_TOPIC,"key", "this is a simple message");
producer.send(message);
producer.close();
Created 09-20-2017 12:41 PM
Created 09-16-2017 10:43 PM
Regarding how to make Spark work with Kerberos enabled Kafka, please refer to Cloudera engineering blog:
https://blog.cloudera.com/blog/2017/05/reading-data-securely-from-apache-kafka-to-apache-spark/
There are explainations on prerequisites, solution and sample code.
Created 09-20-2017 12:41 PM
Created 11-01-2017 12:07 AM
Hi Flore,
We are blocked due to Co-Processor issue in Kerberos environment. It would be great if you can explain bit detail about the steps you have done for running co-processor in Kerberos Environment.
Below are the few points.
I am able to execute my coprocessor code in Non Kerberos cluster but in getting error "org.apache.kafka.common.KafkaException: Jaas configuration not found" while running the code inside the co-processor in Kerberos environment.
Thanks in advance.
Regards
Sumanta
Created on 01-19-2018 03:34 PM - edited 01-19-2018 03:37 PM
We got the working pointing to the HBase keytab, ensuring that the jaas.conf exists on each master/region server.
And my coprocessor produces messages to a secure Kafka topic.
Of course you need to have the master/region server pointing to the jaas.conf file...
ie. Master and region Java configuration...
-Djava.security.auth.login.config=/etc/hbase/jaas.conf
Created 01-24-2018 07:57 AM