Created 01-12-2017 04:27 PM
@kkawamura @Matt @Ali Bajwa @Bryan Bende
I am continually running into the following error and would be extremely grateful for any help. Please let me know if there are any more files or configurations of which you would like to see.
javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user
Both HDF and HDP are Kerberized and running off of the same KDC. HDF and HDP both have Kafka installed, but only the HDP host with Kafka is used in the configurations and the keytabs.I have followed the steps shown in just about every post I could find in order to relieve this issue, but still to no avail. Other pertinent config files will be listed below: On HDF: kafka-jaas.conf
Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/nifi.service.keytab" useTicketCache=false principal="nifi/{_HOST}@ZTEST.LOCAL" }; KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=false renewTicket=true serviceName="kafka" useKeyTab=true keyTab="/etc/security/keytabs/kafka1.service.keytab" principal="kafka/{_HOST}@ZTEST.LOCAL"; };bootstrap.conf
# Java command to use when running NiFi java=java # Username to use when running NiFi. This value will be ignored on Windows. run.as=nifi ##run.as=root # Configure where NiFi's lib and conf directories live lib.dir=/usr/hdf/2.0.2.0-17/nifi/lib conf.dir=/usr/hdf/2.0.2.0-17/nifi/conf # How long to wait after telling NiFi to shutdown before explicitly killing the Process graceful.shutdown.seconds=20 # Disable JSR 199 so that we can use JSP's without running a JDK java.arg.1=-Dorg.apache.jasper.compiler.disablejsr199=true # JVM memory settings java.arg.2=-Xms512m java.arg.3=-Xmx512m # Enable Remote Debugging #java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 java.arg.4=-Djava.net.preferIPv4Stack=true # allowRestrictedHeaders is required for Cluster/Node communications to work properly java.arg.5=-Dsun.net.http.allowRestrictedHeaders=true java.arg.6=-Djava.protocol.handler.pkgs=sun.net.www.protocol # The G1GC is still considered experimental but has proven to be very advantageous in providing great # performance without significant "stop-the-world" delays. java.arg.13=-XX:+UseG1GC #Set headless mode by default java.arg.14=-Djava.awt.headless=true #Ambari Metrics Collector URL - passed in to flow.xml for AmbariReportingTask java.arg.15=-Dambari.metrics.collector.url=http://{_HOST}:6188/ws/v1/timeline/metrics #Application ID - used in flow.xml - passed into flow.xml for AmbariReportingTask java.arg.16=-Dambari.application.id=nifi java.arg.17=-Djava.security.auth.login.config=/etc/nifi/kafka-jaas.conf
PublishKafka_10_0
Kafka Brokers: {_HOST}:6667Security Protocol: SASL_PLAINTEXTKerberos Service Name: kafka
On HDP: kafka_jaas.conf
KafkaServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/kafka.service.keytab" storeKey=true useTicketCache=false serviceName="kafka" principal="kafka/{_HOST}@ZTEST.LOCAL"; }; KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true renewTicket=true serviceName="kafka"; }; Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/kafka.service.keytab" storeKey=true useTicketCache=false serviceName="zookeeper" principal="kafka/{_HOST}@ZTEST.LOCAL"; };Thank you in advance for any help! **Please note all hostnames were replaced with {_HOST}Full Stacktrace from nifi-app.log:
2017-01-12 11:18:42,157 WARN [Timer-Driven Process Thread-10] o.a.n.c.t.ContinuallyRunProcessorTask org.apache.kafka.common.KafkaException: Failed to construct kafka producer at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:335) ~[na:na] at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:163) ~[na:na] at org.apache.nifi.processors.kafka.pubsub.PublisherPool.createLease(PublisherPool.java:61) ~[na:na] at org.apache.nifi.processors.kafka.pubsub.PublisherPool.obtainPublisher(PublisherPool.java:56) ~[na:na] at org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_10.onTrigger(PublishKafka_0_10.java:312) ~[na:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.0.0.2.0.2.0-17.jar:1.0.0.2.0.2.0-17] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) ~[nifi-framework-core-1.0.0.2.0.2.0-17.jar:1.0.0.2.0.2.0-17] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.0.0.2.0.2.0-17.jar:1.0.0.2.0.2.0-17] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.0.0.2.0.2.0-17.jar:1.0.0.2.0.2.0-17] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.0.0.2.0.2.0-17.jar:1.0.0.2.0.2.0-17 ] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86) ~[na:na] at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:71) ~[na:na] at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:83) ~[na:na] at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:277) ~[na:na] ... 16 common frames omitted Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:940) ~[na:1.8.0_77] at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760) ~[na:1.8.0_77] at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) ~[na:1.8.0_77] at sun.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_77] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_77] at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) ~[na:1.8.0_77] at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) ~[na:1.8.0_77] at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) ~[na:1.8.0_77] at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) ~[na:1.8.0_77] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_77] at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) ~[na:1.8.0_77] at javax.security.auth.login.LoginContext.login(LoginContext.java:587) ~[na:1.8.0_77] at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:69) ~[na:na] at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:110) ~[na:na] at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:46) ~[na:na] at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68) ~[na:na] at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78) ~[na:na] ... 19 common frames omitted
Created 01-12-2017 04:31 PM
This may be just be a typo on the post, but you said "On HDF: kafka_jaas.conf" and then in NiFi's bootstrap you have "/etc/nifi/kafka-jaas.conf" so is it kafka_jass.conf or kafka-jass.conf?
Created 01-12-2017 04:31 PM
This may be just be a typo on the post, but you said "On HDF: kafka_jaas.conf" and then in NiFi's bootstrap you have "/etc/nifi/kafka-jaas.conf" so is it kafka_jass.conf or kafka-jass.conf?
Created 01-12-2017 04:33 PM
Typo, let me fix that up.
Created 01-12-2017 04:51 PM
Alright I don't see anything wrong jumping out at me, this is what has worked for me:
https://community.hortonworks.com/questions/63180/error-in-nifi-flow.html#answer-63240
Usually when it gets as far as prompting for a password, it is because something with your keytab and principal was not correct, maybe try listing the keytab and making sure the principal you are using is actually in that keytab.
Created 01-12-2017 05:04 PM
So far looking good, turns out had nifi running as admin user which did not have permissions on that keytab, so far so good, will update with solved once tests confirm. Thank you! For all searching: Make sure the user that nifi is running as has permissions to access the keytab file... Whoops.
Created 01-12-2017 05:12 PM
Now getting error regarding timeout waiting for metadata, any idea here or would you rather I open a new question?
Created 01-12-2017 05:05 PM
For all searching: Make sure the user that nifi is running as has permissions to access the keytab file and if after this you are receiving a metadata timeout error, login to zkCli.sh with Kafka ticket in cache...
kinit kafka/_HOST@REALM.EXAMPLE -kt /etc/security/keytabs/kafka.service.keytab
and rmr /brokers Restart Kafka and everything should be good to go at that point.
Created 01-12-2017 06:55 PM
The issue about waiting for metadata likely means the machine running NiFi can't reach one of the Kafka brokers hosting the topic. There is some property in Kafka that I can't remember that controls what host/ip it presents to external clients which sometimes needs to be set.
Created 01-12-2017 06:59 PM
Changed the FQDN to IP and everything is flowing well now, thank you again for your help!