Member since
01-15-2016
37
Posts
13
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
6168 | 04-24-2017 01:48 PM | |
2367 | 06-21-2016 12:05 PM | |
2825 | 03-07-2016 10:43 PM |
01-12-2017
06:59 PM
Changed the FQDN to IP and everything is flowing well now, thank you again for your help!
... View more
01-12-2017
05:12 PM
Now getting error regarding timeout waiting for metadata, any idea here or would you rather I open a new question?
... View more
01-12-2017
05:05 PM
For all searching: Make sure the user that nifi is running as has permissions to access the keytab file and if after this you are receiving a metadata timeout error, login to zkCli.sh with Kafka ticket in cache...
kinit kafka/_HOST@REALM.EXAMPLE -kt /etc/security/keytabs/kafka.service.keytab and rmr /brokers
Restart Kafka and everything should be good to go at that point.
... View more
01-12-2017
05:04 PM
So far looking good, turns out had nifi running as admin user which did not have permissions on that keytab, so far so good, will update with solved once tests confirm. Thank you!
For all searching:
Make sure the user that nifi is running as has permissions to access the keytab file... Whoops.
... View more
01-12-2017
04:33 PM
Typo, let me fix that up.
... View more
01-12-2017
04:27 PM
@kkawamura @Matt @Ali Bajwa @Bryan Bende I am continually running into the following error and would be extremely grateful for any help. Please let me know if there are any more files or configurations of which you would like to see. javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user
Both HDF and HDP are Kerberized and running off of the same KDC. HDF and HDP both have Kafka installed, but only the HDP host with Kafka is used in the configurations and the keytabs.I have followed the steps shown in just about every post I could find in order to relieve this issue, but still to no avail. Other pertinent config files will be listed below:
On HDF:
kafka-jaas.conf
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/nifi.service.keytab"
useTicketCache=false
principal="nifi/{_HOST}@ZTEST.LOCAL"
};
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=false
renewTicket=true
serviceName="kafka"
useKeyTab=true
keyTab="/etc/security/keytabs/kafka1.service.keytab"
principal="kafka/{_HOST}@ZTEST.LOCAL";
};
bootstrap.conf
# Java command to use when running NiFi
java=java
# Username to use when running NiFi. This value will be ignored on Windows.
run.as=nifi
##run.as=root
# Configure where NiFi's lib and conf directories live
lib.dir=/usr/hdf/2.0.2.0-17/nifi/lib
conf.dir=/usr/hdf/2.0.2.0-17/nifi/conf
# How long to wait after telling NiFi to shutdown before explicitly killing the Process
graceful.shutdown.seconds=20
# Disable JSR 199 so that we can use JSP's without running a JDK
java.arg.1=-Dorg.apache.jasper.compiler.disablejsr199=true
# JVM memory settings
java.arg.2=-Xms512m
java.arg.3=-Xmx512m
# Enable Remote Debugging
#java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
java.arg.4=-Djava.net.preferIPv4Stack=true
# allowRestrictedHeaders is required for Cluster/Node communications to work properly
java.arg.5=-Dsun.net.http.allowRestrictedHeaders=true
java.arg.6=-Djava.protocol.handler.pkgs=sun.net.www.protocol
# The G1GC is still considered experimental but has proven to be very advantageous in providing great
# performance without significant "stop-the-world" delays.
java.arg.13=-XX:+UseG1GC
#Set headless mode by default
java.arg.14=-Djava.awt.headless=true
#Ambari Metrics Collector URL - passed in to flow.xml for AmbariReportingTask
java.arg.15=-Dambari.metrics.collector.url=http://{_HOST}:6188/ws/v1/timeline/metrics
#Application ID - used in flow.xml - passed into flow.xml for AmbariReportingTask
java.arg.16=-Dambari.application.id=nifi
java.arg.17=-Djava.security.auth.login.config=/etc/nifi/kafka-jaas.conf PublishKafka_10_0 Kafka Brokers: {_HOST}:6667Security Protocol: SASL_PLAINTEXTKerberos Service Name: kafka On HDP:
kafka_jaas.conf KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/kafka.service.keytab"
storeKey=true
useTicketCache=false
serviceName="kafka"
principal="kafka/{_HOST}@ZTEST.LOCAL";
};
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
renewTicket=true
serviceName="kafka";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/kafka.service.keytab"
storeKey=true
useTicketCache=false
serviceName="zookeeper"
principal="kafka/{_HOST}@ZTEST.LOCAL";
};
Thank you in advance for any help!
**Please note all hostnames were replaced with {_HOST}Full Stacktrace from nifi-app.log:
2017-01-12 11:18:42,157 WARN [Timer-Driven Process Thread-10] o.a.n.c.t.ContinuallyRunProcessorTask
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:335) ~[na:na]
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:163) ~[na:na]
at org.apache.nifi.processors.kafka.pubsub.PublisherPool.createLease(PublisherPool.java:61) ~[na:na]
at org.apache.nifi.processors.kafka.pubsub.PublisherPool.obtainPublisher(PublisherPool.java:56) ~[na:na]
at org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_10.onTrigger(PublishKafka_0_10.java:312) ~[na:na]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.0.0.2.0.2.0-17.jar:1.0.0.2.0.2.0-17]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) ~[nifi-framework-core-1.0.0.2.0.2.0-17.jar:1.0.0.2.0.2.0-17]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.0.0.2.0.2.0-17.jar:1.0.0.2.0.2.0-17]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.0.0.2.0.2.0-17.jar:1.0.0.2.0.2.0-17]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.0.0.2.0.2.0-17.jar:1.0.0.2.0.2.0-17
]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_77]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86) ~[na:na]
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:71) ~[na:na]
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:83) ~[na:na]
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:277) ~[na:na]
... 16 common frames omitted
Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:940) ~[na:1.8.0_77]
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760) ~[na:1.8.0_77]
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) ~[na:1.8.0_77]
at sun.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_77]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_77]
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) ~[na:1.8.0_77]
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) ~[na:1.8.0_77]
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) ~[na:1.8.0_77]
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) ~[na:1.8.0_77]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_77]
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) ~[na:1.8.0_77]
at javax.security.auth.login.LoginContext.login(LoginContext.java:587) ~[na:1.8.0_77]
at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:69) ~[na:na]
at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:110) ~[na:na]
at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:46) ~[na:na]
at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68) ~[na:na]
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78) ~[na:na]
... 19 common frames omitted
... View more
Labels:
07-08-2016
02:21 PM
If you could kill all of those processes (if your cluster will allow) and try running again. It looks like a lot of Tez resources are busy and thus could be why you are experiencing the hanging. If not you may need to look into YARN Queue Manager
... View more
07-08-2016
12:35 AM
I am was originally using the packaged version of atlas (0.5) but have removed that and am installing 0.8 now and will try again with that version installed. The cluster is currently not kerberized as it is just an internal testing cluster.
... View more
07-07-2016
06:35 PM
So I am jumping the gun on HDP 2.5 and trying to integrate Ranger 0.6.0-SNAPSHOT into my current 2.4.2 cluster and without much luck... Has anyone else been able to successfully do this? I have ranger up and running and the policies created, just having some trouble with the Hive Plugin being recognized and the Ranger TagSync function with Atlas to work although I am getting green lights across the board.
... View more
Labels:
07-05-2016
06:18 PM
The problem could lie with not enough resources available for a Tez session to start. Could you run yarn application -list on a YARN client host and see if there are any processes hogging up resources or possibly check the configs and logs to make sure that Tez isn't just hanging and waiting for resources to become available.
... View more