Support Questions
Find answers, ask questions, and share your expertise

Error in NiFi Flow:

Solved Go to solution
Highlighted

Error in NiFi Flow:

Expert Contributor

Hi,

I have an error while trying to stream the data using NiFi flow, in a kerberized environment with LDAP integrated.

The error "failed while waiting for acks from Kafka" , I attached the error and properties screenshot. By the way there is a conf called " Kerberos Service Name" is that the error,

Any help is highly appreciated.

Thanks,

Sujitha

screen-shot-2016-10-24-at-124017-pm.png

screen-shot-2016-10-24-at-124006-pm.png

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Error in NiFi Flow:

Here is what you would need to do to configure PublishKafka to talk to a kerberized Kafka...

1) You can either rely on /etc/krb5.conf or you can tell NiFi to use a specific krb5.conf by setting nifi.kerberos.krb5.file= in nifi.properties to point to some other krb5.conf file.

2) Create a JAAS file lets say kafka-jaas.conf with the following:

KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  storeKey=true
  keyTab="/path/to/nifi-iotdemot.keytab"
  serviceName="kafka"
  principal="nifi/iotdemo.field.hortonworks.com@LAKE";
};

Changing the keyTab path to the appropriate path.

3) Specify this in NiFi's bootstrap.conf:

java.arg.15=-Djava.security.auth.login.config=/path/to/kafka-jaas.conf

4) Configure PublishKafka:

Security Protocol = PLAINTEXTSASL
Service Name = kafka

The service name should match what is in the JAAS file above.

You don't need to do any of the other stuff related to ZooKeeper, unless your NiFi instance is also using a Kerberized ZooKeeper for NiFi's state management. The above steps are what you need to do only for NiFi talking to Kafka.

View solution in original post

5 REPLIES 5
Highlighted

Re: Error in NiFi Flow:

Are you saying that Kafka is kerberized?

Currently you have the Security Protocol set to PLAINTEXT which means an unsecure Kafka.

Highlighted

Re: Error in NiFi Flow:

Expert Contributor

Hi @Bryan Bende,

Thanks for the reply.

Yes I realized the error and I followed these steps,

https://community.hortonworks.com/articles/26551/accessing-kerberos-enabled-kafka-topics-using-getk....

Also I name my principle as "nifi/iotdemo.field.hortonworks.com@LAKE"

also do I need to mention these lines in my zookeeper.properties?

3. Added 3 additional properties to the bottom of the zookeeper.properties file you have configured per the linked procedure above:

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider

jaasLoginRenew=3600000

requireClientAuthScheme=sasl

Right now my error is:

"Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user"

Please find attached my PutKafka processor configurations.

Any help is highly appreciated..

screen-shot-2016-10-24-at-53412-pm.png

screen-shot-2016-10-24-at-53535-pm.png

Thanks a lot,

Sujitha

Highlighted

Re: Error in NiFi Flow:

Here is what you would need to do to configure PublishKafka to talk to a kerberized Kafka...

1) You can either rely on /etc/krb5.conf or you can tell NiFi to use a specific krb5.conf by setting nifi.kerberos.krb5.file= in nifi.properties to point to some other krb5.conf file.

2) Create a JAAS file lets say kafka-jaas.conf with the following:

KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  storeKey=true
  keyTab="/path/to/nifi-iotdemot.keytab"
  serviceName="kafka"
  principal="nifi/iotdemo.field.hortonworks.com@LAKE";
};

Changing the keyTab path to the appropriate path.

3) Specify this in NiFi's bootstrap.conf:

java.arg.15=-Djava.security.auth.login.config=/path/to/kafka-jaas.conf

4) Configure PublishKafka:

Security Protocol = PLAINTEXTSASL
Service Name = kafka

The service name should match what is in the JAAS file above.

You don't need to do any of the other stuff related to ZooKeeper, unless your NiFi instance is also using a Kerberized ZooKeeper for NiFi's state management. The above steps are what you need to do only for NiFi talking to Kafka.

View solution in original post

Highlighted

Re: Error in NiFi Flow:

New Contributor

Hi Bryan,

I follow the exact steps as mentioned by you but to no avail. I still get the ''kafka.pubsub.PublishKafka_0_10 PublishKafka_0_10[id=91c7ae98-0158-1000-aaa9-945b9c5c9f50] Failed while waiting for acks from Kafka" error. My nifi instance is standalone but configured to be secured. I am also wondering, how nifi will will create the key needed for connection to secured kafka?

Thanks for any help :)

Regards,

Shalini

Highlighted

Re: Error in NiFi Flow:

Can you start a new post describing your problem? Thanks.