Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Issue with auto-renewal of kerberos ticket (Nifi)

avatar
New Contributor

Our Nifi flow has processors that interact with other components of the cluster, in this case, Hbase. When Nifi trying to get data from Hbase to further enrich the flowfile, we see that the LookupAttribute processor opens a thread to get data from the Hbase, but it does not receive the data as the process hangs.

 

From the Nifi User Interface, we don't see any error/info/warn messages or messages about processor misconfigurations, but we see that the processor is working.

 

Our Nifi flow works well after restarting Nifi, or after deactivating and activating the KeytabCredentialsService. This fact confirms the correct configuration of Nifi-processors and Nifi-services. We think, that problem maybe is in tickets renew.

 

In the log files, we see the following (hostname and ip-address are masked by author):

 

nifi-app.log

2020-10-12 14:22:16,986 INFO org.apache.hadoop.hbase.client.RpcRetryingCallerImpl: Call exception, tries=11, retries=31, started=48453 ms ago, cancelled=false, msg=Call to {hostname}/{ip_address} failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)], details=row ‘hb_lp01:tb62_campaign_date,20191101,99999999999999’ on table ‘hbase:meta’ at region=hbase:meta,,1.1588230740, hostname={hostname},16020,1602153441094, seqNum=-1, see https://s.apache.org/timeout

 

nifi-bootstrap.log

2020-10-12 14:22:16,988 ERROR [NiFi logging handler] org.apache.nifi.StdErr java.net.SocketTimeoutException: callTimeout=60000, callDuration=68516: Call to {hostname}/{ip_address} failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] row ‘hb_lp01:tb62_campaign_date,20191101,99999999999999’ on table ‘hbase:meta’ at region=hbase:meta,,1.1588230740, hostname={hostname},16020,1602153441094, seqNum=-1

2020-10-12 14:22:16,989 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: javax.security.sasl.SaslException: Call to {hostname}/{ip_address} failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] [Caused by javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]]

2020-10-12 14:22:16,991 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

2020-10-12 14:22:16,992 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

 

 

Flow configuration :

 

LookupAttribute processor receives incoming flowfile  and opens a connection to hbase to receive data:

photo_2020-10-13 09.51.16.jpeg

 

LookupAttribute processor configuration :

photo_2020-10-13 09.52.18.jpeg

 

DistributedMapCacheLookupService configuration:

photo_2020-10-13 09.53.24.jpeg

 

Hbase_2_ClientMapCacheService configuration:

photo_2020-10-13 09.54.18.jpeg

 

Hbase_2_ClientService configuration:

photo_2020-10-13 09.55.32.jpeg

 

KeytabCredentialsService configuration:

photo_2020-10-13 09.56.15.jpeg

 

Stack versions:

CDP - 7.1.3-1.cdh7.1.3.p0.4992530

CFM 2.0.1.0-71

4 REPLIES 4

avatar
Super Mentor

@nikolayburiak 

Have you tried defining the keytab and principal directly in the the Hbase_2_ClientService configuration rather than using the KeytabCredentialsService to see if ticket renewal works correctly?

This may get you pas the issue now and also help identify if issue is potentially with the controller services.

 

Thanks,

Matt

avatar
New Contributor

Hello, has this problem been solved? I also encountered a similar problem, need help urgently!!! Thanks

avatar
Rising Star

Hi @nikolayburiak , were you able to fix it? I am also facing the same issue where the HBase_2_ClientService cannot renew the Kerberos ticket on its on. I have tried defining the keytab and principal directly in the service but to no success. Interestingly, even if I renew(kinit) or destroy(kdestroy) the Kerberos ticket from the command line on the node running Nifi/Hadoop client, it doesn't have any effect on the client service or the Puthbase processor. So, I am not sure how Nifi creates a kerbros ticket? 

The only workaround is restarting the HBase_2_ClientService .

I am using Nifi 1.12 with HDP 3.14

avatar
Super Mentor

@te04_0172 

 

It appears you have hit a known issue:
https://issues.apache.org/jira/browse/NIFI-7954
https://issues.apache.org/jira/browse/NIFI-7831

Looks like these will be addressed in Apache NiFi 1.13

These fixes have already been incorporated in to the Cloudera HDF 3.5.2 release that is currently available.

Hope this helps,

Matt