Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Issue with auto-renewal of kerberos ticket (Nifi)

Highlighted

Issue with auto-renewal of kerberos ticket (Nifi)

New Contributor

Our Nifi flow has processors that interact with other components of the cluster, in this case, Hbase. When Nifi trying to get data from Hbase to further enrich the flowfile, we see that the LookupAttribute processor opens a thread to get data from the Hbase, but it does not receive the data as the process hangs.

 

From the Nifi User Interface, we don't see any error/info/warn messages or messages about processor misconfigurations, but we see that the processor is working.

 

Our Nifi flow works well after restarting Nifi, or after deactivating and activating the KeytabCredentialsService. This fact confirms the correct configuration of Nifi-processors and Nifi-services. We think, that problem maybe is in tickets renew.

 

In the log files, we see the following (hostname and ip-address are masked by author):

 

nifi-app.log

2020-10-12 14:22:16,986 INFO org.apache.hadoop.hbase.client.RpcRetryingCallerImpl: Call exception, tries=11, retries=31, started=48453 ms ago, cancelled=false, msg=Call to {hostname}/{ip_address} failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)], details=row ‘hb_lp01:tb62_campaign_date,20191101,99999999999999’ on table ‘hbase:meta’ at region=hbase:meta,,1.1588230740, hostname={hostname},16020,1602153441094, seqNum=-1, see https://s.apache.org/timeout

 

nifi-bootstrap.log

2020-10-12 14:22:16,988 ERROR [NiFi logging handler] org.apache.nifi.StdErr java.net.SocketTimeoutException: callTimeout=60000, callDuration=68516: Call to {hostname}/{ip_address} failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] row ‘hb_lp01:tb62_campaign_date,20191101,99999999999999’ on table ‘hbase:meta’ at region=hbase:meta,,1.1588230740, hostname={hostname},16020,1602153441094, seqNum=-1

2020-10-12 14:22:16,989 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: javax.security.sasl.SaslException: Call to {hostname}/{ip_address} failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] [Caused by javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]]

2020-10-12 14:22:16,991 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

2020-10-12 14:22:16,992 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

 

 

Flow configuration :

 

LookupAttribute processor receives incoming flowfile  and opens a connection to hbase to receive data:

photo_2020-10-13 09.51.16.jpeg

 

LookupAttribute processor configuration :

photo_2020-10-13 09.52.18.jpeg

 

DistributedMapCacheLookupService configuration:

photo_2020-10-13 09.53.24.jpeg

 

Hbase_2_ClientMapCacheService configuration:

photo_2020-10-13 09.54.18.jpeg

 

Hbase_2_ClientService configuration:

photo_2020-10-13 09.55.32.jpeg

 

KeytabCredentialsService configuration:

photo_2020-10-13 09.56.15.jpeg

 

Stack versions:

CDP - 7.1.3-1.cdh7.1.3.p0.4992530

CFM 2.0.1.0-71

2 REPLIES 2
Highlighted

Re: Issue with auto-renewal of kerberos ticket (Nifi)

Master Guru

@nikolayburiak 

Have you tried defining the keytab and principal directly in the the Hbase_2_ClientService configuration rather than using the KeytabCredentialsService to see if ticket renewal works correctly?

This may get you pas the issue now and also help identify if issue is potentially with the controller services.

 

Thanks,

Matt

Highlighted

Re: Issue with auto-renewal of kerberos ticket (Nifi)

New Contributor

Hello, has this problem been solved? I also encountered a similar problem, need help urgently!!! Thanks

Don't have an account?
Coming from Hortonworks? Activate your account here