Our Nifi flow has processors that interact with other components of the cluster, in this case, Hbase. When Nifi trying to get data from Hbase to further enrich the flowfile, we see that the LookupAttribute processor opens a thread to get data from the Hbase, but it does not receive the data as the process hangs.
From the Nifi User Interface, we don't see any error/info/warn messages or messages about processor misconfigurations, but we see that the processor is working.
Our Nifi flow works well after restarting Nifi, or after deactivating and activating the KeytabCredentialsService. This fact confirms the correct configuration of Nifi-processors and Nifi-services. We think, that problem maybe is in tickets renew.
In the log files, we see the following (hostname and ip-address are masked by author):
nifi-app.log
2020-10-12 14:22:16,986 INFO org.apache.hadoop.hbase.client.RpcRetryingCallerImpl: Call exception, tries=11, retries=31, started=48453 ms ago, cancelled=false, msg=Call to {hostname}/{ip_address} failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)], details=row ‘hb_lp01:tb62_campaign_date,20191101,99999999999999’ on table ‘hbase:meta’ at region=hbase:meta,,1.1588230740, hostname={hostname},16020,1602153441094, seqNum=-1, see https://s.apache.org/timeout
nifi-bootstrap.log
2020-10-12 14:22:16,988 ERROR [NiFi logging handler] org.apache.nifi.StdErr java.net.SocketTimeoutException: callTimeout=60000, callDuration=68516: Call to {hostname}/{ip_address} failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] row ‘hb_lp01:tb62_campaign_date,20191101,99999999999999’ on table ‘hbase:meta’ at region=hbase:meta,,1.1588230740, hostname={hostname},16020,1602153441094, seqNum=-1
2020-10-12 14:22:16,989 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: javax.security.sasl.SaslException: Call to {hostname}/{ip_address} failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] [Caused by javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]]
2020-10-12 14:22:16,991 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2020-10-12 14:22:16,992 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
Flow configuration :
LookupAttribute processor receives incoming flowfile and opens a connection to hbase to receive data:
LookupAttribute processor configuration :
DistributedMapCacheLookupService configuration:
Hbase_2_ClientMapCacheService configuration:
Hbase_2_ClientService configuration:
KeytabCredentialsService configuration:
Stack versions:
CDP - 7.1.3-1.cdh7.1.3.p0.4992530
CFM 2.0.1.0-71
