question Re: NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling" in Support Questions

NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling"

Raj_B — Sun, 18 Aug 2019 02:08:36 GMT

Hello,

While I was trying to setup the HBase Client Service (disabling and enabling a few times, to edit the properties), I now see the service is stuck in "Disabling" state; when I select the only option that is now available, "Remove", it would not let me delete it, it says "cannot be deleted because it is not disabled"; I tried creating a second Client Service, but that too ended up in the same situation.

What I noticed is, even when enabling it, it never went to the "Enabled" state, it was showing "Enabling" for state.

Our HDP environment is Kerberized, and so I did provide the necessary information in the properties for HBase Client service. Initially, I did not supply the ZooKepper info, because the documentation says it's required only if the Hadoop Configuration Files are not supplied. But when it was showing an error ("The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master."), I added the ZooKepper info in the properties.

As a side note, PutHDFS works perfectly fine with the same Hadoop Configuration Files, Principal, and Keytab;

Please suggest how to make this work, thanks.

Re: NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling"

wbekker — Sat, 06 May 2017 02:03:25 GMT

Is there something in the NIFI log regarding the Hbase service?

Re: NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling"

Raj_B — Wed, 10 May 2017 01:51:03 GMT

@Ward Bekker sorry for the delay in replying.

It turned out that I was missing hbase-site.xml in the Hadoop configuration files, which I added; now the log shows a different error - "Failed to invoke @OnEnabled method due to java.io.IOException: java.lang.reflect.InvocationTargetException" ; it does not seem to be a login issue, because it says successfully logged in.

By the way, we're on NiFi 1.0.1

2017-05-09 13:34:47,645 ERROR [NiFi logging handler] org.apache.nifi.StdErr [StandardProcessScheduler Thread-2] INFO org.apache.nifi.hbase.HBase_1_1_2_ClientService - HBase_1_1_2_ClientService[id=102e119a-19a2-1409-671f-dddd93a063de] HBase Security Enabled, logging in as principal principal@principal.com with keytab /home/nifitest/kdc/nifitest.service.keytab
2017-05-09 13:34:47,663 ERROR [NiFi logging handler] org.apache.nifi.StdErr [StandardProcessScheduler Thread-2] INFO org.apache.nifi.hbase.HBase_1_1_2_ClientService - HBase_1_1_2_ClientService[id=102e119a-19a2-1409-671f-dddd93a063de] Successfully logged in as principal principal@principal.com with keytab /home/nifitest/kdc/nifitest.service.keytab
2017-05-09 13:34:47,664 ERROR [NiFi logging handler] org.apache.nifi.StdErr [StandardProcessScheduler Thread-2] ERROR org.apache.nifi.controller.service.StandardControllerServiceNode - HBase_1_1_2_ClientService[id=102e119a-19a2-1409-671f-dddd93a063de] Failed to invoke @OnEnabled method due to java.io.IOException: java.lang.reflect.InvocationTargetException
2017-05-09 13:34:47,664 ERROR [NiFi logging handler] org.apache.nifi.StdErr [StandardProcessScheduler Thread-2] ERROR org.apache.nifi.controller.service.StandardControllerServiceNode -
2017-05-09 13:34:47,664 ERROR [NiFi logging handler] org.apache.nifi.StdErr java.io.IOException: java.lang.reflect.InvocationTargetException
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.hbase.HBase_1_1_2_ClientService$1.run(HBase_1_1_2_ClientService.java:232)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.hbase.HBase_1_1_2_ClientService$1.run(HBase_1_1_2_ClientService.java:229)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.security.AccessController.doPrivileged(Native Method)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at javax.security.auth.Subject.doAs(Subject.java:422)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.hbase.HBase_1_1_2_ClientService.createConnection(HBase_1_1_2_ClientService.java:229)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.hbase.HBase_1_1_2_ClientService.onEnabled(HBase_1_1_2_ClientService.java:178)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at sun.reflect.GeneratedMethodAccessor706.invoke(Unknown Source)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.reflect.Method.invoke(Method.java:498)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:348)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.Thread.run(Thread.java:745)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: java.lang.reflect.InvocationTargetException
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at sun.reflect.GeneratedConstructorAccessor457.newInstance(Unknown Source)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	... 24 more
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: java.lang.UnsupportedOperationException: Unable to find org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:36)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.ipc.RpcControllerFactory.instantiate(RpcControllerFactory.java:58)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.createAsyncProcess(ConnectionManager.java:2242)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:690)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:630)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	... 28 more
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
2017-05-09 13:34:47,668 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.Class.forName0(Native Method)
2017-05-09 13:34:47,668 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.Class.forName(Class.java:264)
2017-05-09 13:34:47,668 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:32)
2017-05-09 13:34:47,668 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	... 32 more

Re: NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling"

wbekker — Wed, 10 May 2017 01:52:12 GMT

Great that you where able to solve it!

Re: NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling"

TimothySpann — Wed, 10 May 2017 02:19:32 GMT

Did you shut down and restart NIFI?

Also does the user running nifi have permissions to all the directories, files and kerberos.

Can you access hbase from the nifi machine using hbase cli or python?

Re: NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling"

bbende — Wed, 10 May 2017 04:20:50 GMT

Causedby: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory

This class is a Phoenix class, when you install Phoenix it sets this class name somewhere in hbase-site.xml and then when NiFi creates the HBase client, the HBase client is reading hbase-site.xml and trying to instantiate this class which doesn't exist in the HBase client.

In Apache NiFi 1.1.x there is a property on the HBase client service for specifying the location of the phoenix-client.jar. Setting that should solve this.

Re: NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling"

Raj_B — Wed, 10 May 2017 10:39:53 GMT

@Bryan Bende is this the interim solution until we upgrade to 1.1.x - https://community.hortonworks.com/questions/67074/nifi-hbase-service-controller-failing-in-nifi-10.html ?

Re: NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling"

Raj_B — Wed, 10 May 2017 10:51:59 GMT

@Timothy Spann Haven't restarted NiFi yet, that's what I was leaning towards, but wanted to wait before I tried that, in case there are other things that need to be done before the restart.

the account running NiFi doesn't have any issues connecting to and writing to HDP, we have another dataflow running that writes to HDFS on the same NiFi instance.

Re: NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling"

bbende — Wed, 10 May 2017 19:42:26 GMT

@Raj B

Yes that is work around when using versions before 1.1.x, just keep in mind if you ever blow away the work directory you have to make that change again.

Re: NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling"

Raj_B — Wed, 10 May 2017 20:39:25 GMT

@Bryan Bende thank you.

Deleting the NiFi's work directory is a manual step done deliberately to address some issue right ? it's not something that would occur in routine NiFi maintenance activities, like restarting the NiFi instance after OS patches, etc.

Also, I'm guessing the phoenix-client.jar would be found somewhere on the HDP installation; would you happen to know the path to find it, so I can let our Admins know.

Re: NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling"

bbende — Wed, 10 May 2017 20:42:11 GMT

Correct, it wouldn't happen from regular restarts.

I don't know exactly, but its something like /usr/hdp/current/phoenix

Re: NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling"

Raj_B — Wed, 10 May 2017 22:19:50 GMT

@Bryan Bende thank you.

Re: NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling"

Raj_B — Thu, 11 May 2017 02:21:32 GMT

@Bryan Bende, placed the phoenix jar file in NiFi's work directory path and restarted NiFi instance; now seeing a different error in the log when I enable the HBase client service - "org.apache.nifi.StdErr java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(Ljavax/security/auth/Subject;)Z"

Failed to invoke @OnEnabled method due to java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(Ljavax/security/auth/Subject;)Z
2017-05-10 14:13:50,016 ERROR [NiFi logging handler] org.apache.nifi.StdErr [StandardProcessScheduler Thread-6] ERROR org.apache.nifi.controller.service.StandardControllerServiceNode -
2017-05-10 14:13:50,016 ERROR [NiFi logging handler] org.apache.nifi.StdErr java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(Ljavax/security/auth/Subject;)Z
2017-05-10 14:13:50,016 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:623)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1200)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.hadoop.SecurityUtil.loginKerberos(SecurityUtil.java:52)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.hbase.HBase_1_1_2_ClientService.createConnection(HBase_1_1_2_ClientService.java:226)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.hbase.HBase_1_1_2_ClientService.onEnabled(HBase_1_1_2_ClientService.java:178)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2017-05-10 14:13:50,017 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.reflect.Method.invoke(Method.java:498)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:348)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2017-05-10 14:13:50,018 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
2017-05-10 14:13:50,019 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
2017-05-10 14:13:50,020 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017-05-10 14:13:50,020 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-05-10 14:13:50,020 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.Thread.run(Thread.java:745)
2017-05-10 14:13:50,020 ERROR [NiFi logging handler] org.apache.nifi.StdErr [StandardProcessScheduler Thread-6] ERROR org.apache.nifi.controller.service.StandardControllerServiceNode - Failed to invoke @OnEnabled method of HBase_1_1_2_ClientService[id=102e119a-19a2-1409-671f-dddd93a063de] due to java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosKeyTab(Ljavax/security/auth/Subject;)Z