NiFi HBase_1_1_2_ClientService setup issue - stuck on "Disabling"

[#1]

Hello,

While I was trying to setup the HBase Client Service (disabling and enabling a few times, to edit the properties), I now see the service is stuck in "Disabling" state; when I select the only option that is now available, "Remove", it would not let me delete it, it says "cannot be deleted because it is not disabled"; I tried creating a second Client Service, but that too ended up in the same situation.

What I noticed is, even when enabling it, it never went to the "Enabled" state, it was showing "Enabling" for state.

Our HDP environment is Kerberized, and so I did provide the necessary information in the properties for HBase Client service. Initially, I did not supply the ZooKepper info, because the documentation says it's required only if the Hadoop Configuration Files are not supplied. But when it was showing an error ("The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master."), I added the ZooKepper info in the properties.

As a side note, PutHDFS works perfectly fine with the same Hadoop Configuration Files, Principal, and Keytab;

Please suggest how to make this work, thanks.

[#2]

Is there something in the NIFI log regarding the Hbase service?

[#3]

@Ward Bekker sorry for the delay in replying.

It turned out that I was missing hbase-site.xml in the Hadoop configuration files, which I added; now the log shows a different error - "Failed to invoke @OnEnabled method due to java.io.IOException: java.lang.reflect.InvocationTargetException" ; it does not seem to be a login issue, because it says successfully logged in.

By the way, we're on NiFi 1.0.1

2017-05-09 13:34:47,645 ERROR [NiFi logging handler] org.apache.nifi.StdErr [StandardProcessScheduler Thread-2] INFO org.apache.nifi.hbase.HBase_1_1_2_ClientService - HBase_1_1_2_ClientService[id=102e119a-19a2-1409-671f-dddd93a063de] HBase Security Enabled, logging in as principal principal@principal.com with keytab /home/nifitest/kdc/nifitest.service.keytab
2017-05-09 13:34:47,663 ERROR [NiFi logging handler] org.apache.nifi.StdErr [StandardProcessScheduler Thread-2] INFO org.apache.nifi.hbase.HBase_1_1_2_ClientService - HBase_1_1_2_ClientService[id=102e119a-19a2-1409-671f-dddd93a063de] Successfully logged in as principal principal@principal.com with keytab /home/nifitest/kdc/nifitest.service.keytab
2017-05-09 13:34:47,664 ERROR [NiFi logging handler] org.apache.nifi.StdErr [StandardProcessScheduler Thread-2] ERROR org.apache.nifi.controller.service.StandardControllerServiceNode - HBase_1_1_2_ClientService[id=102e119a-19a2-1409-671f-dddd93a063de] Failed to invoke @OnEnabled method due to java.io.IOException: java.lang.reflect.InvocationTargetException
2017-05-09 13:34:47,664 ERROR [NiFi logging handler] org.apache.nifi.StdErr [StandardProcessScheduler Thread-2] ERROR org.apache.nifi.controller.service.StandardControllerServiceNode -
2017-05-09 13:34:47,664 ERROR [NiFi logging handler] org.apache.nifi.StdErr java.io.IOException: java.lang.reflect.InvocationTargetException
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.hbase.HBase_1_1_2_ClientService$1.run(HBase_1_1_2_ClientService.java:232)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.hbase.HBase_1_1_2_ClientService$1.run(HBase_1_1_2_ClientService.java:229)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.security.AccessController.doPrivileged(Native Method)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at javax.security.auth.Subject.doAs(Subject.java:422)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.hbase.HBase_1_1_2_ClientService.createConnection(HBase_1_1_2_ClientService.java:229)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.hbase.HBase_1_1_2_ClientService.onEnabled(HBase_1_1_2_ClientService.java:178)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at sun.reflect.GeneratedMethodAccessor706.invoke(Unknown Source)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.reflect.Method.invoke(Method.java:498)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137)
2017-05-09 13:34:47,665 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:348)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.Thread.run(Thread.java:745)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: java.lang.reflect.InvocationTargetException
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at sun.reflect.GeneratedConstructorAccessor457.newInstance(Unknown Source)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
2017-05-09 13:34:47,666 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	... 24 more
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: java.lang.UnsupportedOperationException: Unable to find org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:36)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.ipc.RpcControllerFactory.instantiate(RpcControllerFactory.java:58)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.createAsyncProcess(ConnectionManager.java:2242)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:690)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:630)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	... 28 more
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
2017-05-09 13:34:47,667 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
2017-05-09 13:34:47,668 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.Class.forName0(Native Method)
2017-05-09 13:34:47,668 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at java.lang.Class.forName(Class.java:264)
2017-05-09 13:34:47,668 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:32)
2017-05-09 13:34:47,668 ERROR [NiFi logging handler] org.apache.nifi.StdErr 	... 32 more 

[#4]

Great that you where able to solve it!

[#5]

Did you shut down and restart NIFI?

Also does the user running nifi have permissions to all the directories, files and kerberos.

Can you access hbase from the nifi machine using hbase cli or python?

[#6]

@Timothy Spann Haven't restarted NiFi yet, that's what I was leaning towards, but wanted to wait before I tried that, in case there are other things that need to be done before the restart.

the account running NiFi doesn't have any issues connecting to and writing to HDP, we have another dataflow running that writes to HDFS on the same NiFi instance.

[#7 · Accepted solution]

Causedby: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory

This class is a Phoenix class, when you install Phoenix it sets this class name somewhere in hbase-site.xml and then when NiFi creates the HBase client, the HBase client is reading hbase-site.xml and trying to instantiate this class which doesn't exist in the HBase client.

In Apache NiFi 1.1.x there is a property on the HBase client service for specifying the location of the phoenix-client.jar. Setting that should solve this.

[#8]

@Bryan Bende is this the interim solution until we upgrade to 1.1.x - https://community.hortonworks.com/questions/67074/nifi-hbase-service-controller-failing-in-nifi-10.h... ?

[#9]

@Raj B

Yes that is work around when using versions before 1.1.x, just keep in mind if you ever blow away the work directory you have to make that change again.

[#10]

@Bryan Bende thank you.

Deleting the NiFi's work directory is a manual step done deliberately to address some issue right ? it's not something that would occur in routine NiFi maintenance activities, like restarting the NiFi instance after OS patches, etc.

Also, I'm guessing the phoenix-client.jar would be found somewhere on the HDP installation; would you happen to know the path to find it, so I can let our Admins know.