Hbase custom filter

[#1]

I have created a custom filter, and loaded into Hbase class path of master server, but it is throwing the below error

I tried loading filter into hbase-site.xml ny providing HDFS jar directory in hbase.dynamic.jars.dir , But still getting same error

Please help me in resolving this issue

Error:-

xception in thread "main" org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1468) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toGet(ProtobufUtil.java:483) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1933) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32201) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1464) ... 8 more Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: hbase.CustomFilter at org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:396) ... 13 more Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: hbase.CustomFilter at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1468) at org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:393) ... 13 more Caused by: java.lang.ClassNotFoundException: hbase.CustomFilter at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at org.apache.hadoop.hbase.util.DynamicClassLoader.tryRefreshClass(DynamicClassLoader.java:171) at org.apache.hadoop.hbase.util.DynamicClassLoader.loadClass(DynamicClassLoader.java:138) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1458) ... 14 more

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:326) at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:858) at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:845) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:862) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:828) at hbase.test_filter_timestamp.main(test_filter_timestamp.java:99) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException): org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1468) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toGet(ProtobufUtil.java:483) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1933) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32201) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1464) ... 8 more Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: hbase.CustomFilter at org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:396) ... 13 more Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: hbase.CustomFilter at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1468) at org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:393) ... 13 more Caused by: java.lang.ClassNotFoundException: hbase.CustomFilter at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at org.apache.hadoop.hbase.util.DynamicClassLoader.tryRefreshClass(DynamicClassLoader.java:171) at org.apache.hadoop.hbase.util.DynamicClassLoader.loadClass(DynamicClassLoader.java:138) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1458) ... 14 more

at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1225) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:32627) at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:854) ... 11 more

[#2]

@Ashok Kumar BM

Your jar file is not in the classpath. What is the value of property hbase.dynamic.jars.dir? You are not supposed to add your jar to master server but to the location pointed by this property. That's your problem.

[#3]

the value of hbase.dynamic.jars.dir? is hdfs://azu021016c72-01:8020/tmp/cstjars

and it contain hload-0.0.1-Hload.jar which contain the customfiler.java and FilterProtos.java classes

[#4]

@Ashok Kumar BM

I deleted my earlier comment because I didn't realize, that patch was old. While, this should work, is it possible for you to add the jar to HBase_CLASSPATH and restart HBase service (not something you should be doing normally but since there is an issue, can you try this)?

HBASE_CLASSPATH=<path to your custom jar on local file system on all region servers>

[#5]

What is the value of hbase.use.dynamic.jar. If this is false, you need to set this to true.

[#6]

yes I have tried that , i have set the HBASE_CALSSPATH value as

export HBASE_CLASSPATH="/root/hload-0.0.1-Hload.jar"

And places the custom filter jar in /root directory of all 3 region servers

still getting the same error, Is there any way i can check the my custom filter jar is load to hbase or not?

or if you have any doc which explain how we can load custom filter jars to Hartonworks hbase running on Ambari server

Error:-

Exception in thread "main" org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1468) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:995) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2255) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1464) ... 8 more Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: hbase.CustomFilter at org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:396) ... 13 more Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: hbase.CustomFilter at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1468) at org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:393) ... 13 more Caused by: java.lang.ClassNotFoundException: hbase.CustomFilter at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at org.apache.hadoop.hbase.util.DynamicClassLoader.tryRefreshClass(DynamicClassLoader.java:171) at org.apache.hadoop.hbase.util.DynamicClassLoader.loadClass(DynamicClassLoader.java:138) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1458) ... 14 more at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:326) at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:380) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:356) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:330) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException): org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1468) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:995) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2255) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1464) ... 8 more Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: hbase.CustomFilter at org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:396) ... 13 more Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: hbase.CustomFilter at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1468) at org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:393) ... 13 more Caused by: java.lang.ClassNotFoundException: hbase.CustomFilter at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at org.apache.hadoop.hbase.util.DynamicClassLoader.tryRefreshClass(DynamicClassLoader.java:171) at org.apache.hadoop.hbase.util.DynamicClassLoader.loadClass(DynamicClassLoader.java:138) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1458) ... 14 more at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1225) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651) at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:372) ... 10 more

[#7 · Accepted solution]

The HBase user probably can't read "/root" (unless you're running HBase as root which is a really bad idea if you are..).

Just copy the jar in to the lib directory for the HBase installation.

[#8]

Thanks Josh elser, have copied the jar to lib directory and restarted zookeeper

Now the custom filters working fine

Thank alot