Reply
Highlighted
New Contributor
Posts: 1
Registered: ‎05-15-2017

Custom hbase filter jar not loaded

[ Edited ]

Hello,

 

I am trying to deploy a custom filter to my hbase cluster. According to the cloudera docs [1] dynamic loading of filter jars should be enabled by default. 

 

My hbase.rootdir is /hbase

 

To deploy my custom filter jar I created the directory /hbase/lib in hdfs and put my jar in it. Then I tried to use the custom filter from a spark-hbase job:

object MyFilterTest {
  def main(args: Array[String]): Unit = {
    [...]
    val filter = new MyFilter()

    val scan = new Scan()
    scan.setFilter(filter)

    try {
      val rdd = hbaseContext.hbaseRDD(TableName.valueOf("some_table"), scan)
      val rowKeys = rdd.map(tuple => Bytes.toString(tuple._1.get))
      rowKeys.saveAsTextFile(outputPath)
    } finally {
      sparkContext.stop()
    }
  }
}

But it failed with a ClassNotFoundException stating that MyFilter could not be found.

To investigate I set the log level to DEBUG and look for org.apache.hbase.util.DynamicClassLoader entries in the logs, but did not find any at all. Intrigued by this fact I tried to set the relevant settings explicitly. In the hbase-site.xml safety valve I put this config:

<property>
  <name>hbase.dynamic.jars.dir</name>
  <value>hdfs://myhdfs.host:8020/hbase/lib</value>
  <final>true</final>
</property>
<property>
  <name>hbase.local.dir</name>
  <value>/tmp/hbase-root/local</value>
  <final>true</final>
</property>
<property>
  <name>hbase.use.dynamic.jars</name>
  <value>true</value>
  <final>true</final>
</property>

But still, it didn't work and there were no logs indicating that hbase even tried to load the jars from hdfs.

 

Is there any configuration I forgot?

 

 

[1] https://www.cloudera.com/documentation/enterprise/5-8-x/topics/admin_hbase_filtering.html

Announcements