New Contributor
Posts: 1
Registered: ‎02-08-2018

How to make HBase find scala-library?

[ Edited ]

 I'm on Spark 1.6.0, HBase 1.2.0 (cdh 5.7).


I can read an entire HBase table from the scala spark-shell just by doing:


user@node:~$ export SPARK_CLASSPATH=$(hbase classpath)
user@node:~$ spark-shell --master local


val df ="org.apache.hadoop.hbase.spark")
                              .option("hbase.columns.mapping", "rowkey STRING :key, anothercol STRING cf:anothercol")


But whenever I filter the rows to retrieve based on a string, such as by doing:


df.where("rowkey >= \"1-2018\"").show()

I get java.lang.NoClassDefFoundError: scala/collection/immutable/StringOps


This actually works fine with a numeric type, for instance, or if I cache the entire table upon loading it. Also notice that if I simply do 


import scala.collection.immutable.StringOps

the import runs fine. I suspect that because the hbase-spark library is part of HBase, this is just HBase not being able to find a scala-library.jar.


Is there any way I can make this class available to hbase-spark?



Posts: 1,657
Kudos: 320
Solutions: 258
Registered: ‎07-31-2013

Re: How to make HBase find scala-library?

Do you face this even with --master yarn? I've not been able to reproduce this on recent CDH versions, but I only tried with --master yarn.