I'm on Spark 1.6.0, HBase 1.2.0 (cdh 5.7).
I can read an entire HBase table from the scala spark-shell just by doing:
user@node:~$ export SPARK_CLASSPATH=$(hbase classpath) user@node:~$ spark-shell --master local
val df = sqlContext.read.format("org.apache.hadoop.hbase.spark") .option("hbase.table","test_table") .option("hbase.columns.mapping", "rowkey STRING :key, anothercol STRING cf:anothercol") .load() df.show()
But whenever I filter the rows to retrieve based on a string, such as by doing:
df.where("rowkey >= \"1-2018\"").show()
I get java.lang.NoClassDefFoundError: scala/collection/immutable/StringOps
This actually works fine with a numeric type, for instance, or if I cache the entire table upon loading it. Also notice that if I simply do
the import runs fine. I suspect that because the hbase-spark library is part of HBase, this is just HBase not being able to find a scala-library.jar.
Is there any way I can make this class available to hbase-spark?
I am also facing the exact issue. I tried master as yarn too but still the issue persists. Were you able to resolve this? If so please let me know the solution