Support Questions
Find answers, ask questions, and share your expertise

missing or invalid dependency detected while loading class file 'HBaseContext.class'

Trying to migrate code from Spark 1.6, Scala 2.10 to Spark 2.4, Scala 2.11, and having trouble compiling code that connects to HBase. Showing dependency versions, minimal example and compilation error below.

 

I understand that CDH 5 does not work with Spark 2. I can migrate to CDH 6.2.0 - but in that case, how can I connect to HBase? (Code samples, please.)

https://docs.cloudera.com/documentation/spark2/latest/topics/spark2_known_issues.html#ki_spark_on_hb...

 

 

 

 

// Dependencies
, "org.apache.spark" %% "spark-core" % "2.4.0"
, "org.apache.spark" %% "spark-sql"  % "2.4.0"
, "org.apache.hbase" % "hbase-server" % "1.2.0-cdh5.14.4"
, "org.apache.hbase" % "hbase-common" % "1.2.0-cdh5.14.4"
, "org.apache.hbase" % "hbase-spark"  % "1.2.0-cdh5.14.4"

// Minimal example
package spark2.hbase
import org.apache.hadoop.hbase.HBaseConfiguration
import org.apache.hadoop.hbase.spark.HBaseContext
import org.apache.spark.SparkContext
import org.apache.spark.sql.SparkSession
object ConnectToHBase {
  def main(args: Array[String]): Unit = {
    implicit val spark: SparkSession = SparkSession.builder.appName("Connect to HBase from Spark 2")
      .config("spark.master", "local")
      .getOrCreate()
    implicit val sc: SparkContext = spark.sparkContext
    val hbaseConf = HBaseConfiguration.create()
    val hbaseContext = new HBaseContext(sc, hbaseConf)
  }
}

// Compilation error
[error] missing or invalid dependency detected while loading class file 'HBaseContext.class'.
[error] Could not access type Logging in package org.apache.spark,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'HBaseContext.class' was compiled against an incompatible version of org.apache.spark.

 

 

 

1 REPLY 1

Cloudera Employee

1. CDH5 works with Spark2 by installing a separate parcel.

2. The document suggests that CDH5 + Spark2 does not support Spark-On-HBase. Spark-On-HBase work with CDH 5 and Spark 1.6

3. With CDH 6.x default Spark version is Spark 2.x+ so as a new feature Spark-On-HBase

https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_600_new_features.ht...

Spark On HBase now runs on top of Apache Spark 2.x.

4. Refer document below to use Spark-on-HBase on CDH6.x:

https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_hbase_import.html#concept_asc_...