Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Integration with Hbase and spark

Integration with Hbase and spark

Explorer

I am new to scala based sbt build. I am not sure whether it is correct build.sbt for given scala code. Please correct me as well

When i am trying to connect Hbase with spark. using scala code. Unable to do so  because of sbt file error. 

 

 I stucked at sbt only. Aim is to add data using scala code programtic manner ( for CDH 5.13 version)

 

Version  

Hbase -   1.2.0-cdh5.13.0

Cloudera  - cdh5.13.0

 

Getting error :

assemblyMergeStrategy Portion is unreolvable. Pathlist as well

 

 

Build.sbt 

 

name := "ScalaHBase"

version := "1.0"

scalaVersion := "2.11.8"

resolvers ++= Seq(
  "Hadoop Releases" at "https://repository.cloudera.com/content/repositories/releases/"
)

libraryDependencies ++= Seq(
  "com.google.guava" % "guava" % "15.0",
  "org.apache.hadoop" % "hadoop-common" % "2.6.0",
  "org.apache.hadoop" % "hadoop-mapred" % "0.22.0",
  "org.apache.hbase" % "hbase-common" % "1.0.0",
  "org.apache.hbase" % "hbase-client" % "1.0.0"
)

dependencyOverrides += "com.google.guava" % "guava" % "15.0"


assemblyMergeStrategy in assembly := {
  case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
  case PathList("javax", "activation", xs @ _*) => MergeStrategy.last
  case PathList("org", "apache", xs @ _*) => MergeStrategy.last
  case PathList("com", "google", xs @ _*) => MergeStrategy.first
  case PathList("com", "yammer", xs @ _*) => MergeStrategy.last
  case "about.html" => MergeStrategy.rename
  case "plugin.properties" => MergeStrategy.last
  case "log4j.properties" => MergeStrategy.last
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

 

 

Scala code 

 

import org.apache.hadoop.hbase.client._
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.hbase.{CellUtil, HBaseConfiguration, TableName}
import org.apache.hadoop.conf.Configuration
import scala.collection.JavaConverters._

object ScalaHBaseExample extends App{

  def printRow(result : Result) = {
    val cells = result.rawCells();
    print( Bytes.toString(result.getRow) + " : " )
    for(cell <- cells){
      val col_name = Bytes.toString(CellUtil.cloneQualifier(cell))
      val col_value = Bytes.toString(CellUtil.cloneValue(cell))
      print("(%s,%s) ".format(col_name, col_value))
    }
    println()
  }

  val conf : Configuration = HBaseConfiguration.create()
  
  val ZOOKEEPER_QUORUM = "WRITE THE ZOOKEEPER CLUSTER THAT HBASE SHOULD USE"
  conf.set("hbase.zookeeper.quorum", ZOOKEEPER_QUORUM);

  val connection = ConnectionFactory.createConnection(conf)
  val table = connection.getTable(TableName.valueOf( Bytes.toBytes("emostafa:test_table") ) )

  // Put example
  var put = new Put(Bytes.toBytes("row1"))
  put.addColumn(Bytes.toBytes("d"), Bytes.toBytes("test_column_name"), Bytes.toBytes("test_value"))
  put.addColumn(Bytes.toBytes("d"), Bytes.toBytes("test_column_name2"), Bytes.toBytes("test_value2"))
  table.put(put)

  // Get example
  println("Get Example:")
  var get = new Get(Bytes.toBytes("row1"))
  var result = table.get(get)
  printRow(result)

  //Scan example
  println("\nScan Example:")
  var scan = table.getScanner(new Scan())
  scan.asScala.foreach(result => {
      printRow(result)
  })
  
  table.close()
  connection.close()
}

 

 

Also I tried  this Build.sbt file 

 

 

 

name := "HbaseExample"

version := "0.1"

scalaVersion := "2.12.5"

libraryDependencies ++= Seq(
//"org.apache.hbase" % "hbase" % "0.98.8-hadoop2",
"org.apache.hbase" % "hbase" % "1.2.0-cdh5.13.0" pomOnly(),

//"org.apache.hbase" % "hbase-client" % "0.98.8-hadoop2",
"org.apache.hbase" % "hbase-client" % "1.2.0-cdh5.13.0",
// "org.apache.hbase" % "hbase-common" % "0.98.8-hadoop2",
"org.apache.hbase" % "hbase-common" % "1.2.0-cdh5.13.0",
//"org.apache.hbase" % "hbase-server" % "0.98.8-hadoop2",
"org.apache.hbase" % "hbase-server" % "1.2.0-cdh5.13.0"
)

mergeStrategy in assembly := {
case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
case m if m.toLowerCase.matches("meta-inf.*\\.sf$") => MergeStrategy.discard
case "META-INF/jersey-module-version" => MergeStrategy.first
case _ => MergeStrategy.first
}

 

when trying to import this build.sbt again this portion : saying cannot resolve symbol 

 

 

 

 

 

 

 

 

 

 

 

Any help on it would be much appreciated ?