Reply
Neo
Explorer
Posts: 7
Registered: ‎03-09-2018

Integration with Hbase and spark

[ Edited ]

I am new to scala based sbt build. I am not sure whether it is correct build.sbt for given scala code. Please correct me as well

When i am trying to connect Hbase with spark. using scala code. Unable to do so  because of sbt file error. 

 

 I stucked at sbt only. Aim is to add data using scala code programtic manner ( for CDH 5.13 version)

 

Version  

Hbase -   1.2.0-cdh5.13.0

Cloudera  - cdh5.13.0

 

Getting error :

assemblyMergeStrategy Portion is unreolvable. Pathlist as well

 

 

Build.sbt 

 

name := "ScalaHBase"

version := "1.0"

scalaVersion := "2.11.8"

resolvers ++= Seq(
  "Hadoop Releases" at "https://repository.cloudera.com/content/repositories/releases/"
)

libraryDependencies ++= Seq(
  "com.google.guava" % "guava" % "15.0",
  "org.apache.hadoop" % "hadoop-common" % "2.6.0",
  "org.apache.hadoop" % "hadoop-mapred" % "0.22.0",
  "org.apache.hbase" % "hbase-common" % "1.0.0",
  "org.apache.hbase" % "hbase-client" % "1.0.0"
)

dependencyOverrides += "com.google.guava" % "guava" % "15.0"


assemblyMergeStrategy in assembly := {
  case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
  case PathList("javax", "activation", xs @ _*) => MergeStrategy.last
  case PathList("org", "apache", xs @ _*) => MergeStrategy.last
  case PathList("com", "google", xs @ _*) => MergeStrategy.first
  case PathList("com", "yammer", xs @ _*) => MergeStrategy.last
  case "about.html" => MergeStrategy.rename
  case "plugin.properties" => MergeStrategy.last
  case "log4j.properties" => MergeStrategy.last
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

 

 

Scala code 

 

import org.apache.hadoop.hbase.client._
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.hbase.{CellUtil, HBaseConfiguration, TableName}
import org.apache.hadoop.conf.Configuration
import scala.collection.JavaConverters._

object ScalaHBaseExample extends App{

  def printRow(result : Result) = {
    val cells = result.rawCells();
    print( Bytes.toString(result.getRow) + " : " )
    for(cell <- cells){
      val col_name = Bytes.toString(CellUtil.cloneQualifier(cell))
      val col_value = Bytes.toString(CellUtil.cloneValue(cell))
      print("(%s,%s) ".format(col_name, col_value))
    }
    println()
  }

  val conf : Configuration = HBaseConfiguration.create()
  
  val ZOOKEEPER_QUORUM = "WRITE THE ZOOKEEPER CLUSTER THAT HBASE SHOULD USE"
  conf.set("hbase.zookeeper.quorum", ZOOKEEPER_QUORUM);

  val connection = ConnectionFactory.createConnection(conf)
  val table = connection.getTable(TableName.valueOf( Bytes.toBytes("emostafa:test_table") ) )

  // Put example
  var put = new Put(Bytes.toBytes("row1"))
  put.addColumn(Bytes.toBytes("d"), Bytes.toBytes("test_column_name"), Bytes.toBytes("test_value"))
  put.addColumn(Bytes.toBytes("d"), Bytes.toBytes("test_column_name2"), Bytes.toBytes("test_value2"))
  table.put(put)

  // Get example
  println("Get Example:")
  var get = new Get(Bytes.toBytes("row1"))
  var result = table.get(get)
  printRow(result)

  //Scan example
  println("\nScan Example:")
  var scan = table.getScanner(new Scan())
  scan.asScala.foreach(result => {
      printRow(result)
  })
  
  table.close()
  connection.close()
}

 

 

Also I tried  this Build.sbt file 

 

 

 

name := "HbaseExample"

version := "0.1"

scalaVersion := "2.12.5"

libraryDependencies ++= Seq(
//"org.apache.hbase" % "hbase" % "0.98.8-hadoop2",
"org.apache.hbase" % "hbase" % "1.2.0-cdh5.13.0" pomOnly(),

//"org.apache.hbase" % "hbase-client" % "0.98.8-hadoop2",
"org.apache.hbase" % "hbase-client" % "1.2.0-cdh5.13.0",
// "org.apache.hbase" % "hbase-common" % "0.98.8-hadoop2",
"org.apache.hbase" % "hbase-common" % "1.2.0-cdh5.13.0",
//"org.apache.hbase" % "hbase-server" % "0.98.8-hadoop2",
"org.apache.hbase" % "hbase-server" % "1.2.0-cdh5.13.0"
)

mergeStrategy in assembly := {
case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
case m if m.toLowerCase.matches("meta-inf.*\\.sf$") => MergeStrategy.discard
case "META-INF/jersey-module-version" => MergeStrategy.first
case _ => MergeStrategy.first
}

 

when trying to import this build.sbt again this portion : saying cannot resolve symbol 

 

 

 

 

 

 

 

 

 

 

 

Any help on it would be much appreciated ?

Announcements