question Spark job reeturns empty rows from HBase in Archives of Support Questions (Read Only)

Spark job reeturns empty rows from HBase

contactvivekjai — Tue, 03 Jul 2018 22:17:19 GMT

Hi Community,

I'm running a basic spark job which reads from an HBase table.

I can see the job is getting complete without any error, but in output I get the empty rows.

Will appreciate any help.

Below is my code

object objectName {
  def catalog = s"""{
         |"table":{"namespace":"namespaceName", "name":"tableName"},
         |"rowkey":"rowKeyAttribute",
         |"columns":{
           |"Key":{"cf":"rowkey", "col":"rowKeyAttribute", "type":"string"},
           |"col1":{"cf":"cfName", "col":"col1", "type":"bigint"},
           |"col2":{"cf":"cfName", "col":"col2", "type":"string"}
          |}
       |}""".stripMargin

  def main(args: Array[String]) {
 
    val spark = SparkSession.builder()
      .appName("dummyApplication")
      .getOrCreate()

    val sc = spark.sparkContext
    val sqlContext = spark.sqlContext   
  
    import sqlContext.implicits._  

    def withCatalog(cat: String): DataFrame = {
      sqlContext
        .read
        .options(Map(HBaseTableCatalog.tableCatalog -> cat))
        .format("org.apache.spark.sql.execution.datasources.hbase")
        .load()
    }


}

Re: Spark job reeturns empty rows from HBase

falbani — Tue, 03 Jul 2018 23:52:23 GMT

@vivek jain

I dont see any code making use of withCatalog function. If this function is not beeing used what is the expected output?

As an example perhaps you could try adding something like this to show some of the content of the hbase table:

val df = withCatalog(catalog)
df.show()

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Re: Spark job reeturns empty rows from HBase

Former Member — Wed, 04 Jul 2018 00:35:30 GMT

Did you check out the docs?

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_spark-component-guide/content/spark-on-hbase.html

Did you look at this other HCC post on a similar topic:

https://community.hortonworks.com/questions/49743/read-hbase-table-by-using-sparkscala.html

Re: Spark job reeturns empty rows from HBase

contactvivekjai — Wed, 04 Jul 2018 02:15:40 GMT

Hi @Felix Albani thanks for your response. Please accept my sincere apologies I somehow missed to include that part of the code. I have updated now.

This is the output I see(Please note that I have changed the number of columns in above code, hence the difference).

+----+----+----+----+----+----+----+----+----+ |col4|col7|col1|col3|col6|col0|col8|col2|col5| +----+----+----+----+----+----+----+----+----+ +----+----+----+----+----+----+----+----+----+ 18/07/03 16:16:27 INFO CodeGenerator: Code generated in 10.60842 ms 18/07/03 16:16:27 INFO CodeGenerator: Code generated in 8.990531 ms +----+ |col0| +----+ +----+

Re: Spark job reeturns empty rows from HBase

falbani — Wed, 04 Jul 2018 02:21:08 GMT

@vivek jain

Please run the following from HBase shell:

hbase> scan 'tableName', {'LIMIT' => 5}

Also check what the describe table prints:

bhase> describe ‘tableName’

Make sure you are using case-sensitive name when referencing table from spark code.

HTH

Re: Spark job reeturns empty rows from HBase

contactvivekjai — Wed, 04 Jul 2018 04:03:54 GMT

Hi @Felix Albani I have checked these things already.

Re: Spark job reeturns empty rows from HBase

falbani — Wed, 04 Jul 2018 04:08:49 GMT

@vivek jain Could you try running the following steps and see if that works:

https://community.hortonworks.com/articles/147327/accessing-hbase-tables-and-querying-on-dataframes.html

including table creation?

Re: Spark job reeturns empty rows from HBase

contactvivekjai — Wed, 04 Jul 2018 04:22:32 GMT

@Felix Albani I too really wanted to try this but these libraries are not deployed in cluster instead I create a dependencies jar and then I use it spark-submit.

Re: Spark job reeturns empty rows from HBase

contactvivekjai — Wed, 04 Jul 2018 04:51:39 GMT

@Felix Albani know what, I tried for a table with default namespace. I'm able to view data. Seems its working for tables without namespace.

Re: Spark job reeturns empty rows from HBase

contactvivekjai — Wed, 04 Jul 2018 05:13:54 GMT

@Felix Albani

just found that if I mention table as "table":{"name":"namespace:tablename"} in catalog then it works. Thanks for your time.

Re: Spark job reeturns empty rows from HBase

falbani — Wed, 04 Jul 2018 05:17:52 GMT

@vivek jain Good to hear that. If you think the answer and followups have helped please take a moment to login and mark as "Accepted"