Member since
08-11-2019
5
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4816 | 09-15-2019 12:20 PM |
09-15-2019
12:20 PM
I was able to solve this by modifying SHC source code and package it using maven shade plugin. By doing this, all the required dependencies are packaged with shc jar https://github.com/dhananjaypatkar/shc
... View more
08-01-2019
12:25 PM
I am running spark 2.4 on Azure HDI version 4.0, which includes HDP 3.0. After the upgrade, we are unable to use spark-hbase connector version 1.1.1-2.1-s_2.11, it fails with below error java.lang.NoSuchMethodError: org.json4s.jackson.JsonMethods$.parse(Lorg/json4s/JsonInput;Z)Lorg/json4s/JsonAST$JValue; at org.apache.spark.sql.execution.datasources.hbase.HBaseTableCatalog$.apply(HBaseTableCatalog.scala:257) at org.apache.spark.sql.execution.datasources.hbase.HBaseRelation.<init>(HBaseRelation.scala:80) at org.apache.spark.sql.execution.datasources.hbase.DefaultSource.createRelation(HBaseRelation.scala:51) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167) at withCatalog(<console>:42) ... 56 elided As per github issue [https://github.com/hortonworks-spark/shc/issues/294 ], I tried building shc master locally and deploy, but ran into several dependency conflicts. Do we have guidelines / work arounds to get shc working with spark 2.4 on HDP 3.0?
... View more
Labels:
05-04-2018
07:53 PM
Is this still valid? I mean there are many improvements on spark-sql & catalyst engine since spark 1.6. Can you please rerun your tests?
... View more
11-28-2017
05:05 PM
If you look at how Phoenix handles in its code, you will get an Idea. Specifically, org.apache.phoenix.query.QueryConstants.java defines zero byte separator as below public static final byte SEPARATOR_BYTE = (byte) 0;
public static final byte[] SEPARATOR_BYTE_ARRAY = new byte[] {SEPARATOR_BYTE}; You can use Bytes.add(byte[], byte[]) to append separator byte array to form a composite row key. Let me know,How it works...
... View more