Cloudera Community
dhananjay_patka
user rank
Explorer
My Accepted Solutions
Show All

Re: SHC on HDP 3.0 With spark 2.4

by Explorer in Support Questions
‎09-15-2019 12:20 PM
‎09-15-2019 12:20 PM
I was able to solve this by modifying SHC source code and package it using maven shade plugin. By doing this, all the required dependencies are packaged with shc jar   https://github.com/dhananjaypatkar/shc     ... View more

SHC on HDP 3.0 With spark 2.4

by Explorer in Support Questions
‎08-01-2019 12:25 PM
‎08-01-2019 12:25 PM
I am running spark 2.4 on Azure HDI version 4.0, which includes HDP 3.0. After the upgrade, we are unable to use spark-hbase connector version 1.1.1-2.1-s_2.11, it fails with below error java.lang.NoSuchMethodError: org.json4s.jackson.JsonMethods$.parse(Lorg/json4s/JsonInput;Z)Lorg/json4s/JsonAST$JValue; at org.apache.spark.sql.execution.datasources.hbase.HBaseTableCatalog$.apply(HBaseTableCatalog.scala:257) at org.apache.spark.sql.execution.datasources.hbase.HBaseRelation.<init>(HBaseRelation.scala:80) at org.apache.spark.sql.execution.datasources.hbase.DefaultSource.createRelation(HBaseRelation.scala:51) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167) at withCatalog(<console>:42) ... 56 elided As per github issue [https://github.com/hortonworks-spark/shc/issues/294 ], I tried building shc master locally and deploy, but ran into several dependency conflicts. Do we have guidelines / work arounds to get shc working with spark 2.4 on HDP 3.0? ... View more

Re: Spark RDDs vs DataFrames vs SparkSQL

by Explorer in Community Articles
‎05-04-2018 07:53 PM
‎05-04-2018 07:53 PM
Is this still valid? I mean there are many improvements on spark-sql & catalyst engine since spark 1.6. Can you please rerun your tests? ... View more

Re: How are the primary keys in Phoenix are conver...

by Explorer in Support Questions
‎11-28-2017 05:05 PM
‎11-28-2017 05:05 PM
If you look at how Phoenix handles in its code, you will get an Idea. Specifically, org.apache.phoenix.query.QueryConstants.java defines zero byte separator as below public static final byte SEPARATOR_BYTE = (byte) 0; public static final byte[] SEPARATOR_BYTE_ARRAY = new byte[] {SEPARATOR_BYTE}; You can use Bytes.add(byte[], byte[]) to append separator byte array to form a composite row key. Let me know,How it works... ... View more
Contact Me
Online
Offline
Last Visited
‎10-17-2019 10:07 AM
Community Statistics
Member Since ‎08-11-2019 10:27 AM
Last Visited ‎10-17-2019 10:07 AM
Posts 5