Created 09-16-2020 05:51 PM
Using Spark Hbase Connector.
Cloudera distribution: 6.3.2
HBase version: 2.1.0
Scala Version: 2.11.12
Error:
Error:
spark-hbase connector version 1.1.1-2.1-s_2.11, it fails with below error java.lang.NoSuchMethodError: org.json4s.jackson.JsonMethods$.parse(Lorg/json4s/JsonInput;Z)Lorg/json4s/JsonAST$JValue; at org.apache.spark.sql.execution.datasources.hbase.HBaseTableCatalog$.apply(HBaseTableCatalog.scala:257)
Searching over internet pointed out, shc-core:1.1.3-2.4-s_2.11.jar has solved the issue.
Could not find a repository to download.
Any suggestion please?
Thanks.
Created 09-17-2020 12:49 AM
You can use public hortonworks repo - https://repo.hortonworks.com/content/groups/public/
You may not find exact version which you mentioned. But you can check repo, you can use dependencies as per your cluster version.
You can try below dependency :
<dependency>
<groupId>com.hortonworks.shc</groupId>
<artifactId>shc-core</artifactId>
<version>1.1.0.3.1.5.0-152</version>
</dependency>
Let me know it works. It should be compatible.
Created 01-29-2021 05:25 AM
Hi i am facing the same error for this topic, but before changing the version of shc-core the error changed to it:
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.parser.AbstractSqlParser: method <init>()V not found
at org.apache.spark.sql.execution.SparkSqlParser.<init>(SparkSqlParser.scala:42)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.sqlParser$lzycompute(BaseSessionStateBuilder.scala:117)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.sqlParser(BaseSessionStateBuilder.scala:116)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:292)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1104)
at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:145)
at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:144)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:144)
at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:141)
at org.apache.spark.sql.DataFrameReader.<init>(DataFrameReader.scala:788)
at org.apache.spark.sql.SparkSession.read(SparkSession.scala:655)
at ContactEngineRun$.main(ContactEngineRun.scala:26) -> this line of code is above
val hbaseDF = sparkSession.read
.options(Map(HBaseTableCatalog.tableCatalog -> catalog))
.format("org.apache.spark.sql.execution.datasources.hbase")
.load()
this is my build dependencies:
val spark_version = "2.4.4"
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-library" % "2.11.12",
// SPARK LIBRARIES
"org.apache.spark" %% "spark-streaming" % spark_version,
"org.apache.spark" %% "spark-core" % spark_version,
"org.apache.spark" %% "spark-sql" % spark_version,
//HBASE
"org.apache.hbase.connectors.spark" % "hbase-spark" % "1.0.0.7.2.1.0-321",
"org.apache.hbase" % "hbase-client" % "2.4.0",
"com.hortonworks.shc" % "shc-core" % "1.1.0.3.1.5.0-152"
)
Can someone help me?
Created 09-17-2020 01:24 AM
@SibDe Sorry to disturb you. You're using Cloudera Manager 6.3.x? Could you please help me to verify a issue I encountered now? FYI. I upgraded Cloudera Manager from 5.10.0 to 6.3.0, then the following API seems not work well.
Can you help to verify if 'wget http://<cm_server_host>:7180/api/v13/clusters/<cluster_name>/services/<service_name>/clientConfig' can work as expected? Thanks in advance!!
Created 11-30-2020 06:07 AM
Were you able to find an alternative since the jar doesn't seem to be available in Maven repository yet?
While trying to import through -- packages ,it throws below error:
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: com.hortonworks#shc;1.1.3-2.4-s_2.11: not found]
Created on 01-29-2021 05:47 AM - edited 01-29-2021 07:01 AM
@lerner like @schhabra1 says:
You can use public hortonworks repo - https://repo.hortonworks.com/content/groups/public/
Created 03-08-2021 06:17 PM
Sorry for distrubing. I am working on a toy project which needs to insert spark data frame into hbase.
Apache Kafka | 2.2.1 |
Apache Spark | 2.4.0 |
Apache HBase | 2.1.4 |
CDH | 6.3.2 |
After reading some posts about spark hbase connector, I decided to to hortonworks spark habse connector.
I am wondering if I need the HBase client configuration file hbase-site.xml for hortonworks spark hbase connector when I am working in CDH environment?
Thanks for your help in advance!