question Re: Managing jars while connect Phoenix to Zeppelin in Archives of Support Questions (Read Only)

Managing jars while connect Phoenix to Zeppelin

daniel_zafar — Wed, 15 Aug 2018 13:03:57 GMT

I'm trying to use the Phoenix-Spark2 connector in Zeppelin as described here and having some confusion about dependencies

Here is the code I'm running:

import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.apache.phoenix.spark._
val sc = new SparkContext("local", "phoenix-test")
val sqlContext = new SQLContext(sc)
val df = sqlContext.load(
  "org.apache.phoenix.spark",
  Map("table" -> "dz_places",
      "zkUrl" -> "driver_node:2181"))

I keep getting ClassNotFound Exceptions so I go and find the related jar and add it to the Zeppelin Spark2 interpreter dependencies. So far I've added these jars:

/usr/hdp/current/phoenix-server/lib/phoenix-spark-5.0.0.3.0.0.0-1634.jar
/usr/hdp/current/phoenix-server/lib/phoenix-core-5.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-master/lib/hbase-common-2.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-client/lib/hbase-client-2.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-client/lib/htrace-core-3.2.0-incubating.jar

Now I'm seeing this error:

java.lang.NoClassDefFoundError: org/apache/tephra/TransactionSystemClient

I'm not seeing this jar in any of the Hbase or Phoenix lib folders. What's going on? Why do I need to add all these? Where is this particular class housed? Is there a better way to specify these? Using /usr/hdp/current/hbase-client/lib/*.jar threw an error.

Re: Managing jars while connect Phoenix to Zeppelin

vmurakami — Thu, 16 Aug 2018 10:21:23 GMT

Hello @Daniel Zafar!
It seems that your Phoenix it's with ACID feat enable.

https://phoenix.apache.org/transactions.html

So that's why Phoenix is complaining about tephra (http://tephra.incubator.apache.org/).

Take a look at the link below, has some good explanations about how to enable tephra.

https://community.hortonworks.com/questions/113991/enabling-transaction-in-phoenix-issue.html

Hope this helps!

Re: Managing jars while connect Phoenix to Zeppelin

daniel_zafar — Thu, 16 Aug 2018 11:40:22 GMT

@Vinicius Higa Murakami

I don't mind providing the jars but is there a way to add all of them in once to Zeppelin? I'm also have trouble figuring out which Phoenix jar this class is from. Do you know?

Re: Managing jars while connect Phoenix to Zeppelin

sandyy006 — Thu, 16 Aug 2018 14:31:39 GMT

@Daniel Zafar Additionally you'd need to add phoenix-client.jar in the classpath.

jar tvf  phoenix-client.jar | grep TransactionSystemClient
1548 Thu Jul 12 20:37:06 UTC 2018 org/apache/tephra/TransactionSystemClient.class

Hope this helps!

Re: Managing jars while connect Phoenix to Zeppelin

daniel_zafar — Thu, 16 Aug 2018 22:41:33 GMT

How about this one?

com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: com/lmax/disruptor/EventFactory

Re: Managing jars while connect Phoenix to Zeppelin

sandyy006 — Thu, 16 Aug 2018 23:44:29 GMT

Some jar conflicts, Try removing the phoenix-core jar from the classpath.

Re: Managing jars while connect Phoenix to Zeppelin

daniel_zafar — Fri, 17 Aug 2018 03:24:49 GMT

you're the best 🙂