Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Managing jars while connect Phoenix to Zeppelin

avatar
Contributor

I'm trying to use the Phoenix-Spark2 connector in Zeppelin as described here and having some confusion about dependencies

Here is the code I'm running:

import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.apache.phoenix.spark._
val sc = new SparkContext("local", "phoenix-test")
val sqlContext = new SQLContext(sc)
val df = sqlContext.load(
  "org.apache.phoenix.spark",
  Map("table" -> "dz_places",
      "zkUrl" -> "driver_node:2181"))

I keep getting ClassNotFound Exceptions so I go and find the related jar and add it to the Zeppelin Spark2 interpreter dependencies. So far I've added these jars:

/usr/hdp/current/phoenix-server/lib/phoenix-spark-5.0.0.3.0.0.0-1634.jar
/usr/hdp/current/phoenix-server/lib/phoenix-core-5.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-master/lib/hbase-common-2.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-client/lib/hbase-client-2.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-client/lib/htrace-core-3.2.0-incubating.jar

Now I'm seeing this error:

java.lang.NoClassDefFoundError: org/apache/tephra/TransactionSystemClient

I'm not seeing this jar in any of the Hbase or Phoenix lib folders. What's going on? Why do I need to add all these? Where is this particular class housed? Is there a better way to specify these? Using /usr/hdp/current/hbase-client/lib/*.jar threw an error.

1 ACCEPTED SOLUTION

avatar

@Daniel Zafar Additionally you'd need to add phoenix-client.jar in the classpath.

jar tvf  phoenix-client.jar | grep TransactionSystemClient
1548 Thu Jul 12 20:37:06 UTC 2018 org/apache/tephra/TransactionSystemClient.class

Hope this helps!

View solution in original post

6 REPLIES 6

avatar

Hello @Daniel Zafar!
It seems that your Phoenix it's with ACID feat enable.

https://phoenix.apache.org/transactions.html

So that's why Phoenix is complaining about tephra (http://tephra.incubator.apache.org/).

Take a look at the link below, has some good explanations about how to enable tephra.

https://community.hortonworks.com/questions/113991/enabling-transaction-in-phoenix-issue.html

Hope this helps!

avatar
Contributor

@Vinicius Higa Murakami

I don't mind providing the jars but is there a way to add all of them in once to Zeppelin? I'm also have trouble figuring out which Phoenix jar this class is from. Do you know?

avatar

@Daniel Zafar Additionally you'd need to add phoenix-client.jar in the classpath.

jar tvf  phoenix-client.jar | grep TransactionSystemClient
1548 Thu Jul 12 20:37:06 UTC 2018 org/apache/tephra/TransactionSystemClient.class

Hope this helps!

avatar
Contributor

How about this one?

com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: com/lmax/disruptor/EventFactory

avatar

Some jar conflicts, Try removing the phoenix-core jar from the classpath.

avatar
Contributor

you're the best 🙂