Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark2 Phoenix Plugin with Zeppelin

Solved Go to solution
Highlighted

Spark2 Phoenix Plugin with Zeppelin

Contributor

Hello, I'm looking to read tables from Phoenix using Zeppelin. I don't want to use the JDBC connection in favor of the plugin described here:

https://phoenix.apache.org/phoenix_spark.html

I believe I am not specifying the dependencies correctly because even though I added this to Zeppelin Interpreter (see zeppelin-phoenix-config.png). I'm still getting the following error:

import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.apache.phoenix.spark._
error: object phoenix is not a member of package org.apache

How do I fix this? I've also added

/usr/hdp/current/phoenix-server/lib/:/usr/hdp/current/phoenix-client/lib/:/usr/hdp/current/phoenix-server/

to both spark.driver.extraClassPath and spark.executor.extraClassPath in Spark2's spark.defaults.conf.

Thanks in advance!

P.S. I'm trying to figure all this out so that I can write a sparklyr extension to interact with Hbase data through Phoenix. Any pointers toward that goal are appreciated as well.

86477-zeppelin-phoenix-config.png

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Spark2 Phoenix Plugin with Zeppelin

3 REPLIES 3

Re: Spark2 Phoenix Plugin with Zeppelin

Re: Spark2 Phoenix Plugin with Zeppelin

@Daniel Zafar, Did the above articles helped you?

Re: Spark2 Phoenix Plugin with Zeppelin

Contributor

@Sandeep Nemuri
Thanks for the pointers. I finally got it working. For others running Spark-Phoenix in Zeppelin you need to:

  • On the spark client node, create a symbolic link of 'hbase-site.xml' into the spark /conf file:
ln -s /usr/hdp/current/hbase-master/conf/hbase-site.xml /usr/hdp/current/spark2-client/conf/hbase-site.xml
  • Add the following to both spark.driver.extraClassPath and spark.executor.extraClassPath in spark-defaults.conf:
/usr/hdp/current/hbase-client/lib/hbase-common.jar:/usr/hdp/current/hbase-client/lib/hbase-client.jar:/usr/hdp/current/hbase-client/lib/hbase-server.jar:/usr/hdp/current/hbase-client/lib/hbase-protocol.jar:/usr/hdp/current/hbase-client/lib/guava-12.0.1.jar:/usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/spark-client/lib/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar
  • Add the following jars in Zeppelin under the Spark2 interpreter's dependencies
/usr/hdp/current/phoenix-server/lib/phoenix-spark-5.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-master/lib/hbase-common-2.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-client/lib/hbase-client-2.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-client/lib/htrace-core-3.2.0-incubating.jar
/usr/hdp/current/phoenix-client/phoenix-client.jar
/usr/hdp/current/phoenix-client/phoenix-server.jar