Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark2 Phoenix Plugin with Zeppelin

avatar
Contributor

Hello, I'm looking to read tables from Phoenix using Zeppelin. I don't want to use the JDBC connection in favor of the plugin described here:

https://phoenix.apache.org/phoenix_spark.html

I believe I am not specifying the dependencies correctly because even though I added this to Zeppelin Interpreter (see zeppelin-phoenix-config.png). I'm still getting the following error:

import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.apache.phoenix.spark._
error: object phoenix is not a member of package org.apache

How do I fix this? I've also added

/usr/hdp/current/phoenix-server/lib/:/usr/hdp/current/phoenix-client/lib/:/usr/hdp/current/phoenix-server/

to both spark.driver.extraClassPath and spark.executor.extraClassPath in Spark2's spark.defaults.conf.

Thanks in advance!

P.S. I'm trying to figure all this out so that I can write a sparklyr extension to interact with Hbase data through Phoenix. Any pointers toward that goal are appreciated as well.

86477-zeppelin-phoenix-config.png

1 ACCEPTED SOLUTION

avatar
4 REPLIES 4

avatar

avatar

@Daniel Zafar, Did the above articles helped you?

avatar
Contributor

@Sandeep Nemuri
Thanks for the pointers. I finally got it working. For others running Spark-Phoenix in Zeppelin you need to:

  • On the spark client node, create a symbolic link of 'hbase-site.xml' into the spark /conf file:
ln -s /usr/hdp/current/hbase-master/conf/hbase-site.xml /usr/hdp/current/spark2-client/conf/hbase-site.xml
  • Add the following to both spark.driver.extraClassPath and spark.executor.extraClassPath in spark-defaults.conf:
/usr/hdp/current/hbase-client/lib/hbase-common.jar:/usr/hdp/current/hbase-client/lib/hbase-client.jar:/usr/hdp/current/hbase-client/lib/hbase-server.jar:/usr/hdp/current/hbase-client/lib/hbase-protocol.jar:/usr/hdp/current/hbase-client/lib/guava-12.0.1.jar:/usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/spark-client/lib/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar
  • Add the following jars in Zeppelin under the Spark2 interpreter's dependencies
/usr/hdp/current/phoenix-server/lib/phoenix-spark-5.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-master/lib/hbase-common-2.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-client/lib/hbase-client-2.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-client/lib/htrace-core-3.2.0-incubating.jar
/usr/hdp/current/phoenix-client/phoenix-client.jar
/usr/hdp/current/phoenix-client/phoenix-server.jar

avatar
New Contributor

Hi;

i tried to make changes that you wrote. But i couldnt able to save my data to phoenix. I m using pyspark within hdp 3.0.1.0. I got error in zeppelin like java.lang.ClassNotFoundException: Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found

Did you meet such a problem like above?

Thanks in advance

My interpreter settings are;(I tried lots of config combinations but none of them worked)

spark.executor.extraClassPath/usr/hdp/3.0.1.0-187/phoenix/phoenix-5.0.0.3.0.1.0-187-client.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-spark-5.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-core-5.0.0.3.0.1.0-187.jar,/usr/hdp/current/hbase-client/lib/hbase-common.jar:/usr/hdp/current/hbase-client/lib/hbase-client.jar,/usr/hdp/current/hbase-client/lib/hbase-server.jar,/usr/hdp/current/hbase-client/lib/hbase-protocol.jar,/usr/hdp/current/hbase-client/lib/guava-11.0.2.jar,/usr/hdp/current/hbase-client/lib/htrace-core-3.2.0-incubating.jar,/usr/hdp/current/phoenix-client/phoenix-client.jar
spark.driver.extraClassPath/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-spark-5.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/phoenix/phoenix-5.0.0.3.0.1.0-187-client.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-core-5.0.0.3.0.1.0-187.jar
spark.jars/usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/zeppelin/lib/shc-core-1.1.1-2.1-s_2.11.jar,/usr/hdp/current/spark-solr-3.5.5-shaded.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-spark-5.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/phoenix/phoenix-5.0.0.3.0.1.0-187-client.jar,/usr/hdp/current/hbase-client/lib/hbase-common-2.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-core-5.0.0.3.0.1.0-187.jar,/usr/hdp/current/hbase-master/lib/hbase-common-2.0.0.3.0.1.0-187.jar,/usr/hdp/current/hbase-client/lib/hbase-client-2.0.0.3.0.1.0-187.jar,/usr/hdp/current/hbase-client/lib/htrace-core-3.2.0-incubating.jar,/usr/hdp/current/phoenix-client/phoenix-client.jar,/usr/hdp/current/phoenix-client/phoenix-server.jar