Support Questions
Find answers, ask questions, and share your expertise

Spark2 Phoenix Plugin with Zeppelin

Contributor

Hello, I'm looking to read tables from Phoenix using Zeppelin. I don't want to use the JDBC connection in favor of the plugin described here:

https://phoenix.apache.org/phoenix_spark.html

I believe I am not specifying the dependencies correctly because even though I added this to Zeppelin Interpreter (see zeppelin-phoenix-config.png). I'm still getting the following error:

import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.apache.phoenix.spark._
error: object phoenix is not a member of package org.apache

How do I fix this? I've also added

/usr/hdp/current/phoenix-server/lib/:/usr/hdp/current/phoenix-client/lib/:/usr/hdp/current/phoenix-server/

to both spark.driver.extraClassPath and spark.executor.extraClassPath in Spark2's spark.defaults.conf.

Thanks in advance!

P.S. I'm trying to figure all this out so that I can write a sparklyr extension to interact with Hbase data through Phoenix. Any pointers toward that goal are appreciated as well.

86477-zeppelin-phoenix-config.png

1 ACCEPTED SOLUTION

Accepted Solutions

4 REPLIES 4

@Daniel Zafar, Did the above articles helped you?

Contributor

@Sandeep Nemuri
Thanks for the pointers. I finally got it working. For others running Spark-Phoenix in Zeppelin you need to:

  • On the spark client node, create a symbolic link of 'hbase-site.xml' into the spark /conf file:
ln -s /usr/hdp/current/hbase-master/conf/hbase-site.xml /usr/hdp/current/spark2-client/conf/hbase-site.xml
  • Add the following to both spark.driver.extraClassPath and spark.executor.extraClassPath in spark-defaults.conf:
/usr/hdp/current/hbase-client/lib/hbase-common.jar:/usr/hdp/current/hbase-client/lib/hbase-client.jar:/usr/hdp/current/hbase-client/lib/hbase-server.jar:/usr/hdp/current/hbase-client/lib/hbase-protocol.jar:/usr/hdp/current/hbase-client/lib/guava-12.0.1.jar:/usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/spark-client/lib/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar
  • Add the following jars in Zeppelin under the Spark2 interpreter's dependencies
/usr/hdp/current/phoenix-server/lib/phoenix-spark-5.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-master/lib/hbase-common-2.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-client/lib/hbase-client-2.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-client/lib/htrace-core-3.2.0-incubating.jar
/usr/hdp/current/phoenix-client/phoenix-client.jar
/usr/hdp/current/phoenix-client/phoenix-server.jar

New Contributor

Hi;

i tried to make changes that you wrote. But i couldnt able to save my data to phoenix. I m using pyspark within hdp 3.0.1.0. I got error in zeppelin like java.lang.ClassNotFoundException: Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found

Did you meet such a problem like above?

Thanks in advance

My interpreter settings are;(I tried lots of config combinations but none of them worked)

spark.executor.extraClassPath/usr/hdp/3.0.1.0-187/phoenix/phoenix-5.0.0.3.0.1.0-187-client.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-spark-5.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-core-5.0.0.3.0.1.0-187.jar,/usr/hdp/current/hbase-client/lib/hbase-common.jar:/usr/hdp/current/hbase-client/lib/hbase-client.jar,/usr/hdp/current/hbase-client/lib/hbase-server.jar,/usr/hdp/current/hbase-client/lib/hbase-protocol.jar,/usr/hdp/current/hbase-client/lib/guava-11.0.2.jar,/usr/hdp/current/hbase-client/lib/htrace-core-3.2.0-incubating.jar,/usr/hdp/current/phoenix-client/phoenix-client.jar
spark.driver.extraClassPath/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-spark-5.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/phoenix/phoenix-5.0.0.3.0.1.0-187-client.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-core-5.0.0.3.0.1.0-187.jar
spark.jars/usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/zeppelin/lib/shc-core-1.1.1-2.1-s_2.11.jar,/usr/hdp/current/spark-solr-3.5.5-shaded.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-spark-5.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/phoenix/phoenix-5.0.0.3.0.1.0-187-client.jar,/usr/hdp/current/hbase-client/lib/hbase-common-2.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-core-5.0.0.3.0.1.0-187.jar,/usr/hdp/current/hbase-master/lib/hbase-common-2.0.0.3.0.1.0-187.jar,/usr/hdp/current/hbase-client/lib/hbase-client-2.0.0.3.0.1.0-187.jar,/usr/hdp/current/hbase-client/lib/htrace-core-3.2.0-incubating.jar,/usr/hdp/current/phoenix-client/phoenix-client.jar,/usr/hdp/current/phoenix-client/phoenix-server.jar