question Re: Spark2 Phoenix Plugin with Zeppelin in Support Questions

Spark2 Phoenix Plugin with Zeppelin

daniel_zafar — Sun, 18 Aug 2019 02:29:42 GMT

Hello, I'm looking to read tables from Phoenix using Zeppelin. I don't want to use the JDBC connection in favor of the plugin described here:

https://phoenix.apache.org/phoenix_spark.html

I believe I am not specifying the dependencies correctly because even though I added this to Zeppelin Interpreter (see zeppelin-phoenix-config.png). I'm still getting the following error:

import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.apache.phoenix.spark._
error: object phoenix is not a member of package org.apache

How do I fix this? I've also added

/usr/hdp/current/phoenix-server/lib/:/usr/hdp/current/phoenix-client/lib/:/usr/hdp/current/phoenix-server/

to both spark.driver.extraClassPath and spark.executor.extraClassPath in Spark2's spark.defaults.conf.

Thanks in advance!

P.S. I'm trying to figure all this out so that I can write a sparklyr extension to interact with Hbase data through Phoenix. Any pointers toward that goal are appreciated as well.

Re: Spark2 Phoenix Plugin with Zeppelin

sandyy006 — Wed, 15 Aug 2018 01:33:43 GMT

@Daniel Zafar

This article should help you out : https://community.hortonworks.com/content/supportkb/150591/how-to-connect-hbase-and-phoenix-tables-in-secure.html

Configure the zeppelin interpreter accordingly.

Edit: If using Spark2 : https://community.hortonworks.com/articles/179762/how-to-connect-to-phoenix-tables-using-spark2.html

Re: Spark2 Phoenix Plugin with Zeppelin

sandyy006 — Wed, 15 Aug 2018 13:33:02 GMT

@Daniel Zafar, Did the above articles helped you?

Re: Spark2 Phoenix Plugin with Zeppelin

daniel_zafar — Fri, 17 Aug 2018 03:32:18 GMT

@Sandeep Nemuri
Thanks for the pointers. I finally got it working. For others running Spark-Phoenix in Zeppelin you need to:

On the spark client node, create a symbolic link of 'hbase-site.xml' into the spark /conf file:

ln -s /usr/hdp/current/hbase-master/conf/hbase-site.xml /usr/hdp/current/spark2-client/conf/hbase-site.xml

Add the following to both spark.driver.extraClassPath and spark.executor.extraClassPath in spark-defaults.conf:

/usr/hdp/current/hbase-client/lib/hbase-common.jar:/usr/hdp/current/hbase-client/lib/hbase-client.jar:/usr/hdp/current/hbase-client/lib/hbase-server.jar:/usr/hdp/current/hbase-client/lib/hbase-protocol.jar:/usr/hdp/current/hbase-client/lib/guava-12.0.1.jar:/usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/spark-client/lib/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar

Add the following jars in Zeppelin under the Spark2 interpreter's dependencies

/usr/hdp/current/phoenix-server/lib/phoenix-spark-5.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-master/lib/hbase-common-2.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-client/lib/hbase-client-2.0.0.3.0.0.0-1634.jar
/usr/hdp/current/hbase-client/lib/htrace-core-3.2.0-incubating.jar
/usr/hdp/current/phoenix-client/phoenix-client.jar
/usr/hdp/current/phoenix-client/phoenix-server.jar

Re: Spark2 Phoenix Plugin with Zeppelin

zeybi — Tue, 24 Aug 2021 09:34:26 GMT

Hi;

i tried to make changes that you wrote. But i couldnt able to save my data to phoenix. I m using pyspark within hdp 3.0.1.0. I got error in zeppelin like java.lang.ClassNotFoundException: Class org.apache.phoenix.mapreduce.PhoenixOutputFormat not found

Did you meet such a problem like above?

Thanks in advance

My interpreter settings are;(I tried lots of config combinations but none of them worked)

spark.executor.extraClassPath

/usr/hdp/3.0.1.0-187/phoenix/phoenix-5.0.0.3.0.1.0-187-client.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-spark-5.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-core-5.0.0.3.0.1.0-187.jar,/usr/hdp/current/hbase-client/lib/hbase-common.jar:/usr/hdp/current/hbase-client/lib/hbase-client.jar,/usr/hdp/current/hbase-client/lib/hbase-server.jar,/usr/hdp/current/hbase-client/lib/hbase-protocol.jar,/usr/hdp/current/hbase-client/lib/guava-11.0.2.jar,/usr/hdp/current/hbase-client/lib/htrace-core-3.2.0-incubating.jar,/usr/hdp/current/phoenix-client/phoenix-client.jar

spark.driver.extraClassPath

/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-spark-5.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/phoenix/phoenix-5.0.0.3.0.1.0-187-client.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-core-5.0.0.3.0.1.0-187.jar

spark.jars

/usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/zeppelin/lib/shc-core-1.1.1-2.1-s_2.11.jar,/usr/hdp/current/spark-solr-3.5.5-shaded.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-spark-5.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/phoenix/phoenix-5.0.0.3.0.1.0-187-client.jar,/usr/hdp/current/hbase-client/lib/hbase-common-2.0.0.3.0.1.0-187.jar,/usr/hdp/3.0.1.0-187/phoenix/lib/phoenix-core-5.0.0.3.0.1.0-187.jar,/usr/hdp/current/hbase-master/lib/hbase-common-2.0.0.3.0.1.0-187.jar,/usr/hdp/current/hbase-client/lib/hbase-client-2.0.0.3.0.1.0-187.jar,/usr/hdp/current/hbase-client/lib/htrace-core-3.2.0-incubating.jar,/usr/hdp/current/phoenix-client/phoenix-client.jar,/usr/hdp/current/phoenix-client/phoenix-server.jar