question How to connect Spark 2.2.0 with Phoenix 4.7 in HDP 2.6.3 ? in Archives of Support Questions (Read Only)

How to connect Spark 2.2.0 with Phoenix 4.7 in HDP 2.6.3 ?

ranjan_raut18 — Sat, 17 Mar 2018 09:56:55 GMT

Re: How to connect Spark 2.2.0 with Phoenix 4.7 in HDP 2.6.3 ?

sandyy006 — Sat, 17 Mar 2018 12:54:09 GMT

Below are the steps to connect Spark 2.2 with phoenix in HDP 2.6.3.

1) Create a symlink of hbase-site.xml in spark2 conf

ln -s /etc/hbase/conf/hbase-site.xml /etc/spark2/conf/hbase-site.xml

2) Launch spark-shell using phoenix spark jars in extra classpath.

spark-shell --conf "spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-4.7.0.2.6.3.0-235-spark2.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar" --conf "spark.driver.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-4.7.0.2.6.3.0-235-spark2.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar"

3) Create a phoenix connection and query the tables.

scala> import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.SQLContext

scala> val sqlContext = new SQLContext(sc)
sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@495e8a3

scala> val df = sqlContext.load("org.apache.phoenix.spark",Map("table" -> "TABLE1", "zkUrl" -> "localhost:2181"))
df: org.apache.spark.sql.DataFrame = [ID: string, COL1: string ... 1 more field]

scala> df.show()
+-----+----------+----+
|   ID|      COL1|COL2|
+-----+----------+----+
|test1|test_row_1|  10|
|test2|test_row_2|  20|
+-----+----------+----+

Re: How to connect Spark 2.2.0 with Phoenix 4.7 in HDP 2.6.3 ?

ranjan_raut18 — Sat, 17 Mar 2018 14:00:01 GMT

Thanks alot @Sandeep Nemuri. It worked (y)

Re: How to connect Spark 2.2.0 with Phoenix 4.7 in HDP 2.6.3 ?

sandyy006 — Sat, 17 Mar 2018 14:29:20 GMT

@Ranjan Raut Glad that it helped you, Would you mind accepting this answer so that this thread will be marked as answered.

Re: How to connect Spark 2.2.0 with Phoenix 4.7 in HDP 2.6.3 ?

MilesYao — Wed, 21 Mar 2018 02:48:35 GMT

Note that phoenix-spark2.jar MUST precede phoenix-client.jar in extraClassPath, otherwise connection will fail with:

java.lang.NoClassDefFoundError: org/apache/spark/sql/DataFrame

Re: How to connect Spark 2.2.0 with Phoenix 4.7 in HDP 2.6.3 ?

ranjan_raut18 — Wed, 28 Mar 2018 00:48:22 GMT

How can I save this "df" to other table of Phoenix ?

Re: How to connect Spark 2.2.0 with Phoenix 4.7 in HDP 2.6.3 ?

ranjan_raut18 — Thu, 31 May 2018 21:26:50 GMT

I found answer by myself ...

use df.saveToPhoenix(Map("table" -> "OUTPUT_TABLE", "zkUrl" -> hbaseConnectionString))

Re: How to connect Spark 2.2.0 with Phoenix 4.7 in HDP 2.6.3 ?

xpelive — Mon, 16 Jul 2018 18:28:06 GMT

any idea for this issue: https://community.hortonworks.com/questions/202521/spark-submit-nosuchmethoderror-savetophoenix.html