Sandbox 2.4 Phoenix Spark classpath issue

jknulst — Thu, 08 Sep 2016 04:34:10 GMT

Hi,

I am trying to add just the right jar to Sparks driver/executor classpaths to make the basic tutorial work. There they mention some fat jar that contains all you need but such a jar is not shipped with HDP 2.4. Where can I find this jar? I am wary of using a non HWX build jar in order not to have shadowing jar in that 3rd party fat jar that break something else...

So far I added these jars to the cp (formatted for readability):

spark.executor.extraClassPath 

/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar:
/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-client.jar:
/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-core-4.4.0.2.4.0.0-169.jar:
/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-server-client-4.4.0.2.4.0.0-169.jar

But still I get :

java.lang.ClassNotFoundException: Failed to find data source: org.apache.phoenix.spark. Please find packages at http://spark-packages.org
       	at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:77)

Re: Sandbox 2.4 Phoenix Spark classpath issue

jknulst — Thu, 08 Sep 2016 05:44:52 GMT

Well it turns out that this is the minimum required:

spark.driver.extraClassPath /usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-client.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-common.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-core-4.4.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-protocol.jar:/usr/hdp/2.4.0.0-169/zeppelin/lib/interpreter/phoenix/hbase-server-1.0.1.jar

And use the same for spark.executor.extraClassPath

All these jar are directly available on the HDP 2.4 sandbox

question Re: Sandbox 2.4 Phoenix Spark classpath issue in Archives of Support Questions (Read Only)

Sandbox 2.4 Phoenix Spark classpath issue

Re: Sandbox 2.4 Phoenix Spark classpath issue