Sandbox 2.4 Phoenix Spark classpath issue

[#1]

Hi,

I am trying to add just the right jar to Sparks driver/executor classpaths to make the basic tutorial work. There they mention some fat jar that contains all you need but such a jar is not shipped with HDP 2.4. Where can I find this jar? I am wary of using a non HWX build jar in order not to have shadowing jar in that 3rd party fat jar that break something else...

So far I added these jars to the cp (formatted for readability):

spark.executor.extraClassPath 

/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar:
/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-client.jar:
/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-core-4.4.0.2.4.0.0-169.jar:
/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-server-client-4.4.0.2.4.0.0-169.jar

But still I get :

java.lang.ClassNotFoundException: Failed to find data source: org.apache.phoenix.spark. Please find packages at http://spark-packages.org
       	at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:77)

[#2 · Accepted solution]

Well it turns out that this is the minimum required:

spark.driver.extraClassPath /usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-client.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-common.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-core-4.4.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-protocol.jar:/usr/hdp/2.4.0.0-169/zeppelin/lib/interpreter/phoenix/hbase-server-1.0.1.jar

And use the same for spark.executor.extraClassPath

All these jar are directly available on the HDP 2.4 sandbox

[#3]

Hi:

I have HDP 2.4.0.0 and Iam trying to use Phoenix 2.4.0 on Spark 1.6.0:

I am using the next classpath but iam receiving this error, I need anything else?

Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
  File "/usr/hdp/2.4.0.0-169/spark/python/pyspark/sql/readwriter.py", line 385, in save
    self._jwrite.save()
  File "/usr/hdp/2.4.0.0-169/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
  File "/usr/hdp/2.4.0.0-169/spark/python/pyspark/sql/utils.py", line 45, in deco
    return f(*a, **kw)
  File "/usr/hdp/2.4.0.0-169/spark/python/lib/py4j-0.9-src.zip/py4j/protocol.py", line 308, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o67.save.
: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy

Driver and executor path Iam using:

/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-client.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-common.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-core-4.4.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-protocol.jar

Regards