Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sandbox 2.4 Phoenix Spark classpath issue

Solved Go to solution

Sandbox 2.4 Phoenix Spark classpath issue

Super Collaborator

Hi,

I am trying to add just the right jar to Sparks driver/executor classpaths to make the basic tutorial work. There they mention some fat jar that contains all you need but such a jar is not shipped with HDP 2.4. Where can I find this jar? I am wary of using a non HWX build jar in order not to have shadowing jar in that 3rd party fat jar that break something else...

So far I added these jars to the cp (formatted for readability):

spark.executor.extraClassPath 

/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar:
/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-client.jar:
/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-core-4.4.0.2.4.0.0-169.jar:
/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-server-client-4.4.0.2.4.0.0-169.jar

But still I get :

java.lang.ClassNotFoundException: Failed to find data source: org.apache.phoenix.spark. Please find packages at http://spark-packages.org
       	at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:77)
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Sandbox 2.4 Phoenix Spark classpath issue

Super Collaborator

Well it turns out that this is the minimum required:

spark.driver.extraClassPath /usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-client.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-common.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-core-4.4.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-protocol.jar:/usr/hdp/2.4.0.0-169/zeppelin/lib/interpreter/phoenix/hbase-server-1.0.1.jar

And use the same for spark.executor.extraClassPath

All these jar are directly available on the HDP 2.4 sandbox

2 REPLIES 2

Re: Sandbox 2.4 Phoenix Spark classpath issue

Super Collaborator

Well it turns out that this is the minimum required:

spark.driver.extraClassPath /usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-client.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-common.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-core-4.4.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-protocol.jar:/usr/hdp/2.4.0.0-169/zeppelin/lib/interpreter/phoenix/hbase-server-1.0.1.jar

And use the same for spark.executor.extraClassPath

All these jar are directly available on the HDP 2.4 sandbox

Re: Sandbox 2.4 Phoenix Spark classpath issue

Super Collaborator

Hi:

I have HDP 2.4.0.0 and Iam trying to use Phoenix 2.4.0 on Spark 1.6.0:

I am using the next classpath but iam receiving this error, I need anything else?

Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
  File "/usr/hdp/2.4.0.0-169/spark/python/pyspark/sql/readwriter.py", line 385, in save
    self._jwrite.save()
  File "/usr/hdp/2.4.0.0-169/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
  File "/usr/hdp/2.4.0.0-169/spark/python/pyspark/sql/utils.py", line 45, in deco
    return f(*a, **kw)
  File "/usr/hdp/2.4.0.0-169/spark/python/lib/py4j-0.9-src.zip/py4j/protocol.py", line 308, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o67.save.
: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy




Driver and executor path Iam using:

/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-client.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-common.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/phoenix-core-4.4.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/phoenix/lib/hbase-protocol.jar

Regards

Don't have an account?
Coming from Hortonworks? Activate your account here