Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

issue with sparkSQL in Spark 2.0.0

issue with sparkSQL in Spark 2.0.0

New Contributor

This query below works in Spark 1.6.0, but when I run the same one in spark 2.0, it has issues:

 

>>> dt = sqlContext.sql("select * from change_points where end_last_dttm > '2015-12-31' and mask = 'MW'").cache()

 

16/10/03 14:52:19 WARN BoneCPConfig: Max Connections < 1. Setting to 20

16/10/03 14:52:21 WARN BoneCPConfig: Max Connections < 1. Setting to 20

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/opt/cloudera/parcels/SPARK2-2.0.0.cloudera.beta1-1.cdh5.7.0.p0.108015/lib/spark2/python/pyspark/sql/context.py", line 360, in sql

    return self.sparkSession.sql(sqlQuery)

  File "/opt/cloudera/parcels/SPARK2-2.0.0.cloudera.beta1-1.cdh5.7.0.p0.108015/lib/spark2/python/pyspark/sql/session.py", line 543, in sql

    return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)

  File "/opt/cloudera/parcels/SPARK2-2.0.0.cloudera.beta1-1.cdh5.7.0.p0.108015/lib/spark2/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py", line 1133, in __call__

  File "/opt/cloudera/parcels/SPARK2-2.0.0.cloudera.beta1-1.cdh5.7.0.p0.108015/lib/spark2/python/pyspark/sql/utils.py", line 69, in deco

    raise AnalysisException(s.split(': ', 1)[1], stackTrace)

pyspark.sql.utils.AnalysisException: u'Table or view not found: change_points; line 1 pos 14'

 

It says ‘change_points’ table is not found, but it’s right there.

1 REPLY 1

Re: issue with sparkSQL in Spark 2.0.0

Expert Contributor

You may want to double check that spark is getting the correct configuration.  This could occur if it isn't connecting to the metastore.  Please ensure the server you are running on is configured as a Spark 2 gateway node.