Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Rising Star

Consolidating below some of the errors thrown by Spark Thrift Server during SQL execution, that could be worked around by configuring certain parameters of spark-thrift-sparkconf.conf and hive-site.xml

Error 1:

Join condition is missing or trivial.Use the CROSS JOIN syntax to allow cartesian products between these relations.;

Resolution: spark.sql.crossjoin.enabled: true

Error 2:

Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method "eval(Lorg/apache/spark/sql/catalyst/InternalRow;)Z" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate" grows beyond 64 KB

Resolution: spark.sql.codegen.wholeStage : false

Error 3:

java.lang.OutOfMemoryError: Java heap space

Resolution: spark.driver.memory : 10g <to a higher-value>

spark.sql.ui.retainedExecutions: 5 <to some lower-value>

Error 4:

org.apache.spark.SparkException: Exception thrown in awaitResult: (state=,code=0)

Resolution: hive.metastore.try.direct.sql: false (in hive-site.xml)

To enable heap dump collection for spark driver and executors, for debugging out of memory errors

spark.driver.extraJavaOptions: '-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path-to-dump-file-location>'

spark.executor.extraJavaOptions: '-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path-to-dump-file-location>'

2,181 Views