Thank you @Dongjoon Hyun! Confirmed it works in HDP 2.6.3 by replacing the jar file with spark-llap-assembly-1.0.0.2.6.3.0-235.jar ... View more

Hello, I followed the instruction with HDP 2.6.3.0, however, Spark2 Thrift Server stops right after starting it with the following error in /var/log/spark2/spark-hive-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-<HOSTNAME>.out: Exception in thread "main" java.lang.IllegalArgumentException: Unable to instantiate SparkSession with LLAP support because LLAP or Hive classes are not found. at org.apache.spark.sql.SparkSession$.isLLAPEnabled(SparkSession.scala:1104) at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$externalCatalogClassName(SharedState.scala:174) at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:95) at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:93) at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:53) at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:81) at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Looks like this error indicates Spark2 Thrift Server fails to load org.apache.hadoop.hive.conf.HiveConf or org.apache.spark.sql.hive.llap.LlapSessionStateBuilder classes. I found com.hortonworks.spark_spark-llap_2.11-1.1.3-2.1.jar, which Spark2 Thrift Server is using, does not have org.apache.hadoop.hive.conf.HiveConf but shadehive.org.apache.hadoop.hive.conf.HiveConf Can I ask if it's a bug? Can I also ask if there is a workaround? Thank you in advance, Mai Nakagawa ... View more

This is great. And can be even better if you fix the broken links to image. Thanks ... View more