Hi, I'm using impala driver to execute queries in spark and encountered following problem. Any suggestion would be appreciated.
sparkVersion = 2.2.0
impalaJdbcVersion = 2.6.3
Before moving to kerberos hadoop cluster, executing join sql and loading into spark are working fine.
spark.read.format("jdbc").option("url", "jdbc:impala://{host}:21050;").option("driver", "com.cloudera.impala.jdbc41.Driver").option("dbtable", "(SELECT s.id as student_id, sc.id as class_id FROM student s JOIN student_to_class sc where sc.id = s.class_id) t").load().createOrReplaceTempView("studentClass")
spark.sql("select * from studentClass").show+----------+--------+
|student_id|class_id|
+----------+--------+
|123 |111 |
|234 |111 |
|456 |111 |
+----------+--------+
After moved to Kerberos hadoop cluster, loading join query in spark return only column names (number of rows are still correct). Loading individual table and run sql on those tables in spark are still working correctly.
spark.read.format("jdbc").option("url", "jdbc:impala://{host}:21050;AuthMech=1;KrbHostFQDN={Krbhost};KrbRealm=TEST-DATA.COM;KrbServiceName=impala").option("driver", "com.cloudera.impala.jdbc41.Driver").option("dbtable", "(SELECT s.id as student_id, sc.id as class_id FROM student s JOIN student_to_class sc where sc.id = s.class_id) t").load().createOrReplaceTempView("studentClass")
spark.sql("select * from studentClass").show+----------+--------+
|student_id|class_id|
+----------+--------+
|student_id|class_id|
|student_id|class_id|
|student_id|class_id|
+----------+--------+
All the queries are working and return correct data in Impala-shell and Hue.
Since we won't be able to know all the tables needed before the spark job, being able to load join query into a table is needed for our task.
I've tried switching different version of Impala driver, but it didn't fix the problem.
