In HDP3 show columns function in hive returns the column list in alphabetical order.How to get the columns list in the actual order as in the create table of hive. ... View more

I am migrating the spark jobs running in HDP 2.6 to HDP 3.1. When executing the spark jobs in HDP3.1 I am getting the following error.       java.util.NoSuchElementException: spark.sql.hive.hiveserver2.jdbc.url at org.apache.spark.sql.internal.SQLConf$$anonfun$getConfString$2.apply(SQLConf.scala:1571) at org.apache.spark.sql.internal.SQLConf$$anonfun$getConfString$2.apply(SQLConf.scala:1571) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.internal.SQLConf.getConfString(SQLConf.scala:1571) at org.apache.spark.sql.RuntimeConfig.get(RuntimeConfig.scala:74) at com.hortonworks.spark.sql.hive.llap.HWConf.getConnectionUrlFromConf(HWConf.java:143) at com.hortonworks.spark.sql.hive.llap.HWConf.getConnectionUrl(HWConf.java:107) at com.hortonworks.spark.sql.hive.llap.HiveWarehouseBuilder.build(HiveWarehouseBuilder.java:97) at com.wunderman.hdp.Hdp3MigrationMain.main(Hdp3MigrationMain.java:16) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)         The way I m creating the spark session        private static SparkSession getSparkSession() { /** * Create an instance of SparkSession to connect to the cluster */ SparkSession sparkSession = SparkSession.builder().appName("Hdp3 Migration").master("yarn").getOrCreate(); return sparkSession; }        But the hiveserver2 jdbc url is configured in the spark config.  I have added the following dependency in the pom.xml       <dependency> <groupId>com.hortonworks.hive</groupId> <artifactId>hive-warehouse-connector_2.11</artifactId> <version>1.0.0.3.1.0.0-78</version> </dependency>     And I am trying to execute the below code     String hdp3Enabled = args[0]; Dataset<Row> dataset; String query="SELECT * FROM schema.tablename where col1='abc' ; //Sample query try { if ("Y".equalsIgnoreCase(hdp3Enabled)) { HiveWarehouseSession hive = HiveWarehouseSession.session(sparkSession).build(); dataset = hive.executeQuery(query); } else { dataset = sparkSession.sql(query); } dataset.show(); } catch(Exception e) { e.printStackTrace(); }      Share your suggestions to fix the issue. ... View more