spark .session() .read() .option("encoding", "UTF-8") .option("delimiter", "^") .option("mode", "PERMISSIVE") .schema(SCHEMA_STORE.getIPDRschema()) .csv( JavaConverters.collectionAsScalaIterableConverter(_files_to_process) .asScala() .toSeq()) .withColumn("filename", org.apache.spark.sql.functions.input_file_name()) .dropDuplicates();   Written in java please convert it into scala hope this will work 🙂 ... View more

As per my understanding and reading their documentation and example, if table is an external table then only --conf "spark.sql.hive.hiveserver2.jdbc.url=jdbc:hive2://abc.comt:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2" should be passed since LLAP is not required for external table. In that case hive.execute("select * from tableA") should return all rows. Its only returning 1000 only in my case also. This might be a bug in HW end. I am also waiting for a response. ... View more