spark .session() .read() .option("encoding", "UTF-8") .option("delimiter", "^") .option("mode", "PERMISSIVE") .schema(SCHEMA_STORE.getIPDRschema()) .csv( JavaConverters.collectionAsScalaIterableConverter(_files_to_process) .asScala() .toSeq()) .withColumn("filename", org.apache.spark.sql.functions.input_file_name()) .dropDuplicates();   Written in java please convert it into scala hope this will work 🙂 ... View more

@manjj To understand the performance difference between two Cloudera Hadoop cluster and Apache Hadoop cluster. I would suggest you to collect the Impala_query profile from both cluster and compare it.  To analysis the Impala query profile. Please use the below articles.  https://docs.cloudera.com/runtime/7.2.9/impala-reference/topics/impala-profile.html https://conferences.oreilly.com/strata/strata-ca-2018/cdn.oreillystatic.com/en/assets/1/event/269/How%20to%20use%20Impala_s%20query%20plan%20and%20profile%20to%20fix%20performance%20issues%20Presentation.pdf https://cloudera.ericlin.me/2018/09/impala-query-profile-explained-part-1/ Cheers! Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.   ... View more

Hi @manjj,   Thanks for reporting back the progress. There is a possible leaking somewhere in Hive even this bug was already fixed.   You can follow below steps if you do want to have the Hive Metastore canary test turned on. Steps: If Hive Metastore canary test is disabled, re-enable the Hive Metastore canary test. For configurations of both HDFS and Hive, find Service Monitor Client Config Overrides and add an entry for "fs.file.impl.disable.cache" with value "true". Restart Service Monitor And observe whether the heap also stays stable, with the canary back on. Thanks and hope this helps, Li ... View more

Thanks @EricL for chime in. @manjj That's correct I found it later today as well this resides in desktop_document2 nowadays 🙂 ... View more

Thanks a lot, this works for me. ... View more