This would also work. import java.io.File val files = getListOfFiles("/tmp") def getListOfFiles(dir: File):List[File] = dir.listFiles.filter(_.isFile).toList ... View more

@Chaitanya D It will be possible in with unix and spark combination. hadoop fs -ls /filedirectory/*txt_processed Above command will return the desired file you need. Then pass the result to spark and process the file as you need. Alternatively in spark you can select the desired file using the below command. val lsResult =Seq("hadoop","fs","-ls","hdfs://filedirectory/*txt_prcoessed").!! Hope it helps ! ... View more

@Alexandros Biratsis i believe that you are not using Insert Overwrite when inserting the incremental records into the target. Assuming that its weird how the data is being overridden. For the union part --> If you wanted to avoid union then you may have to may have to perform left join between the incremental data and target to apply some transformations ( assuming that you are performing SCD type 1). If you wanted to just append the data then insert the incremental data through multiple queries into the target one by one. By if you are inserting the data multiple times then the no of jobs will be more which be more or less equal to performing union over it. Sorry for the late reply. ... View more

Hi @Alexandros Biratsis I could see the work around available in the link which you have mentioned. Anyways let me add few points on top of it. Create a work table Perform union between target and incremental data and insert into the newly created work table Assuming that you are using only external table --> Drop the work table. re-create the target table pointing to the work table location so that you can avoid re-loading the target from the work table. Hope it helps! ... View more

@Ashnee Sharma How many executors are in place? Also are you firing the query in spark-sql directly? What is the size of the table which you are fetching? Try increasing the partitions manually instead of letting spark deciding the no of partitions. No of partitions can be decided based on the table size which has to be splitted across executors. Use the below properties set by SparkConf: conf.set("spark.driver.maxResultSize", "3g") set by spark-defaults.conf: spark.driver.maxResultSize 3g set when calling spark-submit: --conf spark.driver.maxResultSize=3g I believe the above property should work. I could see that you have increase the driver size already. If so then ignore the driver size change property. ... View more

@Sandeep SIngh No, hive doesn't maintain any lock history. show locks; Above command would help you to get the user who has acquired a lock over the table in hive. But however if the lock is released then you will not be able to see the user who has acquired the locks. Also there is no history of locks being recorded as it is not necessarily needed for the namenode for any computation. Hope it helps! ... View more

Hi @Ashnee Sharma Based on the logs I could see that when you run a count query it triggers a mapreduce job and it takes time. Could you check running this command (set hive.stats.fetch.column.stats) and verify that it status is true? Because when this property is enable then stats should be fetched based on stats information available in the metastore which will not trigger any jobs when you run a count query. It should work regardless whether you are using mr/tez as your execution engine. Hope it helps!! ... View more

Got it ! Thanks @James Dinkel ... View more