@Xin Yang could you share the DDL of the table t1. Use this command to get the DDL. show create table t1; ... View more

@mqureshi I have tried it but still the same result. Im unable to group only based on genres. If it is in a query i would have gone with select genres,count(*) from table_name group by genres. I would like to implement the same through pyspark. But stuck here. Any help would be appreciated much. ... View more

@mqureshi I dont think thats the issue here. Im able to perform actions like count(), collect() and take() over tags ... View more

@Varun R Below are few parameters which one would use in general. Ofcourse based on the logics other configuration settings cab also be modified. Enable Compression in Hive hive.exec.compress.output TRUE hive.exec.compress.intermediate TRUE hive.auto.convert.join hive.auto.convert.join.noconditionaltask hive.optimize.bucketmapjoin avoid order by and try to use sort by set hive.vectorized.execution.enabled = true; set hive.vectorized.execution.reduce.enabled = true; set hive.vectorized.execution.reduce.groupby.enabled = true; hive.compute.query.using.stats hive.stats.fetch.partition.stats hive.stats.fetch.column.stats hive.stats.autogather ANALYZE TABLE employee COMPUTE STATISTICS FOR COLUMNS; ANALYZE TABLE employee COMPUTE STATISTICS FOR COLUMNS id, dept; ... View more

Thanks @mqureshi . Thats how I have implemented it. But just wanted to understand why the above code doesnt work. ... View more

@Smart Solutions If source data is less than the block size on a daily basis then small file will be created per day which may be less than the block size. So many small files will be created on a daily basis. Once it is loaded into the table merge property has nothing to do it when you are loading the same table again. However it will have an impact when the source has many small files when the job is trigerred. And the answer for the other question. As the first table has 1200 small files and merge.mapredfiles is set to true will enable the mapper to read as much of files and combine it, if the size of the files is less than the block size. So once the mapred job completes it will merge as much of files and push into hive table. Hope it helps!! ... View more

I dont think window function works better when the data is huge. when you use windowing function all the data gets accumulated in a single reducer which will end up in performance issue. I would suggest to go with distinct/ group by function which you have mentioned to avoid such issues. ... View more

@Simran Kaur Try this hive -hiveconf dt=current_date -e 'select ${hiveconf:dt;' Hope it helps!! ... View more