Support Questions
Find answers, ask questions, and share your expertise

How to improve performance on a table/partition after running concatenate?

After running "concatenate" on a partition in hive I can see the no. of files drastically reduced. However when I run a simple select query with limit function it takes longer to display the result. My question: Is this a normal behaviour after concatenate or the query should execute faster after concatenate. Can someone clarify?


Expert Contributor

@knarendran to observe performance improvement with partitions, you must have a "WHERE" Clause on partition column, that would allow hive to perform desired partition scan of the table included in the "where" clause and skip non-desired partitions.