Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

Impala performance with HDFS caching enabled

SOLVED Go to solution

Impala performance with HDFS caching enabled

Expert Contributor

I got a table of size ~1gb and I tried to setup hdfs caching as described in this 'Using HDFS caching with Impala" doc, with replication factor of 2

 
I noticed that without hdfs caching the queries seem to be performing better. I'm using CDH 5.8.2. Is there anything I might be missing or can check why that is the case?
 
Thanks!
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Impala performance with HDFS caching enabled

Master Collaborator

What kind of performance difference are we talking about? 5%? 100%?

 

It's helpful to look at execution summaries or profiles to drill down on where the difference is (if you're using impala-shell, you can get them with the summary; and profile; commands after running a query).

 

If the whole data set you're querying fits in memory, HDFS caching may not be that beneficial, since the OS buffer cache can be pretty effective at keeping the data in memory, especially if you're re-running the same query on the same data back-to-back. Also if the query is somewhat complex, it can get CPU-bound pretty quickly.

2 REPLIES 2

Re: Impala performance with HDFS caching enabled

Master Collaborator

What kind of performance difference are we talking about? 5%? 100%?

 

It's helpful to look at execution summaries or profiles to drill down on where the difference is (if you're using impala-shell, you can get them with the summary; and profile; commands after running a query).

 

If the whole data set you're querying fits in memory, HDFS caching may not be that beneficial, since the OS buffer cache can be pretty effective at keeping the data in memory, especially if you're re-running the same query on the same data back-to-back. Also if the query is somewhat complex, it can get CPU-bound pretty quickly.

Re: Impala performance with HDFS caching enabled

Expert Contributor

Given the size of the dataset, I believe the data fits in memory and its not providing any additional performance improvement.

 

Thanks!