Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Table with Many Partitions has errors when queried.

Table with Many Partitions has errors when queried.

New Contributor

other non-partitioned and smaller number partition tables still work.

 

This table has 6,000 partitions.

 

in hive:

hive> select * from table_name limit 10;
FAILED: SemanticException org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out

 

 

In Impala:

 

Query: select count(1) , to_date(dt) from table_name group by to_date(dt) order by to_date(dt) limit 10000
ERROR: AnalysisException: Failed to load metadata for table: table_name
CAUSED BY: TableLoadingException: Failed to load metadata for table: table_name
CAUSED BY: TTransportException: java.net.SocketTimeoutException: Read timed out
CAUSED BY: SocketTimeoutException: Read timed out

4 REPLIES 4
Highlighted

Re: Table with Many Partitions has errors when queried.

Cloudera Employee

With that many partitions, it takes time to load them from the metastore.  Set hive.metastore.client.socket.timeout to a large number, like 300 (units are seconds).

 

You may also want/need to increase the heap size for the metastore.

 

Re: Table with Many Partitions has errors when queried.

New Contributor

Thanks,

 

We will change it and when we are able to restart it I will let you know how it worked.

Re: Table with Many Partitions has errors when queried.

Explorer
Hi,

I got the same problem, could you tell me how did you solve this problem? appreciate it~

Re: Table with Many Partitions has errors when queried.

New Contributor

I am facing same issue, pls provide the solution.