Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Errors while running alter table and compute stats on tables with large number of partitions.

Highlighted

Errors while running alter table and compute stats on tables with large number of partitions.

Contributor

We are running Impala v2.7.0-cdh5.10.1 (876895d), and are facing similar issue.

 

While running alter table <table-name> recover partitions; - we get "Error communicating with impalad: TSocket read 0 bytes" error message. after multiple attempts to recover this table, we were finally able to recover partitions on the table.

 

At which point, we ran "compute stats <table-name>;" and received this error message: CatalogException: Table was modified during stats computation.

 

 

Although both these errors may be mutually exclusive, somehow they seems to only happen on the two tables with over 10,000 partition files.

 

3 REPLIES 3

Re: Errors while running alter table and compute stats on tables with large number of partitions.

Explorer

Did anyone look into this issue?

I am also facing the same issue.I am using CDH5.10.2 

Abhishek

Re: Errors while running alter table and compute stats on tables with large number of partitions.

Master Collaborator

If you're seeing a crash, any diagnostics are useful. E.g. log files with crash messages from /var/log/impalad, hs_err_pid*.log files, etc.

 

You can also file a bug report on the Apache JIRA:

https://issues.apache.org/jira/browse/IMPALA - it's generally easier to attach things and track things.
 
The crash could be something like https://issues.apache.org/jira/browse/IMPALA-2648 but I'm just speculating. Generally large numbers of partitions or files  can blow up the size of metadata and cause degraded performance.
 

Re: Errors while running alter table and compute stats on tables with large number of partitions.

Explorer

 

Below is the error message i received .

 

1137 2017-11-07 03:55:11,536  [INFO ] There are no more tasks to run at this time
1138 Starting Impala Shell without Kerberos authentication
1140 Server version: impalad version 2.6.0-cdh5.8.4 RELEASE (build 207450616f75adbe082a4c2e1145a2384da83fa6)
1141 Invalidating Metadata
1142 Query: invalidate metadata
1143
1144 Fetched 0 row(s) in 4.11s
1145 Query: use `DBNAME`
1146 Query: insert overwrite table Table partition(recordtype) select adid,seg,profile,livecount,
1147 count(distinct mc) as nofs,stbnt,1 from table1 where livecount<>0 group by adid,seg,profile,livecount,stbnt
1148 WARNINGS:
1149 CatalogException: Table 'dbname.table' was modified while operation was in progress, aborting execution.
1150
1151

Abhishek
Don't have an account?
Coming from Hortonworks? Activate your account here