Can not ALTER or DROP a big partitionned tables - CAUSED BY: MetaException: Timeout when executing..

[#1]

Hi,

I have a partitionned tables that have more than 50k partitions, it work a good except the Hive Metastore operations, like DROP and ALTER ... RENAME, I face this error message:

Query: drop table cars
ERROR: ImpalaRuntimeException: Error making 'dropTable' RPC to Hive Metastore: 
CAUSED BY: MetaException: Timeout when executing method: drop_table_with_environment_context; 600003ms exceeds 600000ms

I don't now if it's, a memory problem, or it's normal and I should adjust the timeout value.. ? if yes, which one ?

Thanks to help me ASAP.

[#2 · Accepted solution]

Hi,

This is a known issue and we have an internal JIRA to track it. Current there is no better way to improve the performance.

You can, however, to increase the timeout limit via "hive.metastore.client.socket.timeout", and set it to long time value (in seconds) to allow the query to finish. It should finish eventually, but might take sometime.

Our engineers are still discussing internally to see the best fix for the issue, however, it is on going and we do not have solution yet at this stage.

[#3 · Accepted solution]

Hi,

The first one "Hive Metastore Connection Timeout" is the one you should try. If you look closely, "hive.metastore.client.socket.timeout" is just underneath it.

Regarding the JIRA ID, it is an internal JIRA, so you do not have access to it.

Another way is to setup a quick script to drop partition in batches and and then drop the table after number of partitions have reduced to a reasonable level.

[#4]

Note: I use Impala Shell v2.9.0-cdh5.12.0 (03c6ddb) built on Thu Jun 29 04:17:31 PDT 2017
Thanks in advance.

[#5]

Can you try dropping the table using Hive?

[#6]

Thank you very muth for the answer man

I tried to drop it with the hive, but it give this error message:

hive> drop table cars;
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:hive.metastore.sasl.enabled can't be false in non-testing mode)

I think it's another problem that we have to fix, did you have any idea ?
NB: I use Sentry with hive.

[#7 · Accepted solution]

Hi,

This is a known issue and we have an internal JIRA to track it. Current there is no better way to improve the performance.

You can, however, to increase the timeout limit via "hive.metastore.client.socket.timeout", and set it to long time value (in seconds) to allow the query to finish. It should finish eventually, but might take sometime.

Our engineers are still discussing internally to see the best fix for the issue, however, it is on going and we do not have solution yet at this stage.

[#8]

Hi @EricL,
Thanks for the replay,

Can you give me the JIRA track link of this issue please ?
And about 'hive.metastore.client.socket.timeout' I find 3 config variables with the same name:

1- Hive: Hive Metastore Connection Timeout (5m).
2- Hive: in : Service Monitor Client Config Overrides (value: 60).
3- Impala: Catalog Server Hive Metastore Connection Timeout and Impala Daemon Hive Metastore Connection Timeout (value: 1h).

Can you help me to know which one is the concerned ?

Hope this issue iresolved ASAP.
Thanks again.

[#9 · Accepted solution]

Hi,

The first one "Hive Metastore Connection Timeout" is the one you should try. If you look closely, "hive.metastore.client.socket.timeout" is just underneath it.

Regarding the JIRA ID, it is an internal JIRA, so you do not have access to it.

Another way is to setup a quick script to drop partition in batches and and then drop the table after number of partitions have reduced to a reasonable level.

[#10]

Yes man, that what I do now in *drop* statement, but so far no easy solution for *alter*.
Thanks again @EricL

[#11]

Glad that I am helpful here.

[#12]

@EricL Just wanted to know if there is any progress on this issue ? I too face the same issue - wanted to drop a table with 50k partitions.