Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Load data and drop table commands failed with metastore connection failed error

Load data and drop table commands failed with metastore connection failed error

New Contributor

Beeline Error:

 LOAD  DATA  INPATH  '/tmp/actian_test.txt' OVERWRITE INTO TABLE t1_actian_table;
INFO  : Loading data to table default.t1_actian_table from hdfs://HDPUAT/tmp/actian_test.txt
No rows affected (143.199 seconds)
INFO  : [Warning] could not update stats.Failed with exception Unable to alter table. For direct MetaStore DB connections, we don't support retries at the client level.
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table. For direct MetaStore DB connections, we don't support retries at the client level.
  at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:459)
  at org.apache.hadoop.hive.ql.exec.StatsTask.aggregateStats(StatsTask.java:184)
  at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:118)
  at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
  at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:86)
  at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1630)
  at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1389)
  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1197)
  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1024)
  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1019)
  at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:153)
  at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:70)
  at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:205)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
  at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:536)
  at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:217)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
Caused by: MetaException(message:For direct MetaStore DB connections, we don't support retries at the client level.)
  at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:262)
  at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:88)
  at com.sun.proxy.$Proxy12.alter_table(Unknown Source)
  at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:457)
  ... 22 more

Hiveserver2.log shows following error.

==> hiveserver2.log <==
2015-12-21 21:18:00,377 WARN  [HiveServer2-Background-Pool: Thread-4240]: metastore.RetryingMetaStoreClient (RetryingMetaStoreClient.java:invoke(115)) - MetaStoreClient lost connection. Attempting to reconnect.
MetaException(message:For direct MetaStore DB connections, we don't support retries at the client level.)
  at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:262)
  at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:88)
  at com.sun.proxy.$Proxy12.getDatabase(Unknown Source)
  at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1253)
  at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1242)
  at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:2316)
  at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:416)
  at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
  at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:86)
  at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1630)
  at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1389)
  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1197)
  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1024)
  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1019)
  at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:153)
  at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:70)
  at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:205)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
  at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:536)
  at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:217)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)

Thanks

Srinivas

9 REPLIES 9

Re: Load data and drop table commands failed with metastore connection failed error

Hi @Srinivas Varala it looks like Hiveserver2 lost connection with your Metastore. What RDBMS you using for the metastore (the default is MySQL)? Can you please confirm that the RDBMS is up and running.

Alternatively, you can try bouncing the metastore and HiveServer2 from Ambari.

Cheers,

Andrew

Re: Load data and drop table commands failed with metastore connection failed error

@Srinivas Varala Andrew made a good point. Please check the metastore connections.

Re: Load data and drop table commands failed with metastore connection failed error

New Contributor

@Neeraj Sabharwal , @Andrew Watson

Thanks your your response. We bounced mysql server and all hive processes ( metastore and hiveserver2) but we are still gettting same errors. Mysql database is up and running.

Thanks

Srinivas

Re: Load data and drop table commands failed with metastore connection failed error

@Srinivas Varala Can you try to manually connecting to the MySQL DB from the Node that the HiveServer2 is sitting on? Does it properly connect or do you get an error?

mysql -h mysql.server.com -u hive_user_name -p

Re: Load data and drop table commands failed with metastore connection failed error

@Srinivas Varala You may want to remove server names. :)

Re: Load data and drop table commands failed with metastore connection failed error

@Srinivas Varala Are you able to load data using hive cli?

Re: Load data and drop table commands failed with metastore connection failed error

Mentor

@Srinivas Varala has this been resolved? Can you post your solution or accept best answer?

Highlighted

Re: Load data and drop table commands failed with metastore connection failed error

Contributor

@Neeraj Sabharwal, @Andrew Watson, @Srinivas Varala

HDP 2.3 / Ranger / MySQL

I have also exactly same issue. I am trying to drop the table. I tried bounced mysql server and all hive processes ( metastore and hiveserver2) but I am still getting same errors. Mysql database is up and running. All other Hive command including ALTER TABLE works fine on beeline. Please see my log AT https://community.hortonworks.com/questions/23990/dropping-hive-table-lost-metastore-connection.html

Re: Load data and drop table commands failed with metastore connection failed error

Maybe using old or incompatible mysql-connector?