Re: Cloudera manager 5.5 install : HDFS error

dennisli · ‎05-06-2020

just rm -r the dir

dennisli · ‎03-22-2020

have you found a solution？ I'm facing the same issue. I tried using /hive-standalone-metastore-3.1.2000.7.0.3.0-79.jar and hive-metastore-2.1.1-cdh6.2.0.jar, but all of them seemed not working.

dennisli · ‎12-26-2019

Hi mike, did you try that? I'm also going to upgrade Hive with CDH 6.2

dennisli · ‎12-26-2019

I'm using CDH 6.2.0. and I found the version of Hive is 2.1.1. What I want is to upgrade Hive to 3.0 and keep all the Hive data (keep the metastore is enough?), but I can't find any document for upgrading hive in CDH? Is there any document or tutorial for that? Thanks a lot. Kind regards.

dennisli · ‎12-24-2019

Thanks a lot, I take a try.

dennisli · ‎12-23-2019

I simply set a new IP address to the server in my cluster by setting the DHCP in my router. But, after restart the cluster, CDH is still connecting the old IP address, so that it can't get connected. So, where can I update the IP address?

dennisli · ‎08-29-2019

Hi Michalis: I solved this by re-run the command: sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm mypassword This problem is due to my misunderstanding for scm_prepare_database.sh. Thanks a lot for your help. Kind regards

dennisli · ‎08-29-2019

Hi Michalis: I just found that, I apologize for my late reply. That time I cleaned all metastore and recreated the database in MySQL and it worked. Recently however, this error happend again, as I restarted the cloudera-scm-server and cloudera-scm-agent By following your instruction, I found an important information: the CM_VERSION is in scm database!! mysql> SELECT table_name, table_schema AS dbname -> FROM INFORMATION_SCHEMA.TABLES -> WHERE table_name='CM_VERSION'; +------------+--------+ | table_name | dbname | +------------+--------+ | CM_VERSION | scm | +------------+--------+ 1 row in set (2.11 sec) I don't know why it's gonna find in metastore database instead of scm. Could you give me some help, Thanks. Kind regards

dennisli · ‎07-23-2019

clear explanation! Thanks bgooley.

dennisli · ‎07-22-2019

@bgooley I have figured that out. Cause I changed the hive-site.xml in /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive/conf. Below is the wrong config. After I changed it back to the default config, it worked ! <?xml version="1.0" encoding="UTF-8"?>  <configuration> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://device1:3306/metastore?useUnicode=true&characterEncoding=UTF-8</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>mypassword</value> </property> <property> <name>hive.metastore.try.direct.sql.ddl</name> <value>true</value> </property> <property> <name>hive.metastore.try.direct.sql</name> <value>true</value> </property> <property> <name>datanucleus.schema.autoCreateAll</name> <value>false</value> </property> <property> <name>datanucleus.metadata.xml.validate</name> <value>false</value> </property> <property> <name>hive.metastore.schema.verification</name> <value>true</value> </property> <property> <name>datanucleus.autoStartMechanism</name> <value>SchemaTable</value> </property> <property> <name>hive.metastore.warehouse.dir</name> <value>/user/hive/warehouse</value> </property> <property> <name>hive.warehouse.subdir.inherit.perms</name> <value>true</value> </property> <property> <name>hive.server2.logging.operation.enabled</name> <value>true</value> </property> <property> <name>hive.server2.logging.operation.log.location</name> <value>/var/log/hive/operation_logs</value> </property> <property> <name>mapred.reduce.tasks</name> <value>-1</value> </property> <property> <name>hive.exec.reducers.bytes.per.reducer</name> <value>67108864</value> </property> <property> <name>hive.exec.copyfile.maxsize</name> <value>33554432</value> </property> <property> <name>hive.exec.reducers.max</name> <value>1099</value> </property> <property> <name>hive.metastore.execute.setugi</name> <value>true</value> </property> <property> <name>hive.support.concurrency</name> <value>true</value> </property> <property> <name>hive.zookeeper.quorum</name> <value>device1</value> </property> <property> <name>hive.zookeeper.client.port</name> <value>2181</value> </property> <property> <name>hive.zookeeper.namespace</name> <value>hive_zookeeper_namespace_hive</value> </property> <property> <name>hive.metastore.server.min.threads</name> <value>200</value> </property> <property> <name>hive.metastore.server.max.threads</name> <value>100000</value> </property> <property> <name>hive.cluster.delegation.token.store.class</name> <value>org.apache.hadoop.hive.thrift.MemoryTokenStore</value> </property> <property> <name>hive.metastore.fshandler.threads</name> <value>15</value> </property> <property> <name>hive.metastore.event.listeners</name> <value></value> </property> <property> <name>hive.metastore.server.max.message.size</name> <value>196188569</value> </property> <property> <name>hive.service.metrics.file.location</name> <value>/var/log/hive/metrics-hivemetastore/metrics.log</value> </property> <property> <name>hive.metastore.metrics.enabled</name> <value>true</value> </property> <property> <name>hive.service.metrics.file.frequency</name> <value>30000</value> </property> <property> <name>hadoop.security.credential.provider.path</name> <value>localjceks://file//var/run/cloudera-scm-agent/process/47-hive-metastore-validate/creds.localjceks</value> </property> </configuration> I compared the default config to the wrong config, default config use this property to connect metastore, while I use "javax.jdo.option.ConnectionURL" to connect metastore. <property> <name>hive.metastore.uris</name> <value>thrift://device1:9083</value> </property> Thanks a lot for your help.

dennisli · ‎07-22-2019

@bgooley I'm new to CM and I don't know what the machanism of CM behind, I just followed the document to install. In my cluster, there are two servers device1(master) and device2 . I can get the result properly on device2's hive command-line by: show databases; However, it can not run successfully on device1, and error log is metioned above. Then as regard to the gatway you mentioned, I found status of Gateway is None, is it normal? Thanks for your reply.

dennisli · ‎07-22-2019

@bgooley as one of the log said: Caused by: java.lang.RuntimeException: Error getting metastore password: null should I write down my metastore password in hive-site.xml? I checked the config file in `/run/cloudera-scm-agent/process/81-hive-HIVEMETASTORE/hive-site.xml` the part of config is like this, I'm not sure if this is ok, as the password is '******', should I replace it with the plaintext password? <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>********</value> </property>

dennisli · ‎07-22-2019

I can't even run show databases in hive command-line. I turned on debug mode can try again, it turned out as below: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1773) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3815) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3867) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3847) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4101) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:237) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:394) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:338) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:318) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:360) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:356) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:768) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:711) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1425) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1493) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1328) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) [hive-cli-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187) [hive-cli-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409) [hive-cli-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:836) [hive-cli-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:772) [hive-cli-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:699) [hive-cli-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] at org.apache.hadoop.util.RunJar.run(RunJar.java:313) [hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:227) [hadoop-common-3.0.0-cdh6.2.0.jar:?] Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_121] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1771) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] ... 32 more Caused by: java.lang.RuntimeException: Error getting metastore password: null at org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:571) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:298) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:687) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:653) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:647) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:716) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:419) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7034) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_121] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1771) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] ... 32 more Caused by: java.io.IOException at org.apache.hadoop.hive.shims.Hadoop23Shims.getPassword(Hadoop23Shims.java:965) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:566) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:298) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:687) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:653) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:647) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:716) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:419) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7034) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_121] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1771) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] ... 32 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] at org.apache.hadoop.hive.shims.Hadoop23Shims.getPassword(Hadoop23Shims.java:959) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:566) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:298) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:687) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:653) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:647) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:716) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:419) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7034) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_121] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1771) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] ... 32 more Caused by: java.io.IOException: Configuration problem with provider path. at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2272) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2191) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1771) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] ... 32 more Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:865) ~[sunjce_provider.jar:1.8.0_112] at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_121] at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:322) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:86) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:58) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:50) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider$Factory.createProvider(LocalJavaKeyStoreProvider.java:177) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:73) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2253) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2191) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] at org.apache.hadoop.hive.shims.Hadoop23Shims.getPassword(Hadoop23Shims.java:959) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:566) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:298) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:687) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:653) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:647) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:716) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:419) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7034) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_121] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1771) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] ... 32 more Caused by: java.security.UnrecoverableKeyException: Password verification failed at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:865) ~[sunjce_provider.jar:1.8.0_112] at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_121] at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:322) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:86) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:58) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:50) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider$Factory.createProvider(LocalJavaKeyStoreProvider.java:177) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:73) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2253) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2191) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] at org.apache.hadoop.hive.shims.Hadoop23Shims.getPassword(Hadoop23Shims.java:959) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:566) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:298) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:687) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:653) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:647) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:716) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:419) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7034) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_121] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1771) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] ... 32 more 2019-07-23T11:06:10,489 WARN [5b430558-3bb5-4632-8b6a-649bbc40138b main] ql.Driver: Caught exception attempting to write metadata call information org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:242) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:394) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:338) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:318) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:360) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:356) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.Driver.dumpMetaCallTimingWithoutEx(Driver.java:768) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:711) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1425) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1493) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1328) [hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) [hive-cli-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187) [hive-cli-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409) [hive-cli-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:836) [hive-cli-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:772) [hive-cli-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:699) [hive-cli-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] at org.apache.hadoop.util.RunJar.run(RunJar.java:313) [hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:227) [hadoop-common-3.0.0-cdh6.2.0.jar:?] Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1773) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3815) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3867) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3847) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4101) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:237) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] ... 23 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_121] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1771) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3815) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3867) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3847) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4101) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:237) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] ... 23 more Caused by: java.lang.RuntimeException: Error getting metastore password: null at org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:571) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:298) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:687) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:653) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:647) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:716) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:419) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7034) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_121] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1771) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3815) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3867) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3847) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4101) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:237) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] ... 23 more Caused by: java.io.IOException at org.apache.hadoop.hive.shims.Hadoop23Shims.getPassword(Hadoop23Shims.java:965) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:566) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:298) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:687) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:653) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:647) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:716) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:419) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7034) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_121] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1771) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3815) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3867) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3847) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4101) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:237) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] ... 23 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] at org.apache.hadoop.hive.shims.Hadoop23Shims.getPassword(Hadoop23Shims.java:959) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:566) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:298) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:687) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:653) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:647) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:716) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:419) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7034) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_121] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1771) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3815) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3867) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3847) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4101) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:237) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] ... 23 more Caused by: java.io.IOException: Configuration problem with provider path. at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2272) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2191) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] at org.apache.hadoop.hive.shims.Hadoop23Shims.getPassword(Hadoop23Shims.java:959) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:566) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:298) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:687) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:653) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:647) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:716) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:419) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7034) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_121] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_121] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_121] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121] at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1771) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3815) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3867) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3847) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4101) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:254) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:237) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] ... 23 more Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:865) ~[sunjce_provider.jar:1.8.0_112] at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_121] at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:322) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:86) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:58) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:237) ~[hive-exec-2.1.1-cdh6.2.0.jar:2.1.1-cdh6.2.0] ... 23 more Caused by: java.security.UnrecoverableKeyException: Password verification failed I think there is some problem with the metastore (I'm using mysql), but don't know how to fix it.

dennisli · ‎07-22-2019

Hi @bgooley I can use CM web UI to manipulate . But I also need to use hive command-line in bash environment.

dennisli · ‎07-22-2019

The reason why I execute "hive --service metastore" is that there was an Exception when I run hive -e "LOAD ..": FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

dennisli · ‎07-22-2019

When I run: hive --service metastore, it raised errors as below /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 24: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-hadoop2-compat.jar: Permission denied /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 25: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/lib/htrace-core.jar: Permission denied /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 26: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-common.jar: Permission denied /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 27: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-server.jar: Permission denied /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 28: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-hadoop-compat.jar: Permission denied /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hive/conf/hive-env.sh: line 29: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase/hbase-protocol.jar: Permission denied /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hadoop/libexec/hadoop-functions.sh: line 2331: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_USER: bad substitution /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/hadoop/libexec/hadoop-functions.sh: line 2426: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_OPTS: bad substitution Starting Hive Metastore Server WARNING: Use "yarn jar" to launch YARN applications. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] java.lang.RuntimeException: Error getting metastore password: null at org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:571) at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:298) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58) at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:687) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:653) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:647) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:716) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:419) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7028) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7023) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:7281) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:7208) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:313) at org.apache.hadoop.util.RunJar.main(RunJar.java:227) Caused by: java.io.IOException at org.apache.hadoop.hive.shims.Hadoop23Shims.getPassword(Hadoop23Shims.java:965) at org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:566) ... 22 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.shims.Hadoop23Shims.getPassword(Hadoop23Shims.java:959) ... 23 more Caused by: java.io.IOException: Configuration problem with provider path. at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2272) at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2191) ... 28 more Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:865) at java.security.KeyStore.load(KeyStore.java:1445) at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:322) at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:86) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:58) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:50) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider$Factory.createProvider(LocalJavaKeyStoreProvider.java:177) at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:73) at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2253) ... 29 more Caused by: java.security.UnrecoverableKeyException: Password verification failed ... 38 more Exception in thread "main" java.lang.RuntimeException: Error getting metastore password: null at org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:571) at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:298) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58) at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:687) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:653) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:647) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:716) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:419) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7028) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7023) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:7281) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:7208) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:313) at org.apache.hadoop.util.RunJar.main(RunJar.java:227) Caused by: java.io.IOException at org.apache.hadoop.hive.shims.Hadoop23Shims.getPassword(Hadoop23Shims.java:965) at org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:566) ... 22 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.shims.Hadoop23Shims.getPassword(Hadoop23Shims.java:959) ... 23 more Caused by: java.io.IOException: Configuration problem with provider path. at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2272) at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2191) ... 28 more Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:865) at java.security.KeyStore.load(KeyStore.java:1445) at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:322) at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:86) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:58) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider.<init>(LocalJavaKeyStoreProvider.java:50) at org.apache.hadoop.security.alias.LocalJavaKeyStoreProvider$Factory.createProvider(LocalJavaKeyStoreProvider.java:177) at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:73) at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2253) ... 29 more Caused by: java.security.UnrecoverableKeyException: Password verification failed ... 38 more Who can give me some hint, Thanks.

dennisli · ‎07-18-2019

yes me too, solved by cleanning the /dfs/nn and /dfs/dn

dennisli · ‎07-18-2019

I solved it by cleaning the /dfs/nn and /dfs/dn because once I installed hdfs, and there were some data existed in the hdfs.

dennisli · ‎07-18-2019

This is the stderr.log [18/Jul/2019 15:52:04 +0000] 2807 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/hdfs/hdfs.sh format-namenode cluster19 [18/Jul/2019 15:52:04 +0000] 2807 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 2811 [18/Jul/2019 15:52:04 +0000] 2812 MainThread redactor INFO Re-exec redactor: /opt/cloudera/cm-agent/bin/cm redactor --fds 3 5 [18/Jul/2019 15:52:04 +0000] 2812 MainThread redactor INFO Started redactor 2019年 07月 18日星期四 15:52:04 CST + source_parcel_environment + '[' '!' -z /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/meta/cdh_env.sh ']' + OLD_IFS=' ' + IFS=: + SCRIPT_ARRAY=($SCM_DEFINES_SCRIPTS) + DIRNAME_ARRAY=($PARCEL_DIRNAMES) + IFS=' ' + COUNT=1 ++ seq 1 1 + for i in `seq 1 $COUNT` + SCRIPT=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/meta/cdh_env.sh + PARCEL_DIRNAME=CDH-6.2.0-1.cdh6.2.0.p0.967373 + . /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/meta/cdh_env.sh ++ CDH_DIRNAME=CDH-6.2.0-1.cdh6.2.0.p0.967373 ++ export CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop ++ CDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop ++ export CDH_MR1_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-0.20-mapreduce ++ CDH_MR1_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-0.20-mapreduce ++ export CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-hdfs ++ CDH_HDFS_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-hdfs ++ export CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-httpfs ++ CDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-httpfs ++ export CDH_MR2_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-mapreduce ++ CDH_MR2_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-mapreduce ++ export CDH_YARN_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-yarn ++ CDH_YARN_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-yarn ++ export CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase ++ CDH_HBASE_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase ++ export CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/zookeeper ++ CDH_ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/zookeeper ++ export CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive ++ CDH_HIVE_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive ++ export CDH_HUE_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue ++ CDH_HUE_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hue ++ export CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie ++ CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie ++ export CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop ++ CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop ++ export CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/flume-ng ++ CDH_FLUME_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/flume-ng ++ export CDH_PIG_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/pig ++ CDH_PIG_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/pig ++ export CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive-hcatalog ++ CDH_HCAT_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive-hcatalog ++ export CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/sentry ++ CDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/sentry ++ export JSVC_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/bigtop-utils ++ JSVC_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/bigtop-utils ++ export CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/bin/hadoop ++ CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/bin/hadoop ++ export CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/impala ++ CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/impala ++ export CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/solr ++ CDH_SOLR_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/solr ++ export CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase-solr ++ CDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hbase-solr ++ export SEARCH_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/search ++ SEARCH_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/search ++ export CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark ++ CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark ++ export WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/etc/hive-webhcat/conf.dist/webhcat-default.xml ++ WEBHCAT_DEFAULT_XML=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/etc/hive-webhcat/conf.dist/webhcat-default.xml ++ export CDH_KMS_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-kms ++ CDH_KMS_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-kms ++ export CDH_PARQUET_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/parquet ++ CDH_PARQUET_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/parquet ++ export CDH_AVRO_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/avro ++ CDH_AVRO_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/avro ++ export CDH_KAFKA_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/kafka ++ CDH_KAFKA_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/kafka ++ export CDH_KUDU_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/kudu ++ CDH_KUDU_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/kudu + locate_cdh_java_home + '[' -z '' ']' + '[' -z /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/bigtop-utils ']' + local BIGTOP_DETECT_JAVAHOME= + for candidate in "${JSVC_HOME}" "${JSVC_HOME}/.." "/usr/lib/bigtop-utils" "/usr/libexec" + '[' -e /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/bigtop-utils/bigtop-detect-javahome ']' + BIGTOP_DETECT_JAVAHOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/bigtop-utils/bigtop-detect-javahome + break + '[' -z /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/bigtop-utils/bigtop-detect-javahome ']' + . /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/bigtop-utils/bigtop-detect-javahome ++ BIGTOP_DEFAULTS_DIR=/etc/default ++ '[' -n /etc/default -a -r /etc/default/bigtop-utils ']' ++ JAVA11_HOME_CANDIDATES=('/usr/java/jdk-11' '/usr/lib/jvm/jdk-11' '/usr/lib/jvm/java-11-oracle') ++ OPENJAVA11_HOME_CANDIDATES=('/usr/lib/jvm/jdk-11' '/usr/lib64/jvm/jdk-11') ++ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle') ++ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk') ++ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk') ++ case ${BIGTOP_JAVA_MAJOR} in ++ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${JAVA11_HOME_CANDIDATES[@]} ${OPENJAVA11_HOME_CANDIDATES[@]}) ++ '[' -z '' ']' ++ for candidate_regex in ${JAVA_HOME_CANDIDATES[@]} +++ ls -rvd '/usr/java/jdk1.8*' ++ for candidate_regex in ${JAVA_HOME_CANDIDATES[@]} +++ ls -rvd '/usr/java/jre1.8*' ++ for candidate_regex in ${JAVA_HOME_CANDIDATES[@]} +++ ls -rvd '/usr/lib/jvm/j2sdk1.8-oracle*' ++ for candidate_regex in ${JAVA_HOME_CANDIDATES[@]} +++ ls -rvd '/usr/lib/jvm/j2sdk1.8-oracle/jre*' ++ for candidate_regex in ${JAVA_HOME_CANDIDATES[@]} +++ ls -rvd '/usr/lib/jvm/java-8-oracle*' ++ for candidate_regex in ${JAVA_HOME_CANDIDATES[@]} +++ ls -rvd '/Library/Java/Home*' ++ for candidate_regex in ${JAVA_HOME_CANDIDATES[@]} +++ ls -rvd '/usr/java/default*' ++ for candidate_regex in ${JAVA_HOME_CANDIDATES[@]} +++ ls -rvd '/usr/lib/jvm/default-java*' ++ for candidate_regex in ${JAVA_HOME_CANDIDATES[@]} +++ ls -rvd '/usr/lib/jvm/java-openjdk*' ++ for candidate_regex in ${JAVA_HOME_CANDIDATES[@]} +++ ls -rvd '/usr/lib/jvm/jre-openjdk*' ++ for candidate_regex in ${JAVA_HOME_CANDIDATES[@]} +++ ls -rvd /usr/lib/jvm/java-1.8.0-openjdk-amd64 ++ for candidate in `ls -rvd ${candidate_regex}* 2>/dev/null` ++ '[' -e /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java ']' ++ export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64 ++ JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64 ++ break 2 + get_java_major_version JAVA_MAJOR + '[' -z /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java ']' ++ /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -version + local 'VERSION_STRING=openjdk version "1.8.0_162" OpenJDK Runtime Environment (build 1.8.0_162-8u162-b12-1-b12) OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode)' + local 'RE_JAVA=[java|openjdk][[:space:]]version[[:space:]]\"1\.([0-9][0-9]*)\.?+' + [[ openjdk version "1.8.0_162" OpenJDK Runtime Environment (build 1.8.0_162-8u162-b12-1-b12) OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode) =~ [java|openjdk][[:space:]]version[[:space:]]\"1\.([0-9][0-9]*)\.?+ ]] + eval JAVA_MAJOR=8 ++ JAVA_MAJOR=8 + '[' 8 -lt 8 ']' + verify_java_home + '[' -z /usr/lib/jvm/java-1.8.0-openjdk-amd64 ']' + echo JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64 + . /opt/cloudera/cm-agent/service/common/cdh-default-hadoop ++ [[ -z 6 ]] ++ '[' 6 = 3 ']' ++ '[' 6 = -3 ']' ++ '[' 6 -ge 4 ']' ++ export HADOOP_HOME_WARN_SUPPRESS=true ++ HADOOP_HOME_WARN_SUPPRESS=true ++ export HADOOP_PREFIX=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop ++ HADOOP_PREFIX=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop ++ export HADOOP_LIBEXEC_DIR=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/libexec ++ HADOOP_LIBEXEC_DIR=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/libexec ++ export HADOOP_CONF_DIR=/var/run/cloudera-scm-agent/process/40-hdfs-NAMENODE-format ++ HADOOP_CONF_DIR=/var/run/cloudera-scm-agent/process/40-hdfs-NAMENODE-format ++ export HADOOP_COMMON_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop ++ HADOOP_COMMON_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop ++ export HADOOP_HDFS_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-hdfs ++ HADOOP_HDFS_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-hdfs ++ export HADOOP_MAPRED_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-mapreduce ++ HADOOP_MAPRED_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-mapreduce ++ '[' 6 = 4 ']' ++ '[' 6 -ge 5 ']' ++ export HADOOP_YARN_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-yarn ++ HADOOP_YARN_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-yarn ++ replace_pid -Xms1961885696 -Xmx1961885696 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/hdfs_hdfs-NAMENODE-04c87db2a470ce2ac21ae3a8547c2609_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh ++ sed 's#{{PID}}#2811#g' ++ echo -Xms1961885696 -Xmx1961885696 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/hdfs_hdfs-NAMENODE-04c87db2a470ce2ac21ae3a8547c2609_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh + export 'HADOOP_NAMENODE_OPTS=-Xms1961885696 -Xmx1961885696 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/hdfs_hdfs-NAMENODE-04c87db2a470ce2ac21ae3a8547c2609_pid2811.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh' + HADOOP_NAMENODE_OPTS='-Xms1961885696 -Xmx1961885696 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/hdfs_hdfs-NAMENODE-04c87db2a470ce2ac21ae3a8547c2609_pid2811.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh' ++ replace_pid ++ echo ++ sed 's#{{PID}}#2811#g' + export HADOOP_DATANODE_OPTS= + HADOOP_DATANODE_OPTS= ++ replace_pid ++ echo ++ sed 's#{{PID}}#2811#g' + export HADOOP_SECONDARYNAMENODE_OPTS= + HADOOP_SECONDARYNAMENODE_OPTS= ++ replace_pid ++ echo ++ sed 's#{{PID}}#2811#g' + export HADOOP_NFS3_OPTS= + HADOOP_NFS3_OPTS= ++ replace_pid ++ echo ++ sed 's#{{PID}}#2811#g' + export HADOOP_JOURNALNODE_OPTS= + HADOOP_JOURNALNODE_OPTS= + '[' 6 -ge 4 ']' + HDFS_BIN=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-hdfs/bin/hdfs + export 'HADOOP_OPTS=-Djava.net.preferIPv4Stack=true ' + HADOOP_OPTS='-Djava.net.preferIPv4Stack=true ' + echo 'using /usr/lib/jvm/java-1.8.0-openjdk-amd64 as JAVA_HOME' + echo 'using 6 as CDH_VERSION' + echo 'using /var/run/cloudera-scm-agent/process/40-hdfs-NAMENODE-format as CONF_DIR' + echo 'using as SECURE_USER' + echo 'using as SECURE_GROUP' + set_hadoop_classpath + set_classpath_in_var HADOOP_CLASSPATH + '[' -z HADOOP_CLASSPATH ']' + [[ -n /opt/cloudera/cm ]] ++ find /opt/cloudera/cm/lib/plugins -maxdepth 1 -name '*.jar' ++ tr '\n' : + ADD_TO_CP=/opt/cloudera/cm/lib/plugins/tt-instrumentation-6.2.0.jar:/opt/cloudera/cm/lib/plugins/event-publish-6.2.0-shaded.jar: + [[ -n navigator/cdh6 ]] + for DIR in $CM_ADD_TO_CP_DIRS ++ find /opt/cloudera/cm/lib/plugins/navigator/cdh6 -maxdepth 1 -name '*.jar' ++ tr '\n' : + PLUGIN=/opt/cloudera/cm/lib/plugins/navigator/cdh6/audit-plugin-cdh6-6.2.0-shaded.jar: + ADD_TO_CP=/opt/cloudera/cm/lib/plugins/tt-instrumentation-6.2.0.jar:/opt/cloudera/cm/lib/plugins/event-publish-6.2.0-shaded.jar:/opt/cloudera/cm/lib/plugins/navigator/cdh6/audit-plugin-cdh6-6.2.0-shaded.jar: + eval 'OLD_VALUE=$HADOOP_CLASSPATH' ++ OLD_VALUE= + NEW_VALUE=/opt/cloudera/cm/lib/plugins/tt-instrumentation-6.2.0.jar:/opt/cloudera/cm/lib/plugins/event-publish-6.2.0-shaded.jar:/opt/cloudera/cm/lib/plugins/navigator/cdh6/audit-plugin-cdh6-6.2.0-shaded.jar: + export HADOOP_CLASSPATH=/opt/cloudera/cm/lib/plugins/tt-instrumentation-6.2.0.jar:/opt/cloudera/cm/lib/plugins/event-publish-6.2.0-shaded.jar:/opt/cloudera/cm/lib/plugins/navigator/cdh6/audit-plugin-cdh6-6.2.0-shaded.jar + HADOOP_CLASSPATH=/opt/cloudera/cm/lib/plugins/tt-instrumentation-6.2.0.jar:/opt/cloudera/cm/lib/plugins/event-publish-6.2.0-shaded.jar:/opt/cloudera/cm/lib/plugins/navigator/cdh6/audit-plugin-cdh6-6.2.0-shaded.jar + set -x + replace_conf_dir + echo CONF_DIR=/var/run/cloudera-scm-agent/process/40-hdfs-NAMENODE-format + echo CMF_CONF_DIR= + EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks') ++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' hdfs.keytab '*jceks' + find /var/run/cloudera-scm-agent/process/40-hdfs-NAMENODE-format -type f '!' -path '/var/run/cloudera-scm-agent/process/40-hdfs-NAMENODE-format/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name hdfs.keytab '!' -name '*jceks' -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/40-hdfs-NAMENODE-format#g' '{}' ';' Can't open /var/run/cloudera-scm-agent/process/40-hdfs-NAMENODE-format/supervisor_status: Permission denied. + make_scripts_executable + find /var/run/cloudera-scm-agent/process/40-hdfs-NAMENODE-format -regex '.*\.$py\|sh$$' -exec chmod u+x '{}' ';' + '[' DATANODE_MAX_LOCKED_MEMORY '!=' '' ']' + ulimit -l + export HADOOP_IDENT_STRING=hdfs + HADOOP_IDENT_STRING=hdfs + '[' -n '' ']' + '[' mkdir '!=' format-namenode ']' + acquire_kerberos_tgt hdfs.keytab + '[' -z hdfs.keytab ']' + KERBEROS_PRINCIPAL= + '[' '!' -z '' ']' + '[' -n '' ']' + '[' validate-writable-empty-dirs = format-namenode ']' + '[' file-operation = format-namenode ']' + '[' bootstrap = format-namenode ']' + '[' failover = format-namenode ']' + '[' transition-to-active = format-namenode ']' + '[' initializeSharedEdits = format-namenode ']' + '[' initialize-znode = format-namenode ']' + '[' format-namenode = format-namenode ']' + '[' -z /dfs/nn ']' + for dfsdir in $DFS_STORAGE_DIRS + '[' -e /dfs/nn ']' + '[' '!' -d /dfs/nn ']' + CLUSTER_ARGS= + '[' 2 -eq 2 ']' + CLUSTER_ARGS='-clusterId cluster19' + '[' 3 = 6 ']' + '[' -3 = 6 ']' + exec /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-hdfs/bin/hdfs --config /var/run/cloudera-scm-agent/process/40-hdfs-NAMENODE-format namenode -format -clusterId cluster19 -nonInteractive WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX. WARNING: HADOOP_NAMENODE_OPTS has been replaced by HDFS_NAMENODE_OPTS. Using value of HADOOP_NAMENODE_OPTS. Running in non-interactive mode, and data appears to exist in Storage Directory /dfs/nn. Not formatting.

dennisli · ‎07-18-2019

This is the namenode log, and there is no any error STARTUP_MSG: build = http://github.com/cloudera/hadoop -r d1dff3d3a126da44e3458bbf148c3bc16ff55bd8; compiled by 'jenkins' on 2019-03-14T06:39Z STARTUP_MSG: java = 1.8.0_162 ************************************************************/ 2019-07-18 15:52:05,190 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 2019-07-18 15:52:05,234 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode [-format, -clusterId, cluster19, -nonInteractive] 2019-07-18 15:52:05,485 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Edit logging is async:true 2019-07-18 15:52:05,494 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: KeyProvider: null 2019-07-18 15:52:05,494 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsLock is fair: true 2019-07-18 15:52:05,495 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Detailed lock hold time metrics enabled: false 2019-07-18 15:52:05,498 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner = hdfs (auth:SIMPLE) 2019-07-18 15:52:05,499 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup = supergroup 2019-07-18 15:52:05,499 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled = true 2019-07-18 15:52:05,499 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false 2019-07-18 15:52:05,522 INFO org.apache.hadoop.hdfs.server.common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling 2019-07-18 15:52:05,532 WARN org.apache.hadoop.hdfs.util.CombinedHostsFileReader: /var/run/cloudera-scm-agent/process/40-hdfs-NAMENODE-format/dfs_all_hosts.txt has invalid JSON format.Try the old format without top-level token defined. 2019-07-18 15:52:05,573 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000 2019-07-18 15:52:05,573 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 2019-07-18 15:52:05,575 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 2019-07-18 15:52:05,576 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: The block deletion will start around 2019 Jul 18 15:52:05 2019-07-18 15:52:05,577 INFO org.apache.hadoop.util.GSet: Computing capacity for map BlocksMap 2019-07-18 15:52:05,577 INFO org.apache.hadoop.util.GSet: VM type = 64-bit 2019-07-18 15:52:05,578 INFO org.apache.hadoop.util.GSet: 2.0% max memory 1.8 GB = 36.2 MB 2019-07-18 15:52:05,579 INFO org.apache.hadoop.util.GSet: capacity = 2^22 = 4194304 entries 2019-07-18 15:52:05,598 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable = false 2019-07-18 15:52:05,601 INFO org.apache.hadoop.conf.Configuration.deprecation: No unit for dfs.namenode.safemode.extension(30000) assuming MILLISECONDS 2019-07-18 15:52:05,601 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 2019-07-18 15:52:05,602 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 1 2019-07-18 15:52:05,602 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000 2019-07-18 15:52:05,602 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication = 3 2019-07-18 15:52:05,602 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication = 512 2019-07-18 15:52:05,602 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication = 1 2019-07-18 15:52:05,602 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams = 20 2019-07-18 15:52:05,602 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: redundancyRecheckInterval = 3000ms 2019-07-18 15:52:05,602 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer = false 2019-07-18 15:52:05,602 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxNumBlocksToLog = 1000 2019-07-18 15:52:05,618 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: GLOBAL serial map: bits=24 maxEntries=16777215 2019-07-18 15:52:05,627 INFO org.apache.hadoop.util.GSet: Computing capacity for map INodeMap 2019-07-18 15:52:05,627 INFO org.apache.hadoop.util.GSet: VM type = 64-bit 2019-07-18 15:52:05,627 INFO org.apache.hadoop.util.GSet: 1.0% max memory 1.8 GB = 18.1 MB 2019-07-18 15:52:05,627 INFO org.apache.hadoop.util.GSet: capacity = 2^21 = 2097152 entries 2019-07-18 15:52:05,630 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: ACLs enabled? false 2019-07-18 15:52:05,630 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: POSIX ACL inheritance enabled? true 2019-07-18 15:52:05,630 INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: XAttrs enabled? true 2019-07-18 15:52:05,630 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occurring more than 10 times 2019-07-18 15:52:05,634 INFO org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager: Loaded config captureOpenFiles: true, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true 2019-07-18 15:52:05,637 INFO org.apache.hadoop.util.GSet: Computing capacity for map cachedBlocks 2019-07-18 15:52:05,637 INFO org.apache.hadoop.util.GSet: VM type = 64-bit 2019-07-18 15:52:05,637 INFO org.apache.hadoop.util.GSet: 0.25% max memory 1.8 GB = 4.5 MB 2019-07-18 15:52:05,637 INFO org.apache.hadoop.util.GSet: capacity = 2^19 = 524288 entries 2019-07-18 15:52:05,643 INFO org.apache.hadoop.hdfs.server.namenode.top.metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10 2019-07-18 15:52:05,643 INFO org.apache.hadoop.hdfs.server.namenode.top.metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 2019-07-18 15:52:05,643 INFO org.apache.hadoop.hdfs.server.namenode.top.metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 2019-07-18 15:52:05,646 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode is enabled 2019-07-18 15:52:05,646 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 2019-07-18 15:52:05,648 INFO org.apache.hadoop.util.GSet: Computing capacity for map NameNodeRetryCache 2019-07-18 15:52:05,648 INFO org.apache.hadoop.util.GSet: VM type = 64-bit 2019-07-18 15:52:05,648 INFO org.apache.hadoop.util.GSet: 0.029999999329447746% max memory 1.8 GB = 555.9 KB 2019-07-18 15:52:05,648 INFO org.apache.hadoop.util.GSet: capacity = 2^16 = 65536 entries 2019-07-18 15:52:05,666 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: ExitException 2019-07-18 15:52:05,667 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at device1/192.168.0.104 ************************************************************/

dennisli · ‎07-17-2019

Thanks Eric , I'll take a shot for that.

dennisli · ‎07-17-2019

I wanna add a new server to cluster. But there was an issue in the last link. My namenode is device1, and I was going to add device2 to the cluster. The information printed as below: Failed to perform First Run of services. Completed 1 of 1 step(s). Show All Steps Show Only Failed Steps Show Only Running Steps Run a set of services for the first time Completed only 4/5 steps. First failure: Command (Format (102)) has failed Jul 18, 12:52:45 PM 22.51s Execute 7 steps in sequence Completed only 4/5 steps. First failure: Command (Format (102)) has failed Jul 18, 12:52:45 PM 22.45s Execute 5 steps in parallel Completed only 4/5 steps. First failure: Command (Format (102)) has failed Jul 18, 12:52:50 PM 17.34s Formatting the name directories of the current NameNode. If the name directories are not empty, this is expected to fail. Command (Format (102)) has failed NameNode (device1) Jul 18, 12:52:50 PM 17.34s Format Failed to format NameNode. NameNode (device1) Jul 18, 12:52:50 PM 17.33s $> hdfs/hdfs.sh ["format-namenode","cluster19"] stdout stderr Role Log Can't open /var/run/cloudera-scm-agent/process/32-hdfs-NAMENODE-format/supervisor_status: Permission denied. + make_scripts_executable + find /var/run/cloudera-scm-agent/process/32-hdfs-NAMENODE-format -regex '.*\.$py\|sh$$' -exec chmod u+x '{}' ';' + '[' DATANODE_MAX_LOCKED_MEMORY '!=' '' ']' + ulimit -l + export HADOOP_IDENT_STRING=hdfs + HADOOP_IDENT_STRING=hdfs + '[' -n '' ']' + '[' mkdir '!=' format-namenode ']' + acquire_kerberos_tgt hdfs.keytab + '[' -z hdfs.keytab ']' + KERBEROS_PRINCIPAL= + '[' '!' -z '' ']' + '[' -n '' ']' + '[' validate-writable-empty-dirs = format-namenode ']' + '[' file-operation = format-namenode ']' + '[' bootstrap = format-namenode ']' + '[' failover = format-namenode ']' + '[' transition-to-active = format-namenode ']' + '[' initializeSharedEdits = format-namenode ']' + '[' initialize-znode = format-namenode ']' + '[' format-namenode = format-namenode ']' + '[' -z /dfs/nn ']' + for dfsdir in $DFS_STORAGE_DIRS + '[' -e /dfs/nn ']' + '[' '!' -d /dfs/nn ']' + CLUSTER_ARGS= + '[' 2 -eq 2 ']' + CLUSTER_ARGS='-clusterId cluster19' + '[' 3 = 6 ']' + '[' -3 = 6 ']' + exec /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop-hdfs/bin/hdfs --config /var/run/cloudera-scm-agent/process/32-hdfs-NAMENODE-format namenode -format -clusterId cluster19 -nonInteractive WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX. WARNING: HADOOP_NAMENODE_OPTS has been replaced by HDFS_NAMENODE_OPTS. Using value of HADOOP_NAMENODE_OPTS. Running in non-interactive mode, and data appears to exist in Storage Directory /dfs/nn. Not formatting. screenshot

dennisli · ‎07-16-2019

Thanks Michalis, just checked the db.properties as below: -rw-r--r-- 1 cloudera-scm cloudera-scm 757 3月 15 01:45 /etc/cloudera-scm-server/db.properties.~1~ -rw------- 1 cloudera-scm cloudera-scm 446 6月 18 19:37 /etc/cloudera-scm-server/db.properties.~2~ -rw------- 1 cloudera-scm cloudera-scm 448 6月 18 19:39 /etc/cloudera-scm-server/db.properties.~3~ -rw------- 1 cloudera-scm cloudera-scm 448 6月 18 19:39 /etc/cloudera-scm-server/db.properties.~4~ -rw------- 1 cloudera-scm cloudera-scm 446 6月 18 19:40 /etc/cloudera-scm-server/db.properties.~5~ -rw------- 1 cloudera-scm cloudera-scm 448 6月 18 19:40 /etc/cloudera-scm-server/db.properties.~6~ -rw------- 1 cloudera-scm cloudera-scm 443 6月 18 19:41 /etc/cloudera-scm-server/db.properties.~7~ -rw------- 1 cloudera-scm cloudera-scm 447 6月 18 19:41 /etc/cloudera-scm-server/db.properties.~8~ -rw------- 1 cloudera-scm cloudera-scm 441 6月 18 19:41 /etc/cloudera-scm-server/db.properties.~9~ -rw------- 1 cloudera-scm cloudera-scm 446 6月 18 19:41 /etc/cloudera-scm-server/db.properties.~10~ -rw------- 1 cloudera-scm cloudera-scm 452 6月 18 19:42 /etc/cloudera-scm-server/db.properties.~11~ -rw------- 1 cloudera-scm cloudera-scm 450 6月 18 19:42 /etc/cloudera-scm-server/db.properties.~12~ -rw------- 1 cloudera-scm cloudera-scm 450 6月 18 19:42 /etc/cloudera-scm-server/db.properties.~13~ -rw------- 1 cloudera-scm cloudera-scm 453 6月 18 19:43 /etc/cloudera-scm-server/db.properties.~14~ -rw------- 1 cloudera-scm cloudera-scm 446 6月 18 21:20 /etc/cloudera-scm-server/db.properties.~15~ -rw------- 1 cloudera-scm cloudera-scm 453 6月 19 13:05 /etc/cloudera-scm-server/db.properties.~16~ -rw------- 1 cloudera-scm cloudera-scm 442 7月 16 14:34 /etc/cloudera-scm-server/db.properties.~17~ -rw------- 1 cloudera-scm cloudera-scm 446 7月 16 14:34 /etc/cloudera-scm-server/db.properties.~18~ -rw------- 1 cloudera-scm cloudera-scm 458 7月 16 14:37 /etc/cloudera-scm-server/db.properties.~19~ -rw------- 1 cloudera-scm cloudera-scm 453 7月 16 14:37 /etc/cloudera-scm-server/db.properties.~20~ -rw------- 1 cloudera-scm cloudera-scm 453 7月 16 15:02 /etc/cloudera-scm-server/db.properties.~21~ -rw------- 1 cloudera-scm cloudera-scm 453 7月 16 15:03 /etc/cloudera-scm-server/db.properties.~22~ -rw------- 1 cloudera-scm cloudera-scm 453 7月 16 15:04 /etc/cloudera-scm-server/db.properties.~23~ -rw------- 1 cloudera-scm cloudera-scm 453 7月 16 16:46 /etc/cloudera-scm-server/db.properties.~24~ -rw------- 1 cloudera-scm cloudera-scm 453 7月 16 16:47 /etc/cloudera-scm-server/db.properties.~25~ -rw------- 1 cloudera-scm cloudera-scm 454 7月 16 16:47 /etc/cloudera-scm-server/db.properties.~26~ -rw------- 1 cloudera-scm cloudera-scm 453 7月 16 16:47 /etc/cloudera-scm-server/db.properties.~27~ -rw------- 1 cloudera-scm cloudera-scm 446 7月 16 17:03 /etc/cloudera-scm-server/db.properties.~28~ -rw------- 1 cloudera-scm cloudera-scm 446 7月 16 17:03 /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.type=mysql com.cloudera.cmf.db.host=localhost com.cloudera.cmf.db.name=scm com.cloudera.cmf.db.user=scm com.cloudera.cmf.db.setupType=EXTERNAL com.cloudera.cmf.db.password=mypassword few days ago my CM started successfully. But, after restarted my server, it raised that error when I started. Now I tried to drop all the databases in mysql and create again, then run the following command respectively : sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm mypassword sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql amon amon mypassword sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql rman rman mypassword sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql hue hue mypassword sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm mypassword sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql metastore hive mypassword sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql sentry sentry mypassword sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql nav nav mypassword sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql navms navms mypassword sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql oozie oozie mypassword

dennisli · ‎07-16-2019

Hi Eric , my db.properties is as this: com.cloudera.cmf.db.type=mysql com.cloudera.cmf.db.host=localhost com.cloudera.cmf.db.name=scm com.cloudera.cmf.db.user=scm com.cloudera.cmf.db.setupType=EXTERNAL com.cloudera.cmf.db.password=mypassword And I just used the command to configure: sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql metastore hive mypassword It started successfully in the beginning, but it occured that error few days later after I restarted my server. now I dropped the database scm and try starting again.

dennisli · ‎07-16-2019

I 'm using CDH-6.2.0-1.cdh6.2.0. and I use mysql as metastore 1. I have already configured the mysql meta first by running: sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql metastore hive 2. I use this command to start: sudo systemctl start cloudera-scm-server. Then it tells me as below: 2019-07-16 15:06:38,947 ERROR main:com.cloudera.server.cmf.Main: Server failed. org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Unable to obtain CM release version. at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108) at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:634) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1198) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1100) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543) at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:418) at com.cloudera.server.cmf.Main.<init>(Main.java:260) at com.cloudera.server.cmf.Main.main(Main.java:233) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Unable to obtain CM release version. at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1640) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351) ... 17 more Caused by: java.lang.RuntimeException: Unable to obtain CM release version. at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getPersistedCmRelease(EntityManagerFactoryBean.java:311) at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.checkVersionDoFail(EntityManagerFactoryBean.java:358) at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:172) at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:75) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178) ... 22 more Caused by: javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not extract ResultSet at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:147) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:155) at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1407) at org.hibernate.query.internal.AbstractProducedQuery.getSingleResult(AbstractProducedQuery.java:1444) at com.cloudera.cmf.persist.DbVersionDao.getVersion(DbVersionDao.java:53) at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getPersistedCmRelease(EntityManagerFactoryBean.java:307) ... 26 more Caused by: org.hibernate.exception.SQLGrammarException: could not extract ResultSet at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:63) at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79) at org.hibernate.loader.Loader.getResultSet(Loader.java:2123) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1911) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1887) at org.hibernate.loader.Loader.doQuery(Loader.java:932) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349) at org.hibernate.loader.Loader.doList(Loader.java:2615) at org.hibernate.loader.Loader.doList(Loader.java:2598) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2430) at org.hibernate.loader.Loader.list(Loader.java:2425) at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:335) at org.hibernate.internal.SessionImpl.listCustomQuery(SessionImpl.java:2128) at org.hibernate.internal.AbstractSharedSessionContract.list(AbstractSharedSessionContract.java:991) at org.hibernate.query.internal.NativeQueryImpl.doList(NativeQueryImpl.java:147) at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1398) ... 29 more Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'metastore.CM_VERSION' doesn't exist at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.Util.getInstance(Util.java:408) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:944) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484) at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858) at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1966) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:116) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:70) Mysql database is as below: +---------------------------+ | Tables_in_metastore | +---------------------------+ | AUDITS | | BUCKETING_COLS | | CDH_VERSION | | CDS | | COLUMNS_V2 | | COMMANDS | | CONFIGS | | CREDENTIALS | | DATABASE_PARAMS | | DBS | | DB_PRIVS | | DELEGATION_TOKENS | | FUNCS | | FUNC_RU | | GLOBAL_PRIVS | | HOSTS | | IDXS | | INDEX_PARAMS | | KEY_CONSTRAINTS | | MASTER_KEYS | | NOTIFICATION_LOG | | NOTIFICATION_SEQUENCE | | NUCLEUS_TABLES | | PARTITIONS | | PARTITION_EVENTS | | PARTITION_KEYS | | PARTITION_KEY_VALS | | PARTITION_PARAMS | | PART_COL_PRIVS | | PART_COL_STATS | | PART_PRIVS | | PROCESSES | | ROLES | | ROLE_MAP | | SCHEMA_VERSION | | SDS | | SD_PARAMS | | SEQUENCE_TABLE | | SERDES | | SERDE_PARAMS | | SKEWED_COL_NAMES | | SKEWED_COL_VALUE_LOC_MAP | | SKEWED_STRING_LIST | | SKEWED_STRING_LIST_VALUES | | SKEWED_VALUES | | SORT_COLS | | TABLE_PARAMS | | TAB_COL_STATS | | TBLS | | TBL_COL_PRIVS | | TBL_PRIVS | | TYPES | | TYPE_FIELDS | | VERSION |

dennisli · ‎07-16-2019

I use this command to start: sudo systemctl start cloudera-scm-server. I 'm using CDH-6.2.0-1.cdh6.2.0. and I use mysql as metastore. Then it tells me as below: ``` 2019-07-16 15:06:38,947 ERROR main:com.cloudera.server.cmf.Main: Server failed. org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Unable to obtain CM release version. at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108) at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:634) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1198) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1100) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543) at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:418) at com.cloudera.server.cmf.Main.<init>(Main.java:260) at com.cloudera.server.cmf.Main.main(Main.java:233) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Unable to obtain CM release version. at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1640) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351) ... 17 more Caused by: java.lang.RuntimeException: Unable to obtain CM release version. at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getPersistedCmRelease(EntityManagerFactoryBean.java:311) at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.checkVersionDoFail(EntityManagerFactoryBean.java:358) at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:172) at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:75) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178) ... 22 more Caused by: javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not extract ResultSet at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:147) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:155) at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1407) at org.hibernate.query.internal.AbstractProducedQuery.getSingleResult(AbstractProducedQuery.java:1444) at com.cloudera.cmf.persist.DbVersionDao.getVersion(DbVersionDao.java:53) at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getPersistedCmRelease(EntityManagerFactoryBean.java:307) ... 26 more Caused by: org.hibernate.exception.SQLGrammarException: could not extract ResultSet at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:63) at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79) at org.hibernate.loader.Loader.getResultSet(Loader.java:2123) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1911) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1887) at org.hibernate.loader.Loader.doQuery(Loader.java:932) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349) at org.hibernate.loader.Loader.doList(Loader.java:2615) at org.hibernate.loader.Loader.doList(Loader.java:2598) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2430) at org.hibernate.loader.Loader.list(Loader.java:2425) at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:335) at org.hibernate.internal.SessionImpl.listCustomQuery(SessionImpl.java:2128) at org.hibernate.internal.AbstractSharedSessionContract.list(AbstractSharedSessionContract.java:991) at org.hibernate.query.internal.NativeQueryImpl.doList(NativeQueryImpl.java:147) at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1398) ... 29 more Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'metastore.CM_VERSION' doesn't exist at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.Util.getInstance(Util.java:408) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:944) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484) at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858) at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1966) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:116) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:70) ``` Mysql database is as below: ``` +---------------------------+ | Tables_in_metastore | +---------------------------+ | AUDITS | | BUCKETING_COLS | | CDH_VERSION | | CDS | | COLUMNS_V2 | | COMMANDS | | CONFIGS | | CREDENTIALS | | DATABASE_PARAMS | | DBS | | DB_PRIVS | | DELEGATION_TOKENS | | FUNCS | | FUNC_RU | | GLOBAL_PRIVS | | HOSTS | | IDXS | | INDEX_PARAMS | | KEY_CONSTRAINTS | | MASTER_KEYS | | NOTIFICATION_LOG | | NOTIFICATION_SEQUENCE | | NUCLEUS_TABLES | | PARTITIONS | | PARTITION_EVENTS | | PARTITION_KEYS | | PARTITION_KEY_VALS | | PARTITION_PARAMS | | PART_COL_PRIVS | | PART_COL_STATS | | PART_PRIVS | | PROCESSES | | ROLES | | ROLE_MAP | | SCHEMA_VERSION | | SDS | | SD_PARAMS | | SEQUENCE_TABLE | | SERDES | | SERDE_PARAMS | | SKEWED_COL_NAMES | | SKEWED_COL_VALUE_LOC_MAP | | SKEWED_STRING_LIST | | SKEWED_STRING_LIST_VALUES | | SKEWED_VALUES | | SORT_COLS | | TABLE_PARAMS | | TAB_COL_STATS | | TBLS | | TBL_COL_PRIVS | | TBL_PRIVS | | TYPES | | TYPE_FIELDS | | VERSION | ```

Online	Offline
Last Visited	‎05-07-2020 03:13 AM

Date Registered	‎06-26-2019 02:03 AM
Last Visited	‎05-07-2020 03:13 AM
Total Messages Posted	26
Total Kudos Received	1

Re: Cloudera manager 5.5 install : HDFS error

Re: Creating hive table to read data from kafka to...

Re: How to upgrade Hive from 2.1 to 3.0 via CDH 6....

How to upgrade Hive in CDH?

Re: How to update the IP adress of a node on CDH 6...

How to update the IP adress of a node on CDH 6.2?

Re: Table 'metastore.CM_VERSION' doesn't exist whe...

Re: Table 'metastore.CM_VERSION' doesn't exist whe...

Re: Hive Errors happend when execute hive --servic...

Re: Hive Errors happend when execute hive --servic...

Re: Hive Errors happend when execute hive --servic...

Re: Hive Errors happend when execute hive --servic...

Re: Hive Errors happend when execute hive --servic...

Re: Hive Errors happend when execute hive --servic...

Re: Hive Errors happend when execute hive --servic...

Hive Errors happend when execute hive --service me...

Re: Cloudera manager 5.5 install : HDFS error

Re: Cloudera Manager Failed to format namenode whe...

Re: Cloudera Manager Failed to format namenode whe...

Re: Cloudera Manager Failed to format namenode whe...

Re: Cloudera Manager 6.2, table metastore.CM_VERSI...

Cloudera Manager Failed to format namenode when ad...

Re: Table 'metastore.CM_VERSION' doesn't exist whe...

Re: Cloudera Manager 6.2, table metastore.CM_VERSI...

Cloudera Manager 6.2, table metastore.CM_VERSION d...

Table 'metastore.CM_VERSION' doesn't exist when st...