Member since
10-24-2013
26
Posts
2
Kudos Received
5
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 8709 | 10-21-2014 06:53 PM | |
| 3145 | 06-02-2014 01:29 PM | |
| 15784 | 05-08-2014 03:30 PM | |
| 4161 | 12-11-2013 07:53 AM | |
| 4490 | 11-23-2013 03:43 PM |
10-21-2014
06:53 PM
Hello All, I went to the mysql command line and used schema create/update - able to get the metastore working & also able to get the tables form hive as well as impala. database :mysql datastore database: hive mysql> source hive-schema-0.13.0.mysql.sql; mysql> use hive Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> source upgrade-0.10.0-to-0.11.0.mysql.sql +--------------------------------------------------+ | | +--------------------------------------------------+ | Upgrading MetaStore schema from 0.10.0 to 0.11.0 | +--------------------------------------------------+ 1 row in set, 1 warning (0.00 sec) +-----------------------------------------------------------+ | | +-----------------------------------------------------------+ | Finished upgrading MetaStore schema from 0.10.0 to 0.11.0 | +-----------------------------------------------------------+ 1 row in set, 1 warning (0.00 sec) mysql> source upgrade-0.11.0-to-0.12.0.mysql.sql +--------------------------------------------------+ | | +--------------------------------------------------+ | Upgrading MetaStore schema from 0.11.0 to 0.12.0 | +--------------------------------------------------+ 1 row in set, 1 warning (0.00 sec) +---------------------------------------------------+ | | +---------------------------------------------------+ | < HIVE-3255 Master Key and Delegation Token DDL > | +---------------------------------------------------+ 1 row in set, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) +-----------------------------------------------------------+ | | +-----------------------------------------------------------+ | < HIVE-3764 Support metastore version consistency check > | +-----------------------------------------------------------+ 1 row in set, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) ERROR 1062 (23000): Duplicate entry '1' for key 1 Query OK, 1 row affected (0.00 sec) Rows matched: 1 Changed: 1 Warnings: 0 +-----------------------------------------------------------+ | | +-----------------------------------------------------------+ | Finished upgrading MetaStore schema from 0.11.0 to 0.12.0 | +-----------------------------------------------------------+ 1 row in set, 1 warning (0.00 sec) mysql> source upgrade-0.12.0-to-0.13.0.mysql.sql +--------------------------------------------------+ | | +--------------------------------------------------+ | Upgrading MetaStore schema from 0.12.0 to 0.13.0 | +--------------------------------------------------+ 1 row in set, 1 warning (0.00 sec) +-----------------------------------------------------------------------+ | | +-----------------------------------------------------------------------+ | < HIVE-5700 enforce single date format for partition column storage > | +-----------------------------------------------------------------------+ 1 row in set, 1 warning (0.00 sec) Query OK, 0 rows affected (0.00 sec) Rows matched: 0 Changed: 0 Warnings: 0 +--------------------------------------------+ | | +--------------------------------------------+ | < HIVE-6386: Add owner filed to database > | +--------------------------------------------+ 1 row in set, 1 warning (0.00 sec) ERROR 1060 (42S21): Duplicate column name 'OWNER_NAME' ERROR 1060 (42S21): Duplicate column name 'OWNER_TYPE' +---------------------------------------------------------------------------------------------+ | | +---------------------------------------------------------------------------------------------+ | <HIVE-6458 Add schema upgrade scripts for metastore changes related to permanent functions> | +---------------------------------------------------------------------------------------------+ 1 row in set, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) Query OK, 0 rows affected, 1 warning (0.00 sec) +----------------------------------------------------------------------------------+ | | +----------------------------------------------------------------------------------+ | <HIVE-6757 Remove deprecated parquet classes from outside of org.apache package> | +----------------------------------------------------------------------------------+ 1 row in set, 1 warning (0.00 sec) Query OK, 0 rows affected (0.01 sec) Rows matched: 0 Changed: 0 Warnings: 0 Query OK, 0 rows affected (0.00 sec) Rows matched: 0 Changed: 0 Warnings: 0 Query OK, 0 rows affected (0.01 sec) Rows matched: 0 Changed: 0 Warnings: 0 ERROR 1050 (42S01): Table 'TXNS' already exists ERROR 1050 (42S01): Table 'TXN_COMPONENTS' already exists ERROR 1050 (42S01): Table 'COMPLETED_TXN_COMPONENTS' already exists ERROR 1050 (42S01): Table 'NEXT_TXN_ID' already exists Query OK, 1 row affected (0.00 sec) ERROR 1050 (42S01): Table 'HIVE_LOCKS' already exists ERROR 1061 (42000): Duplicate key name 'HL_TXNID_IDX' ERROR 1050 (42S01): Table 'NEXT_LOCK_ID' already exists Query OK, 1 row affected (0.00 sec) ERROR 1050 (42S01): Table 'COMPACTION_QUEUE' already exists ERROR 1050 (42S01): Table 'NEXT_COMPACTION_QUEUE_ID' already exists Query OK, 1 row affected (0.00 sec) Query OK, 1 row affected (0.00 sec) Rows matched: 1 Changed: 1 Warnings: 0 +-----------------------------------------------------------+ | | +-----------------------------------------------------------+ | Finished upgrading MetaStore schema from 0.12.0 to 0.13.0 | +-----------------------------------------------------------+ 1 row in set, 1 warning (0.00 sec) mysql>
... View more
10-21-2014
03:15 PM
Hello All, I upgraded the cluster from CDH-4.7 to CDH 5.2. HDFS and Mapreduce looks good but having issue with hive metastore. Following is error I see on metastore log file. I use mysql database. Please let me know how to fix this issue. I have backup of database if needed. DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. org.apache.hadoop.hive.metastore.MetaStoreDirectSql: Database initialization failed; direct SQL is disabled com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'A0.BIG_DECIMAL_HIGH_VALUE' in 'field list' 2014-10-21 17:02:49,588 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: Starting hive metastore on port 9083 2014-10-21 17:02:49,776 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 2014-10-21 17:02:49,816 INFO org.apache.hadoop.hive.metastore.ObjectStore: ObjectStore, initialize called 2014-10-21 17:02:50,083 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored 2014-10-21 17:02:50,083 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 2014-10-21 17:02:50,895 INFO org.apache.hadoop.hive.metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 2014-10-21 17:02:52,175 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table. 2014-10-21 17:02:52,176 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table. 2014-10-21 17:02:52,290 ERROR org.apache.hadoop.hive.metastore.MetaStoreDirectSql: Database initialization failed; direct SQL is disabled javax.jdo.JDOException: Exception thrown when executing query at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596) at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:230) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.<init>(MetaStoreDirectSql.java:120) at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:253) at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:224) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58) at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:506) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:484) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:532) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:406) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:365) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:55) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:60) at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4953) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:5173) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5093) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) NestedThrowablesStackTrace: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'A0.BIG_DECIMAL_HIGH_VALUE' in 'field list' at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.Util.getInstance(Util.java:386) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4120) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4052) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2503) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2664) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2794) at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155) at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2322) at com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:174) at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeQuery(ParamLoggingPreparedStatement.java:381) at org.datanucleus.store.rdbms.SQLController.executeStatementQuery(SQLController.java:504) at org.datanucleus.store.rdbms.query.JDOQLQuery.performExecute(JDOQLQuery.java:651) at org.datanucleus.store.query.Query.executeQuery(Query.java:1786) "hadoop-cmf-hive1-HIVEMETASTORE-chadvt3endc02.ops.tiaa-cref.org.log.out" 1514L, 139878C
... View more
Labels:
06-02-2014
01:29 PM
Hello All, This issue was resolved - working with Kevin Odel from Cloudera - summary as follows 1.) First time install on Amazon AWS - install done by using package 2.) Configured Kerberos 3.) All the services started good and nice 4.) We can move files in and out of hdfs - no issues 5.) YARN manager status will be good on Cloudera Manager 6.) When you run example or pig scripts it will stuck at 0% with status of scheduled to submitted Troubleshooting: 1.) go to YARN Resource Manager Web UI - click on Nodes - you will not see any nodes listed (basically no node manager on the cluster) - above is the reason you jobs stuck with 0% 2.) Go to each Node Manager look at /var/log/hadoop-yarn/* You see the below error 2014-05-28 12:36:12,911 INFO org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Disk(s) failed. 1/1 local-dirs turned bad: /dfs/dn/yarn/nm; 2014-05-28 12:36:12,911 ERROR org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Most of the disks failed. 1/1 local-dirs turned bad: /dfs/dn/yarn/nm; 2014-05-28 16:29:50,916 INFO org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Disk(s) failed. 1/1 local-dirs turned bad: /dfs/dn/yarn/nm; 2014-05-28 16:29:50,916 ERROR org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Most of the disks failed. 1/1 local-dirs turned bad: /dfs/dn/yarn/nm; 2014-05-29 13:09:54,340 INFO org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Disk(s) failed. 1/1 local-dirs turned bad: /dfs/dn/yarn/nm; 2014-05-29 13:09:54,340 ERROR org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Most of the disks failed. 1/1 local-dirs turned bad: /dfs/dn/yarn/nm; 2014-06-02 13:43:10,736 INFO org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Disk(s) failed. 1/1 local-dirs turned bad: /dfs/dn/yarn/nm; 2014-06-02 13:43:10,736 ERROR org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Most of the disks failed. 1/1 local-dirs turned bad: /dfs/dn/yarn/nm; 2014-06-02 13:58:15,344 INFO org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Disk(s) failed. 1/1 local-dirs turned bad: /dfs/dn/yarn/nm; 2014-06-02 13:58:15,344 ERROR org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Most of the disks failed. 1/1 local-dirs turned bad: /dfs/dn/yarn/nm; To Fix it: go to CM - YARN - make modification to yarn.local.dir (make sure all the nodes on same group or do for each group of nodes) stop YARN cluster deploy the configuration (with new yarn.local.dir) start YARN Cluster now - go back to YARN Resource Manager WEB UI -you should see all the nodes - if node is missing - it means it was not able to create a directory for local - go to node which is not visiable on Resource manager and check at the /var/log/hadoop-yarn/* logs file - fix manual or from CM - redeploy configuration from CM - start YARN services once you make sure all the nodes visiable on Resource Manager - Nodes on Yarn You can submit your jobs now - it will complete. Thanks, Ram
... View more
06-02-2014
07:27 AM
Hello All, I installed the CDH 5.0.1 on Amazon AWS using four node cluster (one name node and three data node) and all looks great from cloudera manager console(i am also using kerberos within AWS). But when I I run the below test as well as pig script they all getting stuck and jobs not getting started. I can move the files in and out of hdfs without any problem. I may need to open up the ports but want to know the port numbers to make this working - pls help. hadoop fs -mkdir /user/hdfs/input/ hadoop fs -put /etc/hadoop/conf/*.xml input hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar grep input output23 'dfs[a-z.]+' 14/06/02 10:17:22 INFO client.RMProxy: Connecting to ResourceManager at awsdve1ahdpnm1.ops.tiaa-cref.org/10.22.10.113:8032 14/06/02 10:17:23 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 20 for hdfs on 10.22.10.113:8020 14/06/02 10:17:23 INFO security.TokenCache: Got dt for hdfs://awsdve1ahdpnm1.ops.tiaa-cref.org:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.22.10.113:8020, Ident: (HDFS_DELEGATION_TOKEN token 20 for hdfs) 14/06/02 10:17:23 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 14/06/02 10:17:23 INFO input.FileInputFormat: Total input paths to process : 4 14/06/02 10:17:23 INFO mapreduce.JobSubmitter: number of splits:4 14/06/02 10:17:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1401383446374_0007 14/06/02 10:17:24 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: 10.22.10.113:8020, Ident: (HDFS_DELEGATION_TOKEN token 20 for hdfs) 14/06/02 10:17:24 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources. 14/06/02 10:17:24 INFO impl.YarnClientImpl: Submitted application application_1401383446374_0007 14/06/02 10:17:24 INFO mapreduce.Job: The url to track the job: http://awsdve1ahdpnm1.ops.tiaa-cref.org:8088/proxy/application_1401383446374_0007/ 14/06/02 10:17:24 INFO mapreduce.Job: Running job: job_1401383446374_0007
... View more
05-08-2014
03:30 PM
1 Kudo
Hello All, It was because of port conflict with other services running the server (impala daemon backend port be_port) 22000 has conflict on new services installed on linux server. The strange think is CM shows all "GREEN" and all the process has been started but when you look at the log file it will show the following error - + exec /usr/lib/impala/../../bin/impalad --flagfile=/var/run/cloudera-scm-agent/process/2050-impala-IMPALAD/impala-conf/impalad_flags E0508 18:20:58.253557 16729 thrift-server.cc:157] ThriftServer 'backend' (on port: 22000) exited due to TException: Could not bind: Transport endpoint is not connected E0508 18:20:58.254359 16152 thrift-server.cc:146] ThriftServer 'backend' (on port: 22000) did not start correctly please make life easier 🙂 Thanksm Ram
... View more
05-08-2014
12:15 PM
Select 1 works on all the cluster nodes - pls see below [CHAS2T3ENDC03.ops.tiaa-cref.org:21000] > select 1 ; Query: select 1 +---+ | 1 | +---+ | 1 | +---+ Returned 1 row(s) in 0.14s [CHAS2T3ENDC03.ops.tiaa-cref.org:21000] > select 1 from vpnagg; Query: select 1 from vpnagg ERROR: ExecPlanRequest rpc query_id=2e432968a16d64b0:b682c60a0df3a799 instance_id=2e432968a16d64b0:b682c60a0df3a79b failed: ECONNRESET [CHAS2T3ENDC03.ops.tiaa-cref.org:21000] >
... View more
05-08-2014
11:56 AM
Hi Lenni, I get the same error when I tried to execute the "select 1 from table_name" from impala-shell, however same select statement works from hive on all the nodes. Error: ERROR: ExecPlanRequest rpc query_id=7c4cc62c0d4c147f:e23ddf1caf162aba instance_id=7c4cc62c0d4c147f:e23ddf1caf162abc failed: ECONNRESET
... View more
05-07-2014
03:02 PM
I get the below error on impala 1.2.3 version. I did the recycle of all the impalad on cluster but still I get the below error when use “select * tablename” but same select statement works from hive. ERROR: ExecPlanRequest rpc query_id=8641141086f3ab8e:8ff3926e0f221e91 instance_id=8641141086f3ab8e:8ff3926e0f221e95 failed: ECONNRESET I have not seen this error before but I ran out of disk space on couple of nodes and I think it caused this issue. Thanks, Ram
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Impala
12-11-2013
07:53 AM
Hello All, I worked with cloudera support to solve this issue. The issue is basically krb5cache file location not at c:\ProgramData\MIT\Kerberos5\ So you need to add the windows system variable KRB5CCNAME and specify value as C:\ProgramData\MIT\Kerberos5\krb5cache than - regenrate the ticket close the ODBC 32 administrator open ODBC 32 bit adaministrator test it now it will work I have documented the full procedure needed to connect tableau 8.0.4 or higher to secure hive. So if you need it ..shoot me message. Thanks, Ram
... View more
12-03-2013
09:43 AM
sorry , I was not able to cut and past the images earlier.
... View more