Member since
05-15-2018
42
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
880 | 06-24-2018 01:34 AM |
09-04-2018
07:21 AM
what do u mean by running same environment? you mean ranger and ranger kms? database is running in google cloud?
... View more
08-30-2018
05:59 AM
I am trying to install HDP 2.6.5 through cloud break in google cloud. I am able to install all the services in google cloud but ranger. Error stack trace is listed below. I tried to follow the posted solutions earlier but none of them is helping including java patch solution by deleting an entry from the table. Can someone help me to resolve this issue? 2018-08-30 05:47:57,312 [I] --------- Verifying Ranger DB connection ---------
2018-08-30 05:47:57,312 [I] Checking connection
2018-08-30 05:47:57,312 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/postgresql-9.1-901-1.jdbc4.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://xx.xxx.xxx.xxx/ranger -u postgres -p '********' -noheader -trim -c \; -query "SELECT 1;"
2018-08-30 05:47:59,027 [I] connection success
2018-08-30 05:47:59,027 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/postgresql-9.1-901-1.jdbc4.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://xx.xxx.xxx.xxx/ranger -u postgres -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and inst_by = 'Ranger 0.7.0.2.6.5.0-292' and active = 'Y';"
2018-08-30 05:48:00,758 [I] ----------------- Applying java patches ------------
2018-08-30 05:48:00,759 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/postgresql-9.1-901-1.jdbc4.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://xx.xxx.xxx.xxx/ranger -u postgres -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'J10001' and active = 'Y';"
2018-08-30 05:48:02,430 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/postgresql-9.1-901-1.jdbc4.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://xx.xxx.xxx.xxx/ranger -u postgres -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'J10001' and active = 'N';"
2018-08-30 05:48:04,113 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/postgresql-9.1-901-1.jdbc4.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://xx.xxx.xxx.xxx/ranger -u postgres -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('J10001', current_timestamp, 'Ranger 0.7.0.2.6.5.0-292', current_timestamp, 'hdpcluster-m-1-20180830031406.c.striped-orbit-213714.internal','N') ;"
2018-08-30 05:48:05,791 [I] java patch PatchPasswordEncryption_J10001 is being applied.
... View more
Labels:
- Labels:
-
Apache Ranger
-
Hortonworks Cloudbreak
08-28-2018
08:34 AM
I looked at this document. i guess it's for on-premise servers. What about cloud? will cloud break support it. Latest cloud break supports HDP 2.6 and HDF 3.1.
... View more
08-28-2018
07:54 AM
I am running services in google cloud through hortonworks cloudbreak. Whats the proper way to restart the services. I could see stop option in cloud break but don't see any restart/start option
... View more
Labels:
- Labels:
-
Hortonworks Cloudbreak
08-23-2018
09:39 AM
Is their anyway to deploy HDP & HDF together in cloud?
... View more
08-21-2018
02:10 PM
We intent to build datalake in cloud. Can some one help me to understand how the data is backed up in cloud for Hive, Hbase, Hadoop (entire stack)?.. In a way how to make sure nodes are never gets corrupted all the time? Looking for some best practices for building datalake in cloud. For ex: For feed or any file drop we can create a S3 bucket in AWS.. Does similar principles also applied for Hadoop, HBase, Hive (I guess no..) as they are distributed and data will be replicated.. (+) How to deploy the Hadoop instances in multi region (i.e. if entire data center goes down)..
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Hortonworks Cloudbreak
08-19-2018
01:51 AM
Thanks for your answer. Our dataset is not huge may be 10-15 TB of data at the moment. or over the period of 2-3 yrs grow to 15-20 TB at max. We intent to build this datalake in cloud but not using cloud provider platform as they don't provide data gov and lineage tracking.. our data set is mostly going to be unstructured..
... View more
08-19-2018
01:47 AM
I assume you are referring to multi-data lake in your organization? or you are referring to processing the same feed in same datalake based on different kind of requirement? Why not storing the data in same datalake? as you mentioned we got multiple production environments..
... View more
08-18-2018
01:51 PM
I would like to know how to upgrade the HDP cluster into cloud deployed through cloud break? Additionally, is it possible to install HDF AND HDP together into cloud if yes how?
... View more
Labels:
08-18-2018
01:48 PM
have any one build a datalake in organisations? Looking for some best practices around it.. Should we have one single datalake or multi data lake within an organisations.. Looking for some guidance around it.. appreciate if some one can answer or point in right direction. I have googled out many links but unable to find one answers so looking for some real live examples..
... View more
Labels:
- Labels:
-
Apache Hadoop
08-14-2018
04:13 AM
Currently i am facing an issue while upgrading hortonworks stack (from 2.6.5 to 3.0.0). I get an error HDFS cluster is not finalised yet. screen-shot-2018-08-14-at-121041-pm.png.. HDP upgrade process is stuck as well. Dismiss button is not functional too. Can some one recommend what to look for?
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Cloudera DataFlow (CDF)
08-14-2018
04:01 AM
screen-shot-2018-08-14-at-120123-pm.pngscreen-shot-2018-08-14-at-120111-pm.pngscreen-shot-2018-08-14-at-120055-pm.png PFA screenshot of process.
... View more
08-14-2018
03:59 AM
I am upgrading HDP stack and currently upgrade is stuck. Dismiss button is not work too. There was an error in installing the hive, it was fixed and hive was restarted from command line. As the process is stuck.. resuming option also not working. what to do next can some1 recommend me..?
... View more
08-14-2018
03:53 AM
I restarted hive metastore by running tool, post command line start (https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-installation/content/validate_installation.html).. but ambari admin ui is still stuck.. is their a way we can restart upgrade or rollback or do something.. thr is no button to restart upgrade even..screen-shot-2018-08-14-at-115110-am.png
... View more
08-14-2018
03:52 AM
screen-shot-2018-08-14-at-115110-am.png I restarted hive metastore by running tool, post command line start (https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-installation/content/validate_installation.html).. but ambari admin ui is still stuck.. is their a way we can restart upgrade or rollback or do something.. thr is no button to restart upgrade even..
... View more
08-14-2018
02:55 AM
I see a similar thread being discussed on MAPR.. https://community.mapr.com/community/exchange/blog/2017/06/19/handling-hive-metastore-db-table-version-inconsistency-when-hive-client-uses-embedded-metastore
... View more
08-14-2018
02:33 AM
Stack trace from hive meta store log: Is there a way to fix it or resolve it? p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px}
span.s1 {font-variant-ligatures: no-common-ligatures}
span.Apple-tab-span {white-space:pre} 2018-08-14T10:26:53,488 INFO[main]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(186)) - Using direct SQL, underlying DB is MYSQL 2018-08-14T10:26:53,490 INFO[main]: metastore.ObjectStore (ObjectStore.java:setConf(396)) - Initialized ObjectStore 2018-08-14T10:26:55,362 ERROR [main]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(201)) - MetaException(message:Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt) at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:9063) at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:9027) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) at com.sun.proxy.$Proxy32.verifySchema(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:698) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:691) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:768) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:539) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8799) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8794) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:9064) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8981) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) 2018-08-14T10:26:55,364 ERROR [main]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:<init>(83)) - HMSHandler Fatal error: MetaException(message:Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt) at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:9063) at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:9027) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) at com.sun.proxy.$Proxy32.verifySchema(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:698) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:691) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:768) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:539) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8799) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8794) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:9064) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8981) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) 2018-08-14T10:26:55,364 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(9183)) - MetaException(message:Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8799) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8794) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:9064) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8981) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) Caused by: MetaException(message:Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt) at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:9063) at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:9027) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) at com.sun.proxy.$Proxy32.verifySchema(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:698) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:691) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:768) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:539) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80) ... 11 more 2018-08-14T10:26:55,365 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(8986)) - Metastore Thrift Server threw an exception... org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8799) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8794) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:9064) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8981) [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.0.3.0.0.0-1634.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.0.3.0.0.0-1634.jar:?] Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:9063) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:9027) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at com.sun.proxy.$Proxy32.verifySchema(Unknown Source) ~[?:?] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:698) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:691) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:768) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:539) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] ... 11 more 2018-08-14T10:26:55,379 INFO[pool-2-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:lambda$main$0(8951)) - Shutting down hive metastore. 2018-08-14T10:26:55,393 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(210)) - Stopping hivemetastore metrics system... 2018-08-14T10:26:55,393 INFO[timeline]: impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted. 2018-08-14T10:26:55,394 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(216)) - hivemetastore metrics system stopped. 2018-08-14T10:26:55,394 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(607)) - hivemetastore metrics system shutdown complete. 2018-08-14T10:26:55,395 INFO[pool-2-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:lambda$startupShutdownMessage$1(9389)) - SHUTDOWN_MSG:
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
08-14-2018
02:28 AM
on further debugging i could see this issue in hive metastore log: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
span.s1 {font-variant-ligatures: no-common-ligatures}
span.Apple-tab-span {white-space:pre} 018-08-14T10:26:55,365 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(8986)) - Metastore Thrift Server threw an exception... org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8799) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8794) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:9064) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8981) [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.0.3.0.0.0-1634.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.0.3.0.0.0-1634.jar:?] Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:9063) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:9027) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at com.sun.proxy.$Proxy32.verifySchema(Unknown Source) ~[?:?] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:698) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:691) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:768) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:539) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] ... 11 more 2018-08-14T10:26:55,379 INFO[pool-2-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:lambda$main$0(8951)) - Shutting down hive metastore. 2018-08-14T10:26:55,393 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(210)) - Stopping hivemetastore metrics system... 2018-08-14T10:26:55,393 INFO[timeline]: impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted. 2018-08-14T10:26:55,394 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(216)) - hivemetastore metrics system stopped. 2018-08-14T10:26:55,394 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(607)) - hivemetastore metrics system shutdown complete. 2018-08-14T10:26:55,395 INFO[pool-2-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:lambda$startupShutdownMessage$1(9389)) - SHUTDOWN_MSG:
... View more
08-14-2018
02:17 AM
screen-shot-2018-08-14-at-101741-am.png uploading few more pictures for guidance how to troubleshoot..
... View more
08-14-2018
02:14 AM
PFA screenshot from the admin page UI. Funny part is DISMISS button is not even working. it's just stuck. Is there something can be tried out from the back end? How and where to check logs can u guide me
... View more
08-14-2018
02:13 AM
PFA screenshot from the admin page UI. Funny part is DISMISS button is not even working. it's just stuck. Is there something can be tried out from the back end? How and where to check logs. can you guide me?
... View more
08-14-2018
02:13 AM
screen-shot-2018-08-14-at-101042-am.pngscreen-shot-2018-08-14-at-101055-am.pngscreen-shot-2018-08-14-at-101107-am.png PFA screenshot from the admin page UI. Funny part is DISMISS button is not even working. it's just stuck. Is there something can be tried out from the back end?
... View more
08-13-2018
04:17 AM
Urgently looking for help. We tried upgrading hortonworks HDP from 2.6.5 to 3.0.0 and it's tuck since last 4-5 days. I am new to hortonworks working since last 7-8 months.. Can someone recommend how to fix it up and where to look for.PFA for reference.. ..screen-shot-2018-08-13-at-121206-pm.png
... View more
Labels:
08-02-2018
01:38 PM
Is it possible to configure Confluent Kafka with HDP platform? As HDP platform only supports version 0.10.X wondering if they have plan to support upgraded version of KAFKA to use Kafka connect and KAFKA stream API. i would like to use Kafka connect and Kafka stream API for our project at the same time want to leverage HDP platform offerings.. I noticed kafka connect scripts in kafka shipped with HDP platform but they left the port default to 9092 while kafka port was changed to 6067... though the port could be changed but wondering if they support all feature of kafka-connect. For now our concern is only lineage tracking of confluent kafka..
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Kafka
07-02-2018
07:44 AM
I apology for not putting the answer earlier. I was able to resolve the issue. Issue was due to ranger policy setup on Kafka topics. After turning off the ranger policy it works out of the box. Kafka cluster creates a meta.properties folder. HDP infra setups the Kafka in cluster from start
... View more
07-01-2018
07:58 AM
Thanks!. After troubleshooting i realized it was an issue due to setup of kafka ranger plugin.. ATLAS and RANGER were unable to communicate.
... View more
06-30-2018
02:49 PM
Can some one point me in right direction of how to setup KAFKA cluster on HDP platform. To setup kafka clusters usually we add broker.id=1,2,3... but hortonworks doesn't provide this option to add in thr.. Issue: is when i try to create kafka topics i am able to create with 2,3 partitions and replicas 2,3.. but starting consumer and producer start throwing exception unknown topic or partition. My guess is it;s due to kafka HA configuration..
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Kafka
06-24-2018
04:11 PM
Can some one help me to understand what is tag sync and how does it work? I don't see any log being created in tag sync folder under /var/log/..tag/sync.. ATLAS tags are not showing up on RANGER UI.
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Ranger
06-24-2018
03:02 PM
For me it doesn't work at 2nd step creating a new policy because both the plocies ar contradictory to each other,
... View more
06-24-2018
03:01 PM
Can some one explain me how PII tag actually works? I am following this link to create working example over here https://hortonworks.com/tutorial/tag-based-policies-with-apache-ranger-and-apache-atlas/#step-6-create-a-ranger-policy-to-limit-access-of-hive-data My doubt is: 1. At first, user raj_ops and maria_dev are given permission to table "Figure 15: Sandbox_hive Repository’s Policies" which allow them to query the record successfully. 2. As per "Figure 26: Add New Ranger Policy" is created a new policy to exlcude PII column. I believe at this point both the policies are contradictory to each other (25, 28/26). user are allowed to select only name column. 3. Later as per PII tag further down the tag use can query the data. Can some one tell me or point me step be step working example of PII tag. or any tag based policies.
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Hive
-
Apache Ranger