About pankaj_pankasin

pankaj_pankasin · ‎01-17-2019

I am running zeppelin in standalone mode and running on windows machine. I am configuring AD authentication in shiro.ini file and want to see debug level logs on console. I made following changes in log4j.properties but it seems its not reading log4j.properties file... Does anyone know how to enable debug level logging on console..

pankaj_pankasin · ‎01-10-2019

I would like to understand how a large level cluster is setup.. consisting of 50-100 or even close to 200 nodes. I have worked on small level cluster setup by using ambari.. What I would like to understand for setting 100 nodes cluster is the process same (i.e. ambari) or hadoop admin uses some different tools to setup cluster.

pankaj_pankasin · ‎09-04-2018

what do u mean by running same environment? you mean ranger and ranger kms? database is running in google cloud?

pankaj_pankasin · ‎08-30-2018

I am trying to install accumulo from installation wizard. I have attached the screen shot of the wizard.. instance secret password text box field is disabled not sure why. Does any one have an idea why is it disabled? (Note: Apache ranger installation is failed.. and installation is being carried out in google cloud by HDP 2.6.5 Cloudbreak...) screen-shot-2018-08-30-at-21326-pm.pngscreen-shot-2018-08-30-at-21210-pm.png

pankaj_pankasin · ‎08-30-2018

I am trying to install HDP 2.6.5 through cloud break in google cloud. I am able to install all the services in google cloud but ranger. Error stack trace is listed below. I tried to follow the posted solutions earlier but none of them is helping including java patch solution by deleting an entry from the table. Can someone help me to resolve this issue? 2018-08-30 05:47:57,312 [I] --------- Verifying Ranger DB connection --------- 2018-08-30 05:47:57,312 [I] Checking connection 2018-08-30 05:47:57,312 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/postgresql-9.1-901-1.jdbc4.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://xx.xxx.xxx.xxx/ranger -u postgres -p '********' -noheader -trim -c \; -query "SELECT 1;" 2018-08-30 05:47:59,027 [I] connection success 2018-08-30 05:47:59,027 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/postgresql-9.1-901-1.jdbc4.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://xx.xxx.xxx.xxx/ranger -u postgres -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and inst_by = 'Ranger 0.7.0.2.6.5.0-292' and active = 'Y';" 2018-08-30 05:48:00,758 [I] ----------------- Applying java patches ------------ 2018-08-30 05:48:00,759 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/postgresql-9.1-901-1.jdbc4.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://xx.xxx.xxx.xxx/ranger -u postgres -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'J10001' and active = 'Y';" 2018-08-30 05:48:02,430 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/postgresql-9.1-901-1.jdbc4.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://xx.xxx.xxx.xxx/ranger -u postgres -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'J10001' and active = 'N';" 2018-08-30 05:48:04,113 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/postgresql-9.1-901-1.jdbc4.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://xx.xxx.xxx.xxx/ranger -u postgres -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('J10001', current_timestamp, 'Ranger 0.7.0.2.6.5.0-292', current_timestamp, 'hdpcluster-m-1-20180830031406.c.striped-orbit-213714.internal','N') ;" 2018-08-30 05:48:05,791 [I] java patch PatchPasswordEncryption_J10001 is being applied.

pankaj_pankasin · ‎08-28-2018

I looked at this document. i guess it's for on-premise servers. What about cloud? will cloud break support it. Latest cloud break supports HDP 2.6 and HDF 3.1.

pankaj_pankasin · ‎08-28-2018

I am running services in google cloud through hortonworks cloudbreak. Whats the proper way to restart the services. I could see stop option in cloud break but don't see any restart/start option

pankaj_pankasin · ‎08-23-2018

Is their anyway to deploy HDP & HDF together in cloud?

pankaj_pankasin · ‎08-21-2018

We intent to build datalake in cloud. Can some one help me to understand how the data is backed up in cloud for Hive, Hbase, Hadoop (entire stack)?.. In a way how to make sure nodes are never gets corrupted all the time? Looking for some best practices for building datalake in cloud. For ex: For feed or any file drop we can create a S3 bucket in AWS.. Does similar principles also applied for Hadoop, HBase, Hive (I guess no..) as they are distributed and data will be replicated.. (+) How to deploy the Hadoop instances in multi region (i.e. if entire data center goes down)..

pankaj_pankasin · ‎08-19-2018

Thanks for your answer. Our dataset is not huge may be 10-15 TB of data at the moment. or over the period of 2-3 yrs grow to 15-20 TB at max. We intent to build this datalake in cloud but not using cloud provider platform as they don't provide data gov and lineage tracking.. our data set is mostly going to be unstructured..

pankaj_pankasin · ‎08-19-2018

I assume you are referring to multi-data lake in your organization? or you are referring to processing the same feed in same datalake based on different kind of requirement? Why not storing the data in same datalake? as you mentioned we got multiple production environments..

pankaj_pankasin · ‎08-18-2018

I would like to know how to upgrade the HDP cluster into cloud deployed through cloud break? Additionally, is it possible to install HDF AND HDP together into cloud if yes how?

pankaj_pankasin · ‎08-18-2018

have any one build a datalake in organisations? Looking for some best practices around it.. Should we have one single datalake or multi data lake within an organisations.. Looking for some guidance around it.. appreciate if some one can answer or point in right direction. I have googled out many links but unable to find one answers so looking for some real live examples..

pankaj_pankasin · ‎08-14-2018

Currently i am facing an issue while upgrading hortonworks stack (from 2.6.5 to 3.0.0). I get an error HDFS cluster is not finalised yet. screen-shot-2018-08-14-at-121041-pm.png.. HDP upgrade process is stuck as well. Dismiss button is not functional too. Can some one recommend what to look for?

pankaj_pankasin · ‎08-14-2018

screen-shot-2018-08-14-at-120123-pm.pngscreen-shot-2018-08-14-at-120111-pm.pngscreen-shot-2018-08-14-at-120055-pm.png PFA screenshot of process.

pankaj_pankasin · ‎08-14-2018

I am upgrading HDP stack and currently upgrade is stuck. Dismiss button is not work too. There was an error in installing the hive, it was fixed and hive was restarted from command line. As the process is stuck.. resuming option also not working. what to do next can some1 recommend me..?

pankaj_pankasin · ‎08-14-2018

I restarted hive metastore by running tool, post command line start (https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-installation/content/validate_installation.html).. but ambari admin ui is still stuck.. is their a way we can restart upgrade or rollback or do something.. thr is no button to restart upgrade even..screen-shot-2018-08-14-at-115110-am.png

pankaj_pankasin · ‎08-14-2018

screen-shot-2018-08-14-at-115110-am.png I restarted hive metastore by running tool, post command line start (https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-installation/content/validate_installation.html).. but ambari admin ui is still stuck.. is their a way we can restart upgrade or rollback or do something.. thr is no button to restart upgrade even..

pankaj_pankasin · ‎08-14-2018

I see a similar thread being discussed on MAPR.. https://community.mapr.com/community/exchange/blog/2017/06/19/handling-hive-metastore-db-table-version-inconsistency-when-hive-client-uses-embedded-metastore

pankaj_pankasin · ‎08-14-2018

Stack trace from hive meta store log: Is there a way to fix it or resolve it? p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff; min-height: 13.0px} span.s1 {font-variant-ligatures: no-common-ligatures} span.Apple-tab-span {white-space:pre} 2018-08-14T10:26:53,488 INFO[main]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(186)) - Using direct SQL, underlying DB is MYSQL 2018-08-14T10:26:53,490 INFO[main]: metastore.ObjectStore (ObjectStore.java:setConf(396)) - Initialized ObjectStore 2018-08-14T10:26:55,362 ERROR [main]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(201)) - MetaException(message:Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt) at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:9063) at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:9027) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) at com.sun.proxy.$Proxy32.verifySchema(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:698) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:691) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:768) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:539) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8799) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8794) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:9064) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8981) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) 2018-08-14T10:26:55,364 ERROR [main]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:<init>(83)) - HMSHandler Fatal error: MetaException(message:Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt) at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:9063) at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:9027) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) at com.sun.proxy.$Proxy32.verifySchema(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:698) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:691) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:768) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:539) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8799) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8794) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:9064) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8981) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) 2018-08-14T10:26:55,364 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(9183)) - MetaException(message:Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8799) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8794) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:9064) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8981) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) Caused by: MetaException(message:Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt) at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:9063) at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:9027) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) at com.sun.proxy.$Proxy32.verifySchema(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:698) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:691) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:768) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:539) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80) ... 11 more 2018-08-14T10:26:55,365 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(8986)) - Metastore Thrift Server threw an exception... org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8799) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8794) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:9064) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8981) [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.0.3.0.0.0-1634.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.0.3.0.0.0-1634.jar:?] Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:9063) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:9027) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at com.sun.proxy.$Proxy32.verifySchema(Unknown Source) ~[?:?] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:698) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:691) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:768) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:539) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] ... 11 more 2018-08-14T10:26:55,379 INFO[pool-2-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:lambda$main$0(8951)) - Shutting down hive metastore. 2018-08-14T10:26:55,393 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(210)) - Stopping hivemetastore metrics system... 2018-08-14T10:26:55,393 INFO[timeline]: impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted. 2018-08-14T10:26:55,394 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(216)) - hivemetastore metrics system stopped. 2018-08-14T10:26:55,394 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(607)) - hivemetastore metrics system shutdown complete. 2018-08-14T10:26:55,395 INFO[pool-2-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:lambda$startupShutdownMessage$1(9389)) - SHUTDOWN_MSG:

pankaj_pankasin · ‎08-14-2018

on further debugging i could see this issue in hive metastore log: p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff} span.s1 {font-variant-ligatures: no-common-ligatures} span.Apple-tab-span {white-space:pre} 018-08-14T10:26:55,365 ERROR [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(8986)) - Metastore Thrift Server threw an exception... org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8799) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8794) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:9064) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8981) [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.0.3.0.0.0-1634.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.0.3.0.0.0-1634.jar:?] Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema version 3.1.0 does not match metastore's schema version 2.1.2000 Metastore is not upgraded or corrupt at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:9063) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:9027) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at com.sun.proxy.$Proxy32.verifySchema(Unknown Source) ~[?:?] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:698) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:691) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:768) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:539) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634] ... 11 more 2018-08-14T10:26:55,379 INFO[pool-2-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:lambda$main$0(8951)) - Shutting down hive metastore. 2018-08-14T10:26:55,393 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(210)) - Stopping hivemetastore metrics system... 2018-08-14T10:26:55,393 INFO[timeline]: impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted. 2018-08-14T10:26:55,394 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(216)) - hivemetastore metrics system stopped. 2018-08-14T10:26:55,394 INFO[pool-2-thread-1]: impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(607)) - hivemetastore metrics system shutdown complete. 2018-08-14T10:26:55,395 INFO[pool-2-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:lambda$startupShutdownMessage$1(9389)) - SHUTDOWN_MSG:

pankaj_pankasin · ‎08-14-2018

screen-shot-2018-08-14-at-101741-am.png uploading few more pictures for guidance how to troubleshoot..

pankaj_pankasin · ‎08-14-2018

PFA screenshot from the admin page UI. Funny part is DISMISS button is not even working. it's just stuck. Is there something can be tried out from the back end? How and where to check logs can u guide me

pankaj_pankasin · ‎08-14-2018

PFA screenshot from the admin page UI. Funny part is DISMISS button is not even working. it's just stuck. Is there something can be tried out from the back end? How and where to check logs. can you guide me?

pankaj_pankasin · ‎08-14-2018

screen-shot-2018-08-14-at-101042-am.pngscreen-shot-2018-08-14-at-101055-am.pngscreen-shot-2018-08-14-at-101107-am.png PFA screenshot from the admin page UI. Funny part is DISMISS button is not even working. it's just stuck. Is there something can be tried out from the back end?

pankaj_pankasin · ‎08-13-2018

Urgently looking for help. We tried upgrading hortonworks HDP from 2.6.5 to 3.0.0 and it's tuck since last 4-5 days. I am new to hortonworks working since last 7-8 months.. Can someone recommend how to fix it up and where to look for.PFA for reference.. ..screen-shot-2018-08-13-at-121206-pm.png

pankaj_pankasin · ‎08-02-2018

Is it possible to configure Confluent Kafka with HDP platform? As HDP platform only supports version 0.10.X wondering if they have plan to support upgraded version of KAFKA to use Kafka connect and KAFKA stream API. i would like to use Kafka connect and Kafka stream API for our project at the same time want to leverage HDP platform offerings.. I noticed kafka connect scripts in kafka shipped with HDP platform but they left the port default to 9092 while kafka port was changed to 6067... though the port could be changed but wondering if they support all feature of kafka-connect. For now our concern is only lineage tracking of confluent kafka..

pankaj_pankasin · ‎07-02-2018

I apology for not putting the answer earlier. I was able to resolve the issue. Issue was due to ranger policy setup on Kafka topics. After turning off the ranger policy it works out of the box. Kafka cluster creates a meta.properties folder. HDP infra setups the Kafka in cluster from start

pankaj_pankasin · ‎07-01-2018

Thanks!. After troubleshooting i realized it was an issue due to setup of kafka ranger plugin.. ATLAS and RANGER were unable to communicate.

pankaj_pankasin · ‎06-30-2018

Can some one point me in right direction of how to setup KAFKA cluster on HDP platform. To setup kafka clusters usually we add broker.id=1,2,3... but hortonworks doesn't provide this option to add in thr.. Issue: is when i try to create kafka topics i am able to create with 2,3 partitions and replicas 2,3.. but starting consumer and producer start throwing exception unknown topic or partition. My guess is it;s due to kafka HA configuration..

Online	Offline
Last Visited	‎01-17-2019 09:53 AM

Date Registered	‎05-15-2018 03:02 PM
Last Visited	‎01-17-2019 09:53 AM
Total Messages Posted	42

Re: unable to import hive mertadata in atlas

Apache zeppelin standalone on windows, how to ena...

Hortonworks how to setup 50-100 nodes of cluster

Re: HDP 2.6.5 fresh installation stuck. ranger pro...

HDP 2.6.5 accumulo installation wizard.. instance ...

HDP 2.6.5 fresh installation stuck. ranger process...

Re: Cloud break Hortonworks deployment into cloud....

Restarting the hortonworks cluster in cloud

Re: Cloud break Hortonworks deployment into cloud....

How to backup the data for hortonworks instances (...

Re: Hortonworks Hadoop platform vs datalake (looki...

Re: Hortonworks Hadoop platform vs datalake (looki...

Cloud break Hortonworks deployment into cloud... (...

Hortonworks Hadoop platform vs datalake (looking f...

HDFS Upgrade finalized state. HDFS cluster is not ...

Re: hortonworks upgrade process stucks. how to can...

hortonworks upgrade process stucks. how to cancel ...

Re: hortonworks HDP 3.0 update stuck in progress f...

Re: hortonworks HDP 3.0 update stuck in progress f...

Re: hortonworks upgrade failed hive metastore sche...

hortonworks upgrade failed hive metastore schema d...

Re: hortonworks HDP 3.0 update stuck in progress f...

Re: hortonworks HDP 3.0 update stuck in progress f...

Re: hortonworks HDP 3.0 update stuck in progress f...

Re: hortonworks HDP 3.0 update stuck in progress f...

Re: hortonworks HDP 3.0 update stuck in progress f...

hortonworks HDP 3.0 update stuck in progress for m...

HDP platform Apache atlas configuration with Confl...

Re: How to setup kafka cluster on hortonworks plat...

Re: Need help in understanding the PII tag in ATLA...

How to setup kafka cluster on hortonworks platform