nord_tramper
nord_tramper
Explorer

Re: Falcon hive mirror - Main class [org.apache.oo...

by Explorer nord_tramper in Support Questions
‎01-26-2018 08:17 AM
‎01-26-2018 08:17 AM
Found in oozie.log error[E1601], E1601: Cannot retrieve JMS connection info [JMSTopicService is not initialized. JMS notificationmay not be enabled]org.apache.oozie.servlet.XServletException: E1601: Cannot retrieve JMS connection info [JMSTopicService is not initialized. JMS notificationmay not be enabled] ... View more

Falcon hive mirror - Main class [org.apache.oozie....

by Explorer nord_tramper in Support Questions
‎01-25-2018 05:28 PM
‎01-25-2018 05:28 PM
HDP 2.5. Try to set up Falcon Hive Mirror for one database DB create on both HiveServers. Got error Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [-1] No logs in YARN. No error in /var/log/oozie/oozie.log ... View more

Re: Falcon: hive mirror in kerberized envs GSS ini...

by Explorer nord_tramper in Support Questions
‎01-16-2018 09:13 AM
‎01-16-2018 09:13 AM
Issue is constant - any try to start Falcon Hive mirror lead to this error. BTW, I add both falcon user and AD_USER_1 to rules in Ranger policy so issue is somewhere deeper ... View more

Re: Falcon: hive mirror in kerberized envs GSS ini...

by Explorer nord_tramper in Support Questions
‎01-16-2018 09:03 AM
‎01-16-2018 09:03 AM
here is the hivemetastore for another try of same job run 2018-01-16 11:42:01,657 INFO [pool-5-thread-197]: metastore.ObjectStore (ObjectStore.java:initializeHelper(377)) - ObjectStore, initialize called 2018-01-16 11:42:01,665 INFO [pool-5-thread-197]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(139)) - Using direct SQL, underlying DB is OTHER 2018-01-16 11:42:01,666 INFO [pool-5-thread-197]: metastore.ObjectStore (ObjectStore.java:setConf(291)) - Initialized ObjectStore 2018-01-16 11:42:07,596 INFO [pool-5-thread-197]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 198: get_databases: NonExistentDatabaseUsedForHealthCheck 2018-01-16 11:42:07,596 INFO [pool-5-thread-197]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=falcon/prjpjnd201lv.EXAMPLE.COM@EXAMPLE.COM ip=10.151.2.69 cmd=get_databases: NonExistentDatabaseUsedForHealthCheck 2018-01-16 11:42:07,599 INFO [pool-5-thread-197]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 198: get_delegation_token 2018-01-16 11:42:07,600 INFO [pool-5-thread-197]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=falcon/prjpjnd201lv.EXAMPLE.COM@EXAMPLE.COM ip=10.151.2.69 cmd=get_delegation_token 2018-01-16 11:42:07,600 INFO [pool-5-thread-197]: delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:createPassword(385)) - Creating password for identifier: owner=AD_USER_1, renewer=hive, realUser=falcon/prjpjnd201lv.EXAMPLE.COM@EXAMPLE.COM, issueDate=1516092127600, maxDate=1516696927600, sequenceNumber=28, masterKeyId=1064, currentKey: 1064 2018-01-16 11:42:07,616 INFO [pool-5-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 1: get_database: falcon_test 2018-01-16 11:42:07,617 INFO [pool-5-thread-1]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=AD_USER_1 ip=10.151.2.69 cmd=get_database: falcon_test 2018-01-16 11:43:41,825 INFO [pool-5-thread-197]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 198: get_databases: NonExistentDatabaseUsedForHealthCheck 2018-01-16 11:43:41,825 INFO [pool-5-thread-197]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=falcon/prjpjnd201lv.EXAMPLE.COM@EXAMPLE.COM ip=10.151.2.69 cmd=get_databases: NonExistentDatabaseUsedForHealthCheck 2018-01-16 11:43:41,829 INFO [pool-5-thread-197]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 198: get_delegation_token 2018-01-16 11:43:41,829 INFO [pool-5-thread-197]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=falcon/prjpjnd201lv.EXAMPLE.COM@EXAMPLE.COM ip=10.151.2.69 cmd=get_delegation_token 2018-01-16 11:43:41,829 INFO [pool-5-thread-197]: delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:createPassword(385)) - Creating password for identifier: owner=AD_USER_1, renewer=hive, realUser=falcon/prjpjnd201lv.EXAMPLE.COM@EXAMPLE.COM, issueDate=1516092221829, maxDate=1516697021829, sequenceNumber=29, masterKeyId=1064, currentKey: 1064 2018-01-16 11:43:41,993 WARN [pool-5-thread-23]: conf.HiveConf (HiveConf.java:initialize(3060)) - HiveConf of name hive.log.dir does not exist 2018-01-16 11:43:41,994 WARN [pool-5-thread-23]: conf.HiveConf (HiveConf.java:initialize(3060)) - HiveConf of name hive.log.file does not exist 2018-01-16 11:43:41,994 INFO [pool-5-thread-23]: metastore.HiveMetaStore (HiveMetaStore.java:newRawStoreForConf(701)) - 24: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 2018-01-16 11:43:42,013 INFO [pool-5-thread-23]: metastore.ObjectStore (ObjectStore.java:initializeHelper(377)) - ObjectStore, initialize called 2018-01-16 11:43:42,019 INFO [pool-5-thread-23]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(139)) - Using direct SQL, underlying DB is OTHER 2018-01-16 11:43:42,019 INFO [pool-5-thread-23]: metastore.ObjectStore (ObjectStore.java:setConf(291)) - Initialized ObjectStore 2018-01-16 11:43:42,023 INFO [pool-5-thread-23]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 24: get_database: falcon_test 2018-01-16 11:43:42,023 INFO [pool-5-thread-23]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=AD_USER_1 ip=10.151.2.69 cmd=get_database: falcon_test 2018-01-16 11:44:01,957 INFO [pool-5-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 1: get_table : db=dm tbl=tnl1 2018-01-16 11:44:01,957 INFO [pool-5-thread-1]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=spark2/prjp1c@EXAMPLE.COM ip=10.151.2.57 cmd=get_table : db=dm tbl=tnl1 2018-01-16 11:44:56,241 INFO [pool-5-thread-23]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 24: get_all_databases 2018-01-16 11:44:56,241 INFO [pool-5-thread-23]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=ambari-qa/prjp1c@EXAMPLE.COM ip=10.151.2.55 cmd=get_all_databases 2018-01-16 11:44:56,257 INFO [pool-5-thread-23]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 24: get_functions: db=bi_test pat=* 2018-01-16 11:44:56,257 INFO [pool-5-thread-23]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=ambari-qa/prjp1c@EXAMPLE.COM ip=10.151.2.55 cmd=get_functions: db=bi_test pat=* 2018-01-16 11:44:56,263 INFO [pool-5-thread-23]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 24: get_functions: db=dds pat=* 2018-01-16 11:44:56,264 INFO [pool-5-thread-23]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=ambari-qa/prjp1c@EXAMPLE.COM ip=10.151.2.55 cmd=get_functions: db=dds pat=* 2018-01-16 11:44:56,267 INFO [pool-5-thread-23]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 24: get_functions: db=default pat=* ... View more

Falcon: hive mirror in kerberized envs GSS initiat...

by Explorer nord_tramper in Support Questions
‎01-16-2018 08:14 AM
‎01-16-2018 08:14 AM
RHEL 7.3 HDP 2.5 Kerberos enabled Create two clusters in Falcon UI. Try to create Hive mirror. Oozie job created but failed with following error: UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient Full stack in /var/log/oozie/oozie.log 2018-01-15 18:45:22,762 WARN ActionStartXCommand:523 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[falcon-dr-hive-workflow] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@last-event] Error starting action [last-event]. ErrorType [ERROR], ErrorCode [UncheckedExecutionException], Message [UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient] org.apache.oozie.action.ActionExecutorException: UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1206) at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1377) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65) at org.apache.oozie.command.XCommand.call(XCommand.java:287) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2234) at com.google.common.cache.LocalCache.get(LocalCache.java:3965) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764) at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:291) at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:266) at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558) at org.apache.hive.hcatalog.api.HCatClientHMSImpl.initialize(HCatClientHMSImpl.java:823) at org.apache.hive.hcatalog.api.HCatClient.create(HCatClient.java:71) at org.apache.oozie.action.hadoop.HCatCredentialHelper.getHCatClient(HCatCredentialHelper.java:105) at org.apache.oozie.action.hadoop.HCatCredentialHelper.set(HCatCredentialHelper.java:57) at org.apache.oozie.action.hadoop.HCatCredentials.addtoJobConf(HCatCredentials.java:61) at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1291) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1128) ... 11 more Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1566) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:92) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:124) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:294) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:291) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) ... 23 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564) ... 33 more Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:486) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:281) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:187) at org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:406) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:92) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:124) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:294) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:291) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) at com.google.common.cache.LocalCache.get(LocalCache.java:3965) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764) at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:291) at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:266) at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558) at org.apache.hive.hcatalog.api.HCatClientHMSImpl.initialize(HCatClientHMSImpl.java:823) at org.apache.hive.hcatalog.api.HCatClient.create(HCatClient.java:71) at org.apache.oozie.action.hadoop.HCatCredentialHelper.getHCatClient(HCatCredentialHelper.java:105) at org.apache.oozie.action.hadoop.HCatCredentialHelper.set(HCatCredentialHelper.java:57) at org.apache.oozie.action.hadoop.HCatCredentials.addtoJobConf(HCatCredentials.java:61) at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1291) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1128) at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1377) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65) at org.apache.oozie.command.XCommand.call(XCommand.java:287) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:532) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:281) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:187) at org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:406) ... 38 more 2018-01-15 18:45:22,762 WARN ActionStartXCommand:523 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[falcon-dr-hive-workflow] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@last-event] Setting Action Status to [DONE] 2018-01-15 18:45:22,787 INFO ActionEndXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[falcon-dr-hive-workflow] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@last-event] ERROR is considered as FAILED for SLA 2018-01-15 18:45:22,854 INFO ActionStartXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[falcon-dr-hive-workflow] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@fail] Start action [0001108-180109162406621-oozie-oozi-W@fail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2018-01-15 18:45:22,854 INFO ActionStartXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[falcon-dr-hive-workflow] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@fail] [***0001108-180109162406621-oozie-oozi-W@fail***]Action status=DONE 2018-01-15 18:45:22,855 INFO ActionStartXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[falcon-dr-hive-workflow] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@fail] [***0001108-180109162406621-oozie-oozi-W@fail***]Action updated in DB! 2018-01-15 18:45:23,025 INFO WorkflowNotificationXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@fail] No Notification URL is defined. Therefore nothing to notify for job 0001108-180109162406621-oozie-oozi-W@fail 2018-01-15 18:45:23,026 INFO CallbackServlet:520 - SERVER[prjnnd201lv.examle.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@user-action] callback for action [0001107-180109162406621-oozie-oozi-W@user-action] 2018-01-15 18:45:23,027 INFO WorkflowNotificationXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@last-event] No Notification URL is defined. Therefore nothing to notify for job 0001108-180109162406621-oozie-oozi-W@last-event 2018-01-15 18:45:23,067 INFO ActionEndXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_falcon3] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@user-action] ERROR is considered as FAILED for SLA 2018-01-15 18:45:23,201 INFO ActionStartXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_falcon3] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@failed-post-processing] Start action [0001107-180109162406621-oozie-oozi-W@failed-post-processing] with user-retry state : userRetryCount [0], userRetryMax [3], userRetryInterval [1] 2018-01-15 18:45:23,748 WARN JobResourceUploader:64 - SERVER[prjnnd201lv.examle.com] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2018-01-15 18:45:23,752 WARN JobResourceUploader:171 - SERVER[prjnnd201lv.examle.com] No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2018-01-15 18:45:25,157 INFO JavaActionExecutor:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_falcon3] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@failed-post-processing] Trying to get job [job_1513087944025_15487], attempt [1] 2018-01-15 18:45:25,192 INFO JavaActionExecutor:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_falcon3] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@failed-post-processing] checking action, hadoop job ID [job_1513087944025_15487] status [RUNNING] 2018-01-15 18:45:25,195 INFO ActionStartXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_falcon3] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@failed-post-processing] [***0001107-180109162406621-oozie-oozi-W@failed-post-processing***]Action status=RUNNING 2018-01-15 18:45:25,195 INFO ActionStartXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_falcon3] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@failed-post-processing] [***0001107-180109162406621-oozie-oozi-W@failed-post-processing***]Action updated in DB! 2018-01-15 18:45:25,205 INFO WorkflowNotificationXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@user-action] No Notification URL is defined. Therefore nothing to notify for job 0001107-180109162406621-oozie-oozi-W@user-action 2018-01-15 18:45:25,205 INFO WorkflowNotificationXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@failed-post-processing] No Notification URL is defined. Therefore nothing to notify for job 0001107-180109162406621-oozie-oozi-W@failed-post-processing 2018-01-15 18:45:29,256 ERROR CompletedActionXCommand:517 - SERVER[prjnnd201lv.examle.com] USER[-] GROUP[-] TOKEN[] APP[-] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@user-action] XException, org.apache.oozie.command.CommandException: E0800: Action it is not running its in [ERROR] state, action [0001107-180109162406621-oozie-oozi-W@user-action] at org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:92) at org.apache.oozie.command.XCommand.call(XCommand.java:258) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ... View more

Re: Hive View + Kerberos error: User: ambari-serve...

by Explorer nord_tramper in Support Questions
‎11-22-2017 10:17 AM
‎11-22-2017 10:17 AM
@Jay Kumar SenSharma Hello, I add following line to auth_to_local RULE:[2:$1@$0](ambari-server@REALM)s/.*/ambari-server/ Still same errors Caused by: org.apache.hive.service.cli.HiveSQLException: Failed to validate proxy privilege of ambari-server for admin .... .... Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User: ambari-server is not allowed to impersonate admin ... View more

Re: Hive View + Kerberos error: User: ambari-serve...

by Explorer nord_tramper in Support Questions
‎11-21-2017 01:44 PM
‎11-21-2017 01:44 PM
@Jay Kumar SenSharma Here it is. Yep there was advised principal like ambari-serve-ClusterName@REALM, but our AD has strict rule that Principal should be in format <service-name>/<host>@<realm> so I modify it to meet requirements. So is there any thing that I can do with this Principal (assume that I can't change it to for ambari-serve-ClusterName@REALM)? klist -kte /etc/security/keytabs/ambari.server.keytab Keytab name: FILE:/etc/security/keytabs/ambari.server.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 0 01/01/1970 03:00:00 ambari-server/ClusterName@REALM (des-cbc-crc) 0 01/01/1970 03:00:00 ambari-server/ClusterName@REALM (des-cbc-md5) 0 01/01/1970 03:00:00 ambari-server/ClusterName@REALM (arcfour-hmac) 0 01/01/1970 03:00:00 ambari-server/ClusterName@REALM (aes256-cts-hmac-sha1-96) 0 01/01/1970 03:00:00 ambari-server/ClusterName@REALM (aes128-cts-hmac-sha1-96) ... View more

Re: Hive View + Kerberos error: User: ambari-serve...

by Explorer nord_tramper in Support Questions
‎11-21-2017 01:00 PM
‎11-21-2017 01:00 PM
Also in HDFS View I have error java.lang.IllegalArgumentException: Invalid value: "ambari-server/JournalNode" does not belong to the domain ^[A-Za-z_][A-Za-z0-9._-]*[$]?$ ... View more

Re: Hive View + Kerberos error: User: ambari-serve...

by Explorer nord_tramper in Support Questions
‎11-21-2017 12:55 PM
1 Kudo
‎11-21-2017 12:55 PM
1 Kudo
@Aditya Sirna Added hadoop.proxyuser.ambari-server.users=* into custom core-site section. Restart both cluster and Ambari - no effect ... View more

Re: Hive View + Kerberos error: User: ambari-serve...

by Explorer nord_tramper in Support Questions
‎11-21-2017 09:37 AM
‎11-21-2017 09:37 AM
@Jay Kumar SenSharma I have following settings # cat /etc/ambari-server/conf/krb5JAASLogin.conf com.sun.security.jgss.krb5.initiate { com.sun.security.auth.module.Krb5LoginModule required renewTGT=false doNotPrompt=true useKeyTab=true keyTab="/etc/security/keytabs/ambari.server.keytab" principal=<a href="mailto:ambari-server/JounralNode@REALM">ambari-server/JounralNode@REALM</a> storeKey=true useTicketCache=false; }; Principal name is ambari-server/JounralNode@REALM . /JounralNode has been add as a requirement from AD admins to create Principal. So I have to set up hadoop.proxyuser.ambari-server/JounralNode.groups=* hadoop.proxyuser.ambari-server/JounralNode.hosts=* But it is not possible ... View more

Hive View + Kerberos error: User: ambari-server is...

by Explorer nord_tramper in Support Questions
‎11-20-2017 04:38 PM
‎11-20-2017 04:38 PM
Hello, all! Environment: RHEL 7.3, HDP 2.6, Kerberos, ActiveDirectory Hive view error: Message: Cannot open a hive connection with connect string jdbc:hive2://nn1:2181,nn2:2181,dn1:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;hive.server2.proxy.user=admin And deep in logs: Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User: ambari-server is not allowed to impersonate admin Following settings already in place in core-site.xml hadoop.proxyuser.ambari-server.groups=* hadoop.proxyuser.ambari-server.hosts=* hadoop.proxyuser.root.groups=* hadoop.proxyuser.root.hosts=* ... View more
Labels:

Re: YARN logs + HTTP auth

by Explorer nord_tramper in Support Questions
‎10-30-2017 12:19 PM
1 Kudo
‎10-30-2017 12:19 PM
1 Kudo
Also works for me after some experiments: yarn.admin.acl=yarn,dr.who,<AD LOGIN IN UPPERCASE WITHOUT REALM> ... View more

Re: YARN logs + HTTP auth

by Explorer nord_tramper in Support Questions
‎10-30-2017 11:42 AM
‎10-30-2017 11:42 AM
Your settings solve the problem. I have default values yarn.admin.acl=yarn,dr.who yarn.acl.enable=true hadoop.http.staticuser.user = yarn ... View more

Re: I can't add new Services via Ambari UI

by Explorer nord_tramper in Support Questions
‎10-30-2017 11:13 AM
‎10-30-2017 11:13 AM
Restart ambari-agent on nodes ... View more

YARN logs + HTTP auth

by Explorer nord_tramper in Support Questions
‎10-30-2017 10:33 AM
‎10-30-2017 10:33 AM
Hello! I have HDP 2.5 cluster with KERBEROS enabled, connected to Active Directory. When I try to switch on HTTP AUTH - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_security/content/_configuring_http_authentication_for_HDFS_YARN_MapReduce2_HBase_Oozie_Falcon_and_Storm.html Logs can be retrived using shell yarn logs -applicationId application_1509115509826_0001 I can't access any logs from YARN UI for example <YARN-RM-HOST>:19888/jobhistory/logs/<NODE>:45454/container_e56_1509115509826_0001_01_000001/container_e56_1509115509826_0001_01_000001/hive With following error: User <MY Active Directory User> is not authorized to view the logs for container_e56_1509115509826_0001_01_000001 in log file [<NODE>_45454_1509118017724]No logs available for container container_e56_1509115509826_0001_01_000001 ... View more
Labels:

Re: YARN containers failed when started from Polyb...

by Explorer nord_tramper in Support Questions
‎10-30-2017 10:17 AM
‎10-30-2017 10:17 AM
Hello! Same logs I can find using web-ui - there is nothing about cause why job got faild. Containers simply Task Transitioned from RUNNING to KILL_WAIT ... View more

Re: YARN containers failed when started from Polyb...

by Explorer nord_tramper in Support Questions
‎10-12-2017 01:39 PM
‎10-12-2017 01:39 PM
Can't do kinit because this job initiated by MS SQL Polybase connector ... View more

Re: HiveServer2 action in Oozie on kerberized clus...

by Explorer nord_tramper in Support Questions
‎10-11-2017 11:42 AM
1 Kudo
‎10-11-2017 11:42 AM
1 Kudo
@Nils Sjögren thanks - your advice was helpful A couple of remarks: 1. If you have HA enabled for YARN - port for RM should be 8032 and you should point to Active RM host 2. If you have HA enabled for NameNode - use hdfs://HACLUSTERNAME instead of url for NameNode ... View more

Re: Identify active YARN RM

by Explorer nord_tramper in Support Questions
‎10-10-2017 06:36 PM
‎10-10-2017 06:36 PM
Thanks! Which keytab I should use to get access to this Zookepper Node (kerberized cluster)? Is it possible to use ZooKeeper Java API to get same info? ... View more

Identify active YARN RM

by Explorer nord_tramper in Support Questions
‎10-10-2017 01:21 PM
3 Kudos
‎10-10-2017 01:21 PM
3 Kudos
Is it possible to identify active YARN ResourceManager using ZooKeeper or any other method? ... View more
Labels:

Re: YARN containers failed when started from Polyb...

by Explorer nord_tramper in Support Questions
‎10-06-2017 08:16 AM
‎10-06-2017 08:16 AM
Same logs I found in YARN RM console One more suspicious line - before start container 2017-10-05 18:20:26,264 WARN [main] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] ... View more

YARN containers failed when started from Polybase

by Explorer nord_tramper in Support Questions
‎10-05-2017 02:50 PM
‎10-05-2017 02:50 PM
Environment: HDP 2.6 with Kerberson enabled OS RedHat 7.3 MS SQL Server 2017 with Polybase Polybase works fine when during direct loading from HDFS. However if I use FORCE EXTERNALPUSHDOWN option (it means that Polybase start MR job on the cluster) - all containers fail with following error 2017-10-05 16:29:25,066 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e50_1507209892753_0004_01_0000292017-10-05 16:29:25,067 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e50_1507209892753_0004_01_0000162017-10-05 16:29:25,068 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:27 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:29 ContRel:0 HostLocal:25 RackLocal:42017-10-05 16:29:25,075 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1507209892753_0004_m_000016_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP2017-10-05 16:29:25,075 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1507209892753_0004_m_000016_0: Exception from container-launch.Container id: container_e50_1507209892753_0004_01_000029Exit code: 1Stack trace: org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime.launchContainer(DefaultLinuxContainerRuntime.java:109) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:89) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:392) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Where to find root cause? ... View more
Labels:

HiveServer2 action in Oozie on kerberized cluster ...

by Explorer nord_tramper in Support Questions
‎09-13-2017 10:17 AM
1 Kudo
‎09-13-2017 10:17 AM
1 Kudo
HDP 2.6. Kerberos enabled. Try run HiveServer2 job using oozie like in this article - https://community.hortonworks.com/articles/75107/oozie-hive-2-action-in-a-kerberized-cluster.html job.properties nameNode=hdfs://NN2:8020 jobTracker=NN1:8050 exampleDir=${nameNode}/user/${user.name}/ooziedemo/hivedem ooozie.wf.application.path=${exampleDir}/app oozie.use.system.libpath=true # Hive2 action hivescript=${oozie.wf.application.path}/add_row.sql outputHiveDatabase=dm jdbcURL=jdbc:hive2://NN2:10000/dm jdbcPrincipal=hive/_HOST@REALM workflow.xml <workflow-app name="hivedemo" xmlns="uri:oozie:workflow:0.4"> <global> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> </global> <credentials> <credential name="hs2-creds" type="hive2"> <property> <name>hive2.jdbc.url</name> <value>${jdbcURL}</value> </property> <property> <name>hive2.server.principal</name> <value>${jdbcPrincipal}</value> </property> </credential> </credentials> <start to="hive2"/> <action name="hive2" cred="hs2-creds"> <hive2 xmlns="uri:oozie:hive2-action:0.1"> <jdbc-url>${jdbcURL}</jdbc-url> <script>${hivescript}</script> </hive2> <ok to="End"/> <error to="Kill"/> </action> <kill name="Kill"> <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="End"/> </workflow-app> Run oozie job under oozie user with kinit under oozie In oozie console I got error : HiveSQLException: Error retrieving delegation token for user oozie What I do wrong? ... View more

Re: Ranger + AD: sync users from groups question

by Explorer nord_tramper in Support Questions
‎06-01-2017 07:59 AM
‎06-01-2017 07:59 AM
@spolavarapu thank! It is exactly my case ... View more

Re: Ranger + AD: sync users from groups question

by Explorer nord_tramper in Support Questions
‎05-31-2017 08:54 AM
‎05-31-2017 08:54 AM
@spolavarapu Filter on user was there for ages. And a short time ago sync user from groups task appears and looks like the filter prevent user from sync ... View more

Re: Ranger + AD: sync users from groups question

by Explorer nord_tramper in Support Questions
‎05-29-2017 08:54 AM
‎05-29-2017 08:54 AM
Thanks, all! I have all settings in place but was not sure that it is correct. Only thing that prevent from correct sync was User Filter where I restrict only exact user list and new users for groups can't be synced into Ranger because of filter ... View more

Ranger + AD: sync users from groups question

by Explorer nord_tramper in Support Questions
‎05-26-2017 08:13 AM
‎05-26-2017 08:13 AM
Hi, all! Environment: RHEL 7.2 + Winbind HDP 2.5 Ranger 0.6.0.2.5 AD: Windows 2008 R2 Server User sync and group sync configured. QUESTION: I have some groups in AD with users inside. User in group pointed as member=CN=FirstName LastName, DN=EXAMPLE, DN=COM Exactly the same FirstName LastName synced inside Ranger while usersync working. However Ranger use sAMAccountName in policy and sAMAccountName came from Kerberos. Is it possible to sync user from groups with sAMAccountName instead of CN? ... View more

DataNode can't start after enabling Kerberos

by Explorer nord_tramper in Support Questions
‎12-21-2016 10:32 AM
‎12-21-2016 10:32 AM
Hello, all! After enabling Kerberos all services start fine except all DataNodes. RHEL 7 HDP 2.5 krb5.conf [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log [libdefaults] dns_lookup_realm = false ticket_lifetime = 24h renew_lifetime = 7d forwardable = true rdns = false default_realm = REALM.COM #default_ccache_name = KEYRING:persistent:%{uid} [realms] REALM.COM = { kdc = DC08 admin_server = DC08 } [domain_realm] .realm.com = REALM.COM realm.com = REALM.COM Erro in logfile ==> /var/log/hadoop/hdfs/jsvc.err <== Initializing secure datanode resources java.lang.IllegalArgumentException: Can't get Kerberos realm at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:289) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:274) at org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:351) at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:345) at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.getSecureResources(SecureDataNodeStarter.java:92) at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.init(SecureDataNodeStarter.java:71) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:207) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:88) at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63) ... 11 more Caused by: KrbException: Cannot locate default realm at sun.security.krb5.Config.getDefaultRealm(Config.java:1029) ... 17 more Cannot load daemon Service exit with a return value of 3 Initializing secure datanode resources ... View more

Re: Hive View 1.5.0 error - The client is configur...

by Explorer nord_tramper in Support Questions
‎10-24-2016 03:09 PM
‎10-24-2016 03:09 PM
Thanks for help! Issue gone after ambari-server reatrt. Don't know what was the root cause. BTW I've done all steps from your article ... View more

Re: HiveServer can't start after switch on Kerbero...

by Explorer nord_tramper in Support Questions
‎10-24-2016 03:05 PM
‎10-24-2016 03:05 PM
kinit is working well. Problem solved adter reading this https://hortonworks.jira.com/browse/BUG-42602 and change hive-site.xml hive.cluster.delegation.token.store.class=org.apache.hadoop.hive.thrift.DBTokenStore ... View more
Contact Me
Online
Offline
Last Visited
‎01-29-2018 04:00 PM
Public Statistics
Date Registered ‎10-14-2016 09:58 AM
Last Visited ‎01-29-2018 04:00 PM
Total Messages Posted 35
Total Kudos Received 13