Member since
10-14-2016
35
Posts
13
Kudos Received
0
Solutions
01-26-2018
08:17 AM
Found in oozie.log error[E1601], E1601: Cannot retrieve JMS connection info [JMSTopicService is not initialized. JMS notificationmay not be enabled]org.apache.oozie.servlet.XServletException: E1601: Cannot retrieve JMS connection info [JMSTopicService is not initialized. JMS notificationmay not be enabled]
... View more
01-25-2018
05:28 PM
HDP 2.5. Try to set up Falcon Hive Mirror for one database DB create on both HiveServers. Got error Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [-1] No logs in YARN. No error in /var/log/oozie/oozie.log
... View more
Labels:
- Labels:
-
Apache Falcon
-
Apache Hive
01-16-2018
09:13 AM
Issue is constant - any try to start Falcon Hive mirror lead to this error. BTW, I add both falcon user and AD_USER_1 to rules in Ranger policy so issue is somewhere deeper
... View more
01-16-2018
09:03 AM
here is the hivemetastore for another try of same job run 2018-01-16 11:42:01,657 INFO [pool-5-thread-197]: metastore.ObjectStore (ObjectStore.java:initializeHelper(377)) - ObjectStore, initialize called
2018-01-16 11:42:01,665 INFO [pool-5-thread-197]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(139)) - Using direct SQL, underlying DB is OTHER
2018-01-16 11:42:01,666 INFO [pool-5-thread-197]: metastore.ObjectStore (ObjectStore.java:setConf(291)) - Initialized ObjectStore
2018-01-16 11:42:07,596 INFO [pool-5-thread-197]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 198: get_databases: NonExistentDatabaseUsedForHealthCheck
2018-01-16 11:42:07,596 INFO [pool-5-thread-197]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=falcon/prjpjnd201lv.EXAMPLE.COM@EXAMPLE.COM ip=10.151.2.69 cmd=get_databases: NonExistentDatabaseUsedForHealthCheck
2018-01-16 11:42:07,599 INFO [pool-5-thread-197]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 198: get_delegation_token
2018-01-16 11:42:07,600 INFO [pool-5-thread-197]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=falcon/prjpjnd201lv.EXAMPLE.COM@EXAMPLE.COM ip=10.151.2.69 cmd=get_delegation_token
2018-01-16 11:42:07,600 INFO [pool-5-thread-197]: delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:createPassword(385)) - Creating password for identifier: owner=AD_USER_1, renewer=hive, realUser=falcon/prjpjnd201lv.EXAMPLE.COM@EXAMPLE.COM, issueDate=1516092127600, maxDate=1516696927600, sequenceNumber=28, masterKeyId=1064, currentKey: 1064
2018-01-16 11:42:07,616 INFO [pool-5-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 1: get_database: falcon_test
2018-01-16 11:42:07,617 INFO [pool-5-thread-1]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=AD_USER_1 ip=10.151.2.69 cmd=get_database: falcon_test
2018-01-16 11:43:41,825 INFO [pool-5-thread-197]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 198: get_databases: NonExistentDatabaseUsedForHealthCheck
2018-01-16 11:43:41,825 INFO [pool-5-thread-197]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=falcon/prjpjnd201lv.EXAMPLE.COM@EXAMPLE.COM ip=10.151.2.69 cmd=get_databases: NonExistentDatabaseUsedForHealthCheck
2018-01-16 11:43:41,829 INFO [pool-5-thread-197]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 198: get_delegation_token
2018-01-16 11:43:41,829 INFO [pool-5-thread-197]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=falcon/prjpjnd201lv.EXAMPLE.COM@EXAMPLE.COM ip=10.151.2.69 cmd=get_delegation_token
2018-01-16 11:43:41,829 INFO [pool-5-thread-197]: delegation.AbstractDelegationTokenSecretManager (AbstractDelegationTokenSecretManager.java:createPassword(385)) - Creating password for identifier: owner=AD_USER_1, renewer=hive, realUser=falcon/prjpjnd201lv.EXAMPLE.COM@EXAMPLE.COM, issueDate=1516092221829, maxDate=1516697021829, sequenceNumber=29, masterKeyId=1064, currentKey: 1064
2018-01-16 11:43:41,993 WARN [pool-5-thread-23]: conf.HiveConf (HiveConf.java:initialize(3060)) - HiveConf of name hive.log.dir does not exist
2018-01-16 11:43:41,994 WARN [pool-5-thread-23]: conf.HiveConf (HiveConf.java:initialize(3060)) - HiveConf of name hive.log.file does not exist
2018-01-16 11:43:41,994 INFO [pool-5-thread-23]: metastore.HiveMetaStore (HiveMetaStore.java:newRawStoreForConf(701)) - 24: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
2018-01-16 11:43:42,013 INFO [pool-5-thread-23]: metastore.ObjectStore (ObjectStore.java:initializeHelper(377)) - ObjectStore, initialize called
2018-01-16 11:43:42,019 INFO [pool-5-thread-23]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(139)) - Using direct SQL, underlying DB is OTHER
2018-01-16 11:43:42,019 INFO [pool-5-thread-23]: metastore.ObjectStore (ObjectStore.java:setConf(291)) - Initialized ObjectStore
2018-01-16 11:43:42,023 INFO [pool-5-thread-23]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 24: get_database: falcon_test
2018-01-16 11:43:42,023 INFO [pool-5-thread-23]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=AD_USER_1 ip=10.151.2.69 cmd=get_database: falcon_test
2018-01-16 11:44:01,957 INFO [pool-5-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 1: get_table : db=dm tbl=tnl1
2018-01-16 11:44:01,957 INFO [pool-5-thread-1]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=spark2/prjp1c@EXAMPLE.COM ip=10.151.2.57 cmd=get_table : db=dm tbl=tnl1
2018-01-16 11:44:56,241 INFO [pool-5-thread-23]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 24: get_all_databases
2018-01-16 11:44:56,241 INFO [pool-5-thread-23]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=ambari-qa/prjp1c@EXAMPLE.COM ip=10.151.2.55 cmd=get_all_databases
2018-01-16 11:44:56,257 INFO [pool-5-thread-23]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 24: get_functions: db=bi_test pat=*
2018-01-16 11:44:56,257 INFO [pool-5-thread-23]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=ambari-qa/prjp1c@EXAMPLE.COM ip=10.151.2.55 cmd=get_functions: db=bi_test pat=*
2018-01-16 11:44:56,263 INFO [pool-5-thread-23]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 24: get_functions: db=dds pat=*
2018-01-16 11:44:56,264 INFO [pool-5-thread-23]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(406)) - ugi=ambari-qa/prjp1c@EXAMPLE.COM ip=10.151.2.55 cmd=get_functions: db=dds pat=*
2018-01-16 11:44:56,267 INFO [pool-5-thread-23]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(855)) - 24: get_functions: db=default pat=*
... View more
01-16-2018
08:14 AM
RHEL 7.3 HDP 2.5 Kerberos enabled Create two clusters in Falcon UI. Try to create Hive mirror. Oozie job created but failed with following error: UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient Full stack in /var/log/oozie/oozie.log 2018-01-15 18:45:22,762 WARN ActionStartXCommand:523 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[falcon-dr-hive-workflow] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@last-event] Error starting action [last-event]. ErrorType [ERROR], ErrorCode [UncheckedExecutionException], Message [UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient] org.apache.oozie.action.ActionExecutorException: UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1206) at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1377) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65) at org.apache.oozie.command.XCommand.call(XCommand.java:287) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2234) at com.google.common.cache.LocalCache.get(LocalCache.java:3965) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764) at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:291) at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:266) at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558) at org.apache.hive.hcatalog.api.HCatClientHMSImpl.initialize(HCatClientHMSImpl.java:823) at org.apache.hive.hcatalog.api.HCatClient.create(HCatClient.java:71) at org.apache.oozie.action.hadoop.HCatCredentialHelper.getHCatClient(HCatCredentialHelper.java:105) at org.apache.oozie.action.hadoop.HCatCredentialHelper.set(HCatCredentialHelper.java:57) at org.apache.oozie.action.hadoop.HCatCredentials.addtoJobConf(HCatCredentials.java:61) at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1291) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1128) ... 11 more Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1566) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:92) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:124) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:294) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:291) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) ... 23 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564) ... 33 more Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:486) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:281) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:187) at org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:406) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:92) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:124) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:294) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:291) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) at com.google.common.cache.LocalCache.get(LocalCache.java:3965) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764) at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:291) at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:266) at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558) at org.apache.hive.hcatalog.api.HCatClientHMSImpl.initialize(HCatClientHMSImpl.java:823) at org.apache.hive.hcatalog.api.HCatClient.create(HCatClient.java:71) at org.apache.oozie.action.hadoop.HCatCredentialHelper.getHCatClient(HCatCredentialHelper.java:105) at org.apache.oozie.action.hadoop.HCatCredentialHelper.set(HCatCredentialHelper.java:57) at org.apache.oozie.action.hadoop.HCatCredentials.addtoJobConf(HCatCredentials.java:61) at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1291) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1128) at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1377) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:65) at org.apache.oozie.command.XCommand.call(XCommand.java:287) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:532) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:281) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:187) at org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:406) ... 38 more 2018-01-15 18:45:22,762 WARN ActionStartXCommand:523 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[falcon-dr-hive-workflow] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@last-event] Setting Action Status to [DONE] 2018-01-15 18:45:22,787 INFO ActionEndXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[falcon-dr-hive-workflow] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@last-event] ERROR is considered as FAILED for SLA 2018-01-15 18:45:22,854 INFO ActionStartXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[falcon-dr-hive-workflow] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@fail] Start action [0001108-180109162406621-oozie-oozi-W@fail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2018-01-15 18:45:22,854 INFO ActionStartXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[falcon-dr-hive-workflow] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@fail] [***0001108-180109162406621-oozie-oozi-W@fail***]Action status=DONE 2018-01-15 18:45:22,855 INFO ActionStartXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[falcon-dr-hive-workflow] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@fail] [***0001108-180109162406621-oozie-oozi-W@fail***]Action updated in DB! 2018-01-15 18:45:23,025 INFO WorkflowNotificationXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@fail] No Notification URL is defined. Therefore nothing to notify for job 0001108-180109162406621-oozie-oozi-W@fail 2018-01-15 18:45:23,026 INFO CallbackServlet:520 - SERVER[prjnnd201lv.examle.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@user-action] callback for action [0001107-180109162406621-oozie-oozi-W@user-action] 2018-01-15 18:45:23,027 INFO WorkflowNotificationXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001108-180109162406621-oozie-oozi-W] ACTION[0001108-180109162406621-oozie-oozi-W@last-event] No Notification URL is defined. Therefore nothing to notify for job 0001108-180109162406621-oozie-oozi-W@last-event 2018-01-15 18:45:23,067 INFO ActionEndXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_falcon3] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@user-action] ERROR is considered as FAILED for SLA 2018-01-15 18:45:23,201 INFO ActionStartXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_falcon3] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@failed-post-processing] Start action [0001107-180109162406621-oozie-oozi-W@failed-post-processing] with user-retry state : userRetryCount [0], userRetryMax [3], userRetryInterval [1] 2018-01-15 18:45:23,748 WARN JobResourceUploader:64 - SERVER[prjnnd201lv.examle.com] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2018-01-15 18:45:23,752 WARN JobResourceUploader:171 - SERVER[prjnnd201lv.examle.com] No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2018-01-15 18:45:25,157 INFO JavaActionExecutor:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_falcon3] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@failed-post-processing] Trying to get job [job_1513087944025_15487], attempt [1] 2018-01-15 18:45:25,192 INFO JavaActionExecutor:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_falcon3] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@failed-post-processing] checking action, hadoop job ID [job_1513087944025_15487] status [RUNNING] 2018-01-15 18:45:25,195 INFO ActionStartXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_falcon3] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@failed-post-processing] [***0001107-180109162406621-oozie-oozi-W@failed-post-processing***]Action status=RUNNING 2018-01-15 18:45:25,195 INFO ActionStartXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[AD_USER_1] GROUP[-] TOKEN[] APP[FALCON_PROCESS_DEFAULT_falcon3] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@failed-post-processing] [***0001107-180109162406621-oozie-oozi-W@failed-post-processing***]Action updated in DB! 2018-01-15 18:45:25,205 INFO WorkflowNotificationXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@user-action] No Notification URL is defined. Therefore nothing to notify for job 0001107-180109162406621-oozie-oozi-W@user-action 2018-01-15 18:45:25,205 INFO WorkflowNotificationXCommand:520 - SERVER[prjnnd201lv.examle.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@failed-post-processing] No Notification URL is defined. Therefore nothing to notify for job 0001107-180109162406621-oozie-oozi-W@failed-post-processing 2018-01-15 18:45:29,256 ERROR CompletedActionXCommand:517 - SERVER[prjnnd201lv.examle.com] USER[-] GROUP[-] TOKEN[] APP[-] JOB[0001107-180109162406621-oozie-oozi-W] ACTION[0001107-180109162406621-oozie-oozi-W@user-action] XException, org.apache.oozie.command.CommandException: E0800: Action it is not running its in [ERROR] state, action [0001107-180109162406621-oozie-oozi-W@user-action] at org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:92) at org.apache.oozie.command.XCommand.call(XCommand.java:258) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
... View more
Labels:
- Labels:
-
Apache Falcon
-
Apache Hive
11-22-2017
10:17 AM
@Jay Kumar SenSharma Hello, I add following line to auth_to_local RULE:[2:$1@$0](ambari-server@REALM)s/.*/ambari-server/ Still same errors Caused by: org.apache.hive.service.cli.HiveSQLException: Failed to validate proxy privilege of ambari-server for admin
....
....
Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User: ambari-server is not allowed to impersonate admin
... View more
11-21-2017
01:44 PM
@Jay Kumar SenSharma Here it is. Yep there was advised principal like ambari-serve-ClusterName@REALM, but our AD has strict rule that Principal should be in format <service-name>/<host>@<realm> so I modify it to meet requirements. So is there any thing that I can do with this Principal (assume that I can't change it to for ambari-serve-ClusterName@REALM)? klist -kte /etc/security/keytabs/ambari.server.keytab
Keytab name: FILE:/etc/security/keytabs/ambari.server.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
0 01/01/1970 03:00:00 ambari-server/ClusterName@REALM (des-cbc-crc)
0 01/01/1970 03:00:00 ambari-server/ClusterName@REALM (des-cbc-md5)
0 01/01/1970 03:00:00 ambari-server/ClusterName@REALM (arcfour-hmac)
0 01/01/1970 03:00:00 ambari-server/ClusterName@REALM (aes256-cts-hmac-sha1-96)
0 01/01/1970 03:00:00 ambari-server/ClusterName@REALM (aes128-cts-hmac-sha1-96)
... View more
11-21-2017
01:00 PM
Also in HDFS View I have error java.lang.IllegalArgumentException: Invalid value: "ambari-server/JournalNode" does not belong to the domain ^[A-Za-z_][A-Za-z0-9._-]*[$]?$
... View more
11-21-2017
12:55 PM
1 Kudo
@Aditya Sirna Added hadoop.proxyuser.ambari-server.users=* into custom core-site section. Restart both cluster and Ambari - no effect
... View more
11-21-2017
09:37 AM
@Jay Kumar SenSharma I have following settings # cat /etc/ambari-server/conf/krb5JAASLogin.conf
com.sun.security.jgss.krb5.initiate {
com.sun.security.auth.module.Krb5LoginModule required
renewTGT=false
doNotPrompt=true
useKeyTab=true
keyTab="/etc/security/keytabs/ambari.server.keytab"
principal=<a href="mailto:ambari-server/JounralNode@REALM">ambari-server/JounralNode@REALM</a>
storeKey=true
useTicketCache=false;
};
Principal name is ambari-server/JounralNode@REALM . /JounralNode has been add as a requirement from AD admins to create Principal. So I have to set up
hadoop.proxyuser.ambari-server/JounralNode.groups=*
hadoop.proxyuser.ambari-server/JounralNode.hosts=*
But it is not possible
... View more
11-20-2017
04:38 PM
Hello, all! Environment: RHEL 7.3, HDP 2.6, Kerberos, ActiveDirectory Hive view error: Message: Cannot open a hive connection with connect string jdbc:hive2://nn1:2181,nn2:2181,dn1:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;hive.server2.proxy.user=admin
And deep in logs: Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User: ambari-server is not allowed to impersonate admin Following settings already in place in core-site.xml hadoop.proxyuser.ambari-server.groups=*
hadoop.proxyuser.ambari-server.hosts=*
hadoop.proxyuser.root.groups=*
hadoop.proxyuser.root.hosts=*
... View more
Labels:
- Labels:
-
Apache Ambari
10-30-2017
12:19 PM
1 Kudo
Also works for me after some experiments: yarn.admin.acl=yarn,dr.who,<AD LOGIN IN UPPERCASE WITHOUT REALM>
... View more
10-30-2017
11:42 AM
Your settings solve the problem. I have default values yarn.admin.acl=yarn,dr.who
yarn.acl.enable=true
hadoop.http.staticuser.user = yarn
... View more
10-30-2017
11:13 AM
Restart ambari-agent on nodes
... View more
10-30-2017
10:33 AM
Hello! I have HDP 2.5 cluster with KERBEROS enabled, connected to Active Directory. When I try to switch on HTTP AUTH - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_security/content/_configuring_http_authentication_for_HDFS_YARN_MapReduce2_HBase_Oozie_Falcon_and_Storm.html Logs can be retrived using shell yarn logs -applicationId application_1509115509826_0001 I can't access any logs from YARN UI for example <YARN-RM-HOST>:19888/jobhistory/logs/<NODE>:45454/container_e56_1509115509826_0001_01_000001/container_e56_1509115509826_0001_01_000001/hive With following error: User <MY Active Directory User> is not authorized to view the logs for container_e56_1509115509826_0001_01_000001 in log file [<NODE>_45454_1509118017724]No logs available for container container_e56_1509115509826_0001_01_000001
... View more
Labels:
- Labels:
-
Apache YARN
10-30-2017
10:17 AM
Hello! Same logs I can find using web-ui - there is nothing about cause why job got faild. Containers simply Task Transitioned from RUNNING to KILL_WAIT
... View more
10-12-2017
01:39 PM
Can't do kinit because this job initiated by MS SQL Polybase connector
... View more
10-11-2017
11:42 AM
1 Kudo
@Nils Sjögren thanks - your advice was helpful A couple of remarks: 1. If you have HA enabled for YARN - port for RM should be 8032 and you should point to Active RM host 2. If you have HA enabled for NameNode - use hdfs://HACLUSTERNAME instead of url for NameNode
... View more
10-10-2017
06:36 PM
Thanks! Which keytab I should use to get access to this Zookepper Node (kerberized cluster)? Is it possible to use ZooKeeper Java API to get same info?
... View more
10-10-2017
01:21 PM
3 Kudos
Is it possible to identify active YARN ResourceManager using ZooKeeper or any other method?
... View more
Labels:
- Labels:
-
Apache YARN
10-06-2017
08:16 AM
Same logs I found in YARN RM console One more suspicious line - before start container 2017-10-05 18:20:26,264 WARN [main] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
... View more
10-05-2017
02:50 PM
Environment: HDP 2.6 with Kerberson enabled OS RedHat 7.3 MS SQL Server 2017 with Polybase Polybase works fine when during direct loading from HDFS. However if I use FORCE EXTERNALPUSHDOWN option (it means that Polybase start MR job on the cluster) - all containers fail with following error 2017-10-05 16:29:25,066 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e50_1507209892753_0004_01_0000292017-10-05 16:29:25,067 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e50_1507209892753_0004_01_0000162017-10-05
16:29:25,068 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:27 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:29 ContRel:0 HostLocal:25 RackLocal:42017-10-05
16:29:25,075 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1507209892753_0004_m_000016_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP2017-10-05
16:29:25,075 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1507209892753_0004_m_000016_0: Exception from container-launch.Container id: container_e50_1507209892753_0004_01_000029Exit code: 1Stack trace: org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime.launchContainer(DefaultLinuxContainerRuntime.java:109) at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:89) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:392) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Where to find root cause?
... View more
Labels:
- Labels:
-
Apache YARN
09-13-2017
10:17 AM
1 Kudo
HDP 2.6. Kerberos enabled. Try run HiveServer2 job using oozie like in this article - https://community.hortonworks.com/articles/75107/oozie-hive-2-action-in-a-kerberized-cluster.html job.properties nameNode=hdfs://NN2:8020
jobTracker=NN1:8050
exampleDir=${nameNode}/user/${user.name}/ooziedemo/hivedem
ooozie.wf.application.path=${exampleDir}/app
oozie.use.system.libpath=true
# Hive2 action
hivescript=${oozie.wf.application.path}/add_row.sql
outputHiveDatabase=dm
jdbcURL=jdbc:hive2://NN2:10000/dm
jdbcPrincipal=hive/_HOST@REALM workflow.xml <workflow-app name="hivedemo" xmlns="uri:oozie:workflow:0.4">
<global>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
</global>
<credentials>
<credential name="hs2-creds" type="hive2">
<property>
<name>hive2.jdbc.url</name>
<value>${jdbcURL}</value>
</property>
<property>
<name>hive2.server.principal</name>
<value>${jdbcPrincipal}</value>
</property>
</credential>
</credentials>
<start to="hive2"/>
<action name="hive2" cred="hs2-creds">
<hive2 xmlns="uri:oozie:hive2-action:0.1">
<jdbc-url>${jdbcURL}</jdbc-url>
<script>${hivescript}</script>
</hive2>
<ok to="End"/>
<error to="Kill"/>
</action>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="End"/>
</workflow-app>
Run oozie job under oozie user with kinit under oozie In oozie console I got error : HiveSQLException: Error retrieving delegation token for user oozie What I do wrong?
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Oozie
06-01-2017
07:59 AM
@spolavarapu thank! It is exactly my case
... View more
05-31-2017
08:54 AM
@spolavarapu Filter on user was there for ages. And a short time ago sync user from groups task appears and looks like the filter prevent user from sync
... View more
05-29-2017
08:54 AM
Thanks, all! I have all settings in place but was not sure that it is correct. Only thing that prevent from correct sync was User Filter where I restrict only exact user list and new users for groups can't be synced into Ranger because of filter
... View more
05-26-2017
08:13 AM
Hi, all! Environment: RHEL 7.2 + Winbind HDP 2.5 Ranger 0.6.0.2.5 AD: Windows 2008 R2 Server User sync and group sync configured. QUESTION: I have some groups in AD with users inside. User in group pointed as member=CN=FirstName LastName, DN=EXAMPLE, DN=COM Exactly the same FirstName LastName synced inside Ranger while usersync working. However Ranger use sAMAccountName in policy and sAMAccountName came from Kerberos. Is it possible to sync user from groups with sAMAccountName instead of CN?
... View more
- Tags:
- Security
- user-groups
Labels:
10-24-2016
03:09 PM
Thanks for help! Issue gone after ambari-server reatrt. Don't know what was the root cause. BTW I've done all steps from your article
... View more
10-24-2016
03:05 PM
kinit is working well. Problem solved adter reading this https://hortonworks.jira.com/browse/BUG-42602 and change hive-site.xml hive.cluster.delegation.token.store.class=org.apache.hadoop.hive.thrift.DBTokenStore
... View more
10-24-2016
02:20 PM
Try to switch on Kerberos with manual creation of keytabs and principals. Principal and keytabs has been created. Keytabs distributed to all nodes. Almost all services started. Hive, HBase and Kafk don't. Now I try to solve Hive issue. hiveserver2.log 2016-10-24 16:57:10,382 ERROR [Thread-15-SendThread(DN2.REALM.COM:2181)]: client.ZooKeeperSaslClient (ZooKeeperSaslClient.java:createSaslToken(384)) - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2016-10-24 16:57:10,382 ERROR [Thread-15-SendThread(DN2.REALM.COM:2181)]: zookeeper.ClientCnxn (ClientCnxn.java:run(1059)) - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2016-10-24 16:57:10,383 ERROR [Thread-15-EventThread]: curator.ConnectionState (ConnectionState.java:checkState(245)) - Authentication failed
2016-10-24 16:57:10,390 FATAL [Thread-15]: thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2: could not start ThriftBinaryCLIService
org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: Error creating path /hive/cluster/delegation/HIVESERVER2/keys
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:166)
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:236)
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:469)
at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:92)
at org.apache.hive.service.auth.HiveAuthFactory.<init>(HiveAuthFactory.java:128)
at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:57)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /hive/cluster/delegation/HIVESERVER2/keys
at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:688)
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:672)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:668)
at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423)
at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:257)
at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
at org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:160)
... 6 more
2016-10-24 16:57:10,399 INFO [Thread-4]: server.HiveServer2 (HiveStringUtils.java:run(711)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at DN1.REALM.COM/172.26.140.200
************************************************************/
2016-10-24 16:57:10,419 INFO [main]: server.HiveServer2 (HiveServer2.java:addServerInstanceToZooKeeper(249)) - Created a znode on ZooKeeper for HiveServer2 uri: DN1.REALM.COM:10000
2016-10-24 16:57:10,420 INFO [Thread-13]: server.HiveServer2 (HiveServer2.java:stop(397)) - Shutting down HiveServer2
2016-10-24 16:57:10,420 INFO [Thread-13]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
2016-10-24 16:57:10,420 INFO [Thread-13]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2016-10-24 16:57:10,420 INFO [Thread-13]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
... View more
Labels:
- Labels:
-
Apache Hive