Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to execute workflows with WFM in a secure cluster.

Highlighted

Unable to execute workflows with WFM in a secure cluster.

New Contributor

Hi guys,

I'm using the Workflow Manager View to test distcp between a secure cluster and a non-secure cluster but YARN get stucks and I can't figure out why

The same distcp command executed by the same user in a terminal works fine.

Coud you give me a clue of where I can look information about it?

Thanks in advance!

APP Master log

2017-06-29 15:04:13,641 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2017-06-29 15:04:13,641 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:hdpcluster, Ident: (HDFS_DELEGATION_TOKEN token 500 for zeppelin)
2017-06-29 15:04:13,778 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 11 cluster_timestamp: 1498666390524 } attemptId: 1 } keyId: 991727874)
2017-06-29 15:04:13,845 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 10.137.147.25:8032,10.137.147.24:8032, Ident: (owner=zeppelin, renewer=yarn, realUser=oozie/management06.hdp-test.com@HDP-REALM.COM, issueDate=1498759444261, maxDate=1499364244261, sequenceNumber=38, masterKeyId=77)
2017-06-29 15:04:13,846 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: TIMELINE_DELEGATION_TOKEN, Service: 10.137.147.25:8188, Ident: (owner=zeppelin, renewer=yarn, realUser=oozie, issueDate=1498759444941, maxDate=1499364244941, sequenceNumber=29, masterKeyId=57)
2017-06-29 15:04:13,846 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: MR_DELEGATION_TOKEN, Service: 10.137.147.25:10020, Ident: (owner=zeppelin, renewer=yarn, realUser=oozie/management06.hdp-test.com@HDP-REALM.COM, issueDate=1498759444583, maxDate=1499364244583, sequenceNumber=8, masterKeyId=48)
2017-06-29 15:04:13,860 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
2017-06-29 15:04:13,861 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2017-06-29 15:04:18,556 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1498666390524_0011_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2017-06-29 15:04:18,562 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e33_1498666390524_0011_01_000002 taskAttempt attempt_1498666390524_0011_m_000000_0
2017-06-29 15:04:18,563 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1498666390524_0011_m_000000_0
2017-06-29 15:04:18,564 INFO [ContainerLauncher #0] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : datanode29.hdp-test.com:45454
2017-06-29 15:04:18,775 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1498666390524_0011_m_000000_0 : 13562
2017-06-29 15:04:18,776 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1498666390524_0011_m_000000_0] using containerId: [container_e33_1498666390524_0011_01_000002 on NM: [datanode29.hdp-test.com:45454]
2017-06-29 15:04:18,779 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1498666390524_0011_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2017-06-29 15:04:18,779 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1498666390524_0011_m_000000 Task Transitioned from SCHEDULED to RUNNING
2017-06-29 15:04:19,371 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1498666390524_0011: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:6848512, vCores:1396> knownNMs=32
2017-06-29 15:04:24,167 INFO [Socket Reader #1 for port 43879] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1498666390524_0011 (auth:SIMPLE)
2017-06-29 15:04:24,187 INFO [Socket Reader #1 for port 43879] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for job_1498666390524_0011 (auth:TOKEN) for protocol=interface org.apache.hadoop.mapred.TaskUmbilicalProtocol
2017-06-29 15:04:24,202 INFO [IPC Server handler 1 on 43879] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1498666390524_0011_m_36283883716610 asked for a task
2017-06-29 15:04:24,202 INFO [IPC Server handler 1 on 43879] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1498666390524_0011_m_36283883716610 given task: attempt_1498666390524_0011_m_000000_0
2017-06-29 15:04:31,210 INFO [IPC Server handler 2 on 43879] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1498666390524_0011_m_000000_0 is : 1.0
2017-06-29 15:05:01,294 INFO [IPC Server handler 11 on 43879] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1498666390524_0011_m_000000_0 is : 1.0
2017-06-29 15:05:31,377 INFO [IPC Server handler 20 on 43879] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1498666390524_0011_m_000000_0 is : 1.0
2017-06-29 15:06:01,446 INFO [IPC Server handler 2 on 43879] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1498666390524_0011_m_000000_0 is : 1.0
2017-06-29 15:06:31,523 INFO [IPC Server handler 11 on 43879] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1498666390524_0011_m_000000_0 is : 1.0
2017-06-29 15:06:58,578 INFO [IPC Server handler 19 on 43879] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1498666390524_0011_m_000000_0 is : 1.0
2017-06-29 15:07:28,633 INFO [IPC Server handler 1 on 43879] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1498666390524_0011_m_000000_0 is : 1.0
2017-06-29 15:07:58,688 INFO [IPC Server handler 12 on 43879] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1498666390524_0011_m_000000_0 is : 1.0
2017-06-29 15:08:28,752 INFO [IPC Server handler 19 on 43879] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1498666390524_0011_m_000000_0 is : 1.0
2017-06-29 15:08:58,811 INFO [IPC Server handler 1 on 43879] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1498666390524_0011_m_000000_0 is : 1.0
2017-06-29 15:09:28,871 INFO [IPC Server handler 12 on 43879] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1498666390524_0011_m_000000_0 is : 1.0

Container Log

2017-06-29 15:04:23,782 INFO [main] org.apache.hadoop.security.SecurityUtil: Updating Configuration
2017-06-29 15:04:23,821 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2017-06-29 15:04:23,879 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2017-06-29 15:04:23,879 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2017-06-29 15:04:23,893 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2017-06-29 15:04:23,893 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:hdpcluster, Ident: (HDFS_DELEGATION_TOKEN token 500 for zeppelin)
2017-06-29 15:04:24,041 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1498666390524_0011, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@359df09a)
2017-06-29 15:04:24,041 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: RM_DELEGATION_TOKEN, Service: 10.137.147.25:8032,10.137.147.24:8032, Ident: (owner=zeppelin, renewer=yarn, realUser=oozie/managementnode06.hdp-test.com@HDP-REALM.COM, issueDate=1498759444261, maxDate=1499364244261, sequenceNumber=38, masterKeyId=77)
2017-06-29 15:04:24,047 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: TIMELINE_DELEGATION_TOKEN, Service: 10.137.147.25:8188, Ident: (owner=zeppelin, renewer=yarn, realUser=oozie, issueDate=1498759444941, maxDate=1499364244941, sequenceNumber=29, masterKeyId=57)
2017-06-29 15:04:24,047 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: MR_DELEGATION_TOKEN, Service: 10.137.147.25:10020, Ident: (owner=zeppelin, renewer=yarn, realUser=oozie/managementnode06.hdp-test.com@HDP-REALM.COM, issueDate=1498759444583, maxDate=1499364244583, sequenceNumber=8, masterKeyId=48)
2017-06-29 15:04:24,092 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2017-06-29 15:04:24,264 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /apps/hadoop/yarn/local/usercache/zeppelin/appcache/application_1498666390524_0011
2017-06-29 15:04:24,630 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2017-06-29 15:04:25,134 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-06-29 15:04:25,134 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2017-06-29 15:04:25,143 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-06-29 15:04:25,332 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: org.apache.oozie.action.hadoop.OozieLauncherInputFormat$EmptySplit@6691490c
2017-06-29 15:04:25,337 INFO [main] org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
2017-06-29 15:04:25,367 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
2017-06-29 15:04:25,707 INFO [main] org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
2017-06-29 15:04:25,716 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:04:40,720 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:04:55,722 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:05:10,725 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:05:25,726 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:05:40,728 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:05:55,731 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:06:10,732 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:06:25,734 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:06:40,736 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:06:55,737 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:07:10,739 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:07:25,741 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:07:40,743 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:07:55,744 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:08:10,746 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:08:25,748 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:08:40,749 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:08:55,751 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:09:10,753 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:09:25,755 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:09:40,756 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:09:55,758 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:10:10,760 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:10:25,762 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:10:40,764 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
2017-06-29 15:10:55,766 WARN [pool-7-thread-1] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
1 REPLY 1

Re: Unable to execute workflows with WFM in a secure cluster.

New Contributor

Solution:

hadoop.security.token.service.use_ip : false.

https://community.hortonworks.com/articles/24277/parameters-for-multi-homing.html