Created 05-31-2016 06:57 AM
Hi, I am trying https://falcon.apache.org/HiveIntegration.html with HDP 2.3.4 in both source and target cluster.
There is know limitation mentioned in the link as follows:
Oozie 4.x with Hadoop-2.x
Replication jobs are submitted to oozie on the destination cluster.
Oozie runs a table export job on RM on source cluster.
Oozie server on the target cluster must be configured with source hadoop configs else jobs fail with errors on secure and non-secure clusters as below:
org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not found for ApplicationAttempt appattempt_1395965672651_0010_000002
So I have configured my oozie as given below:
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name> <value>*=/etc/hadoop/conf,nr1.hwxblr.com:8020=/etc/primary_conf/conf,nr3.hwxblr.com:8030=/etc/primary_conf/conf,nr21.hwxblr.com:8020=/etc/hadoop/conf,nr23.hwxblr.com:8030=/etc/hadoop/conf</value>
Still I am facing below error.
2016-05-31 06:31:34,170 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33 2016-05-31 06:31:34,204 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at nr23.hwxblr.com/10.0.1.25:8030 2016-05-31 06:31:34,235 WARN [main] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): appattempt_1464674342681_0001_000002 not found in AMRMTokenSecretManager. at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:558) at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:373) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:727) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:723) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422)
However note that this job is running on source cluster RM from target cluster oozie and as per the log it looks like AM is trying to connect to target RM instead of source RM.
Anyone can share any details on what could be wrong?
Created 05-31-2016 09:43 AM
Hi All,
I discovered that the issue was with my below configuration:
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name><value>*=/etc/hadoop/conf,nr1.hwxblr.com:8020=/etc/primary_conf/conf,nr3.hwxblr.com:8030=/etc/primary_conf/conf,nr21.hwxblr.com:8020=/etc/hadoop/conf,nr23.hwxblr.com:8030=/etc/hadoop/conf</value>
Instead of 8030 port it should be 8050.
Thanks @Kuldeep Kulkarni for finding this.
Created 05-31-2016 09:43 AM
Hi All,
I discovered that the issue was with my below configuration:
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name><value>*=/etc/hadoop/conf,nr1.hwxblr.com:8020=/etc/primary_conf/conf,nr3.hwxblr.com:8030=/etc/primary_conf/conf,nr21.hwxblr.com:8020=/etc/hadoop/conf,nr23.hwxblr.com:8030=/etc/hadoop/conf</value>
Instead of 8030 port it should be 8050.
Thanks @Kuldeep Kulkarni for finding this.