Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Falcon HiveIntegration

Solved Go to solution
Highlighted

Falcon HiveIntegration

Hi, I am trying https://falcon.apache.org/HiveIntegration.html with HDP 2.3.4 in both source and target cluster.

There is know limitation mentioned in the link as follows:

Oozie 4.x with Hadoop-2.x

Replication jobs are submitted to oozie on the destination cluster.

Oozie runs a table export job on RM on source cluster.

Oozie server on the target cluster must be configured with source hadoop configs else jobs fail with errors on secure and non-secure clusters as below:

org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not found for ApplicationAttempt appattempt_1395965672651_0010_000002

So I have configured my oozie as given below:

<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=/etc/hadoop/conf,nr1.hwxblr.com:8020=/etc/primary_conf/conf,nr3.hwxblr.com:8030=/etc/primary_conf/conf,nr21.hwxblr.com:8020=/etc/hadoop/conf,nr23.hwxblr.com:8030=/etc/hadoop/conf</value>

Still I am facing below error.

2016-05-31 06:31:34,170 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2016-05-31 06:31:34,204 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at nr23.hwxblr.com/10.0.1.25:8030
2016-05-31 06:31:34,235 WARN [main] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): appattempt_1464674342681_0001_000002 not found in AMRMTokenSecretManager.
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:558)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:373)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:727)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:723)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422) 

However note that this job is running on source cluster RM from target cluster oozie and as per the log it looks like AM is trying to connect to target RM instead of source RM.

Anyone can share any details on what could be wrong?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Falcon HiveIntegration

Hi All,

I discovered that the issue was with my below configuration:

<name>oozie.service.HadoopAccessorService.hadoop.configurations</name><value>*=/etc/hadoop/conf,nr1.hwxblr.com:8020=/etc/primary_conf/conf,nr3.hwxblr.com:8030=/etc/primary_conf/conf,nr21.hwxblr.com:8020=/etc/hadoop/conf,nr23.hwxblr.com:8030=/etc/hadoop/conf</value>

Instead of 8030 port it should be 8050.

Thanks @Kuldeep Kulkarni for finding this.

View solution in original post

1 REPLY 1

Re: Falcon HiveIntegration

Hi All,

I discovered that the issue was with my below configuration:

<name>oozie.service.HadoopAccessorService.hadoop.configurations</name><value>*=/etc/hadoop/conf,nr1.hwxblr.com:8020=/etc/primary_conf/conf,nr3.hwxblr.com:8030=/etc/primary_conf/conf,nr21.hwxblr.com:8020=/etc/hadoop/conf,nr23.hwxblr.com:8030=/etc/hadoop/conf</value>

Instead of 8030 port it should be 8050.

Thanks @Kuldeep Kulkarni for finding this.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here