Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

problem with making external cluster on HDP sandbox 2.4

Solved Go to solution
Highlighted

problem with making external cluster on HDP sandbox 2.4

Explorer

As following HDP falcon tutorial, I succeeded to test mirroring function from sandbox cluster to sandbox cluster.

Then using falcon mirror function, I want to make a replication from sandbox cluster to external cluster what I have. (the name is "edumst01.sbp.com")

But I failed to make a cluster entity. here are the information what I write on falcon web ui and the falcon.applicaiton.log message.

Please check it and tell me the right way to make a external cluster.

For your information, my cluster don't have falcon, jmx, only is installed hdfs, yarn, oozie.

<the data what I write>

Readonly hftp://edumst01.sbp.com:50070

Write hdfs://edumst01.sbp.com:8020

Execute edumst01.sbp.com:8050

Workflow http://edumst01.sbp.com:11000/oozie/

Messaging tcp://sandbox.hortonworks.com:61616?daemon=true

<the falcon log>

2016-05-17 08:53:21,308 INFO - [679586926@qtp-1820624068-3 - 8835df07-48b3-4f5b-9780-53efb2a9429e:ambari-qa:POST//entities/submit/cluster] ~ Retrying connect to server: edumst01.sbp.com/112.175.246.34:8050. Already tried 41 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) (Client:869)

2016-05-17 08:53:23,328 INFO - [679586926@qtp-1820624068-3 - 8835df07-48b3-4f5b-9780-53efb2a9429e:ambari-qa:POST//entities/submit/cluster] ~ Retrying connect to server: edumst01.sbp.com/112.175.246.34:8050. Already tried 42 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) (Client:869)

1 ACCEPTED SOLUTION

Accepted Solutions

Re: problem with making external cluster on HDP sandbox 2.4

Explorer

I solved the problem!! I added the property "hadoop.proxyuser.falcon.groups", "hadoop.proxyuser.falcon.hosts" on my external cluster core-stie.xml.

View solution in original post

6 REPLIES 6
Highlighted

Re: problem with making external cluster on HDP sandbox 2.4

Rising Star
@Hanna Lee

Please make sure the execute endpoint is using port 8050. You can find this in Yarn configs under yarn.resourcemanager.address. This port can be 8032 on some nodes, but I think sandbox uses 8050. Once you have the right port, please restart falcon and try submitting cluster again. The restart is required because there could be a stale FileSystem object in Falcon server.

Highlighted

Re: problem with making external cluster on HDP sandbox 2.4

Explorer

Thank you! After I changed the execute port to 8032, I passed the step to validate a yarn resource manager.

But I still have a problem. I got a error message while I make the Locations. Here are the information what I write on falcon web UI and the falcon.application.log message.

On my external cluster, I made these two staging/working directory with ambari-qa user.

<Locations Information on Falcon web UI>

staging: /user/falcon/sbpRndCluster/staging

temp: /tmp

working: /user/falcon/sbpRndCluster/working

<falcon log message>

2016-05-18 01:28:54,989 INFO - [1431190494@qtp-1526367312-10 - cf529336-f2e6-4078-805c-e60346e79ed7:ambari-qa:POST//entities/submit/cluster] ~ Creating FS impersonating user ambari-qa (Hado opClientFactory:196)

2016-05-18 01:28:55,003 ERROR - [1431190494@qtp-1526367312-10 - cf529336-f2e6-4078-805c-e60346e79ed7:ambari-qa:POST//entities/submit/cluster] ~ Unable to persist entity object (AbstractEntit yManager:198) org.apache.falcon.entity.parser.ValidationException: Unable to validate the location with path: /apps/falcon/primaryCluster/staging for cluster:sbpRndCluster due to transient failures

<Cluster Directory>

[root@edumst01 ~]# hadoop fs -ls /user/falcon/sbpRndCluster Found 2 items

drwxrwxrwx - ambari-qa supergroup 0 2016-05-17 17:24 /user/falcon/sbpRndCluster/staging

drwxrwxrwx - ambari-qa supergroup 0 2016-05-17 17:24 /user/falcon/sbpRndCluster/working

Highlighted

Re: problem with making external cluster on HDP sandbox 2.4

Rising Star

The working and staging dir should have the following permissions.

hadoop fs -chown -R falcon /user/falcon/sbpRndCluster/

hadoop fs -chmod -R 777 /user/falcon/sbpRndCluster/staging

hadoop fs -chmod -R 755 /user/falcon/sbpRndCluster/working

Once you set these permissions, submitting cluster should be straightforward. When you see errors like "Unable to validate the location with path: /apps/falcon/primaryCluster/staging for cluster:sbpRndCluster due to transient failures", can you please share the logs from falcon.application.log, this will give us more information about what is happening.

Highlighted

Re: problem with making external cluster on HDP sandbox 2.4

Explorer

When I set up new external cluster on HDP sandbox, readonly/write/execute path is new cluster info but workflow, message path is sandbox info. Among external cluster and sandbox cluster, where can I make staging/temp/working directory?

Highlighted

Re: problem with making external cluster on HDP sandbox 2.4

Explorer

And Falcon is only installed on sandbox, now my cluster.

Re: problem with making external cluster on HDP sandbox 2.4

Explorer

I solved the problem!! I added the property "hadoop.proxyuser.falcon.groups", "hadoop.proxyuser.falcon.hosts" on my external cluster core-stie.xml.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here