[FALCON] : hive replication failed

[#1]

Hi all,

I'm follow my tuto : https://community.hortonworks.com/articles/38285/falcon-hive-integration.html

and checked : https://falcon.apache.org/HiveIntegration.html

I don't know what argument is expected in this feed :

[falcon@master003 HIVE]$ more replication-feed.xml
<?xml version="1.0" encoding="UTF-8"?>
<feed description="Monthly Analytics Summary" name="replication-feed" xmlns="uri:falcon:feed:0.1">
    <tags>EntityType=Feed</tags>
    <frequency>months(1)</frequency>

    <clusters>

         <cluster name="c-source-current" type="source">
         <validity start="2016-06-20T00:00Z" end="2016-06-30T00:00Z"/>
         <retention limit="months(36)" action="delete"/>
         <table uri="catalog:falcon_landing_db:summary_table#ds=${YEAR}-${MONTH}"/>
         </cluster>

         <cluster name="c-target-next" type="target">
         <validity start="2016-06-20T00:00Z" end="2016-06-30T00:00Z"/>
         <retention limit="months(180)" action="delete"/>
         <table uri="catalog:falcon_archive_db:falcon_summary_archive_table#ds=${YEAR}-${MONTH}"/>
         </cluster>

    </clusters>

    <table uri="catalog:falcon_landing_db:summary_table#ds=${YEAR}-${MONTH}" />
    <schema location="hcat" provider="hcat"/>
    <ACL owner="falcon" group="hadoop" permission="0755"/>

</feed>
[falcon@master003 HIVE]$ falcon entity  -type feed -submit -file replication-feed.xml
log4j:WARN No appenders could be found for logger (org.apache.hadoop.security.authentication.client.KerberosAuthenticator).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
ERROR: Bad Request;javax.xml.bind.UnmarshalException
 - with linked exception:
[org.xml.sax.SAXParseException; lineNumber: 23; columnNumber: 46; cvc-complex-type.2.4.a: Invalid content was found starting with element 'schema'. One of '{"uri:falcon:feed:0.1":notification, "uri:falcon:feed:0.1":ACL}' is expected.]
[falcon@master003 HIVE]$ https://community.hortonworks.com/articles/38285/falcon-hive-integration.html

i'm also try with this parameter but same error:

<schema location="" provider="hcatalog"/>

[#2 · Accepted solution]

@mayki wogno

You need to place

<ACL owner="falcon"group="hadoop" permission="0755"/>

before

<schema location="hcat" provider="hcat"/>

Ordering is important.

Also remove <table uri="catalog:falcon_landing_db:summary_table#ds=${YEAR}-${MONTH}"/> from source cluster. It is not required.

[#3]

@mayki wogno Order of schema and ACL is required to changed. It must be like this :

...
<ACL owner="falcon" group="hadoop" permission="0755"/>
<schema location="hcat" provider="hcat"/>
....

[#4]

@Rahul Pathak and @peeyush thanks !!

but why falcon check this order or write this information in falcon documentation.

[#5]

JA009: org.apache.hive.hcatalog.common.HCatException : 9001 : Exception 
occurred while processing HCat request : MetaException while getting 
delegation token.. Cause : MetaException(message:Unauthorized connection
 for super-user: oozie/master003.next.rec.mapreduce.m1.p.fti.net@FTI.NET
 from IP 10.98.138.87)

@Rahul Pathak and @peeyush : I have this error on target. it's missing some proxyuser ?

[#6]

Check this https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_ambari_views_guide/content/_setup_WebHCa...

[#7]

@Rahul Pathak : i've add

webhcat.proxyuser.oozie.hosts=* to hive service

but i've have new error :

JA009: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit 
application_1467203595416_0099 to YARN : Failed to renew token: Kind: 
HDFS_DELEGATION_TOKEN, Service: ha-hdfs:bigdata-next, Ident: 
(HDFS_DELEGATION_TOKEN token 10848 for falcon)

[#8]

Probably it runs in this bug : https://issues.apache.org/jira/browse/YARN-3021

My yarn version is : 2.7.1.2.3

[#9]

user falcon can submit yarn jar

[falcon@master003 hadoop-mapreduce-historyserver]$ yarn jar hadoop-mapreduce-examples-2.7.1.2.3.4.0-3485.jar pi 16 1000
Number of Maps  = 16
Samples per Map = 1000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Wrote input for Map #10
Wrote input for Map #11
Wrote input for Map #12
Wrote input for Map #13
Wrote input for Map #14
Wrote input for Map #15
Starting Job
16/06/29 16:27:07 INFO impl.TimelineClientImpl: Timeline service address: http://xxxx:8188/ws/v1/timeline/
16/06/29 16:27:07 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 10914 for falcon on ha-hdfs:bigdata-next
16/06/29 16:27:07 INFO security.TokenCache: Got dt for hdfs://bigdata-next; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:bigdata-next, Ident: (HDFS_DELEGATION_TOKEN token 10914 for falcon)
16/06/29 16:27:07 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
16/06/29 16:27:08 INFO input.FileInputFormat: Total input paths to process : 16
16/06/29 16:27:08 INFO mapreduce.JobSubmitter: number of splits:16
16/06/29 16:27:08 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1467207300026_0007
16/06/29 16:27:08 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:bigdata-next, Ident: (HDFS_DELEGATION_TOKEN token 10914 for falcon)
16/06/29 16:27:08 INFO impl.YarnClientImpl: Submitted application application_1467207300026_0007
16/06/29 16:27:08 INFO mapreduce.Job: The url to track the job: http://xxxxx:8088/proxy/application_1467207300026_0007/
16/06/29 16:27:08 INFO mapreduce.Job: Running job: job_1467207300026_0007
16/06/29 16:27:17 INFO mapreduce.Job: Job job_1467207300026_0007 running in uber mode : false

[#10]

More detail about renew token error from workflow :

2016-06-29 18:46:00,127 DEBUG HadoopAccessorService:526 - SERVER[xxxx] USER[falcon] GROUP[-] TOKEN[] APP[FALCON_FEED_REPLICATION_replication-feed-hive] JOB[0000101-160629105530892-oozie-oozi-W] ACTION[0000101-160629105530892-oozie-oozi-W@table-export] Checking if filesystem hdfs is supported
2016-06-29 18:46:00,129 DEBUG HiveActionExecutor:526 - SERVER[xxxx] USER[falcon] GROUP[-] TOKEN[] APP[FALCON_FEED_REPLICATION_replication-feed-hive] JOB[0000101-160629105530892-oozie-oozi-W] ACTION[0000101-160629105530892-oozie-oozi-W@table-export] Submitting the job through Job Client for action 0000101-160629105530892-oozie-oozi-W@table-export
2016-06-29 18:46:00,131 DEBUG HiveActionExecutor:526 - SERVER[xxxx] USER[falcon] GROUP[-] TOKEN[] APP[FALCON_FEED_REPLICATION_replication-feed-hive] JOB[0000101-160629105530892-oozie-oozi-W] ACTION[0000101-160629105530892-oozie-oozi-W@table-export] ADDING TOKEN: HIVE_DELEGATION_TOKEN_
2016-06-29 18:46:01,145  WARN ActionStartXCommand:523 - SERVER[xxxx] USER[falcon] GROUP[-] TOKEN[] APP[FALCON_FEED_REPLICATION_replication-feed-hive] JOB[0000101-160629105530892-oozie-oozi-W] ACTION[0000101-160629105530892-oozie-oozi-W@table-export] Error starting action [table-export]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1467203595416_0440 to YARN : Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:bigdata-next, Ident: (HDFS_DELEGATION_TOKEN token 10947 for falcon)]
org.apache.oozie.action.ActionExecutorException: JA009: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1467203595416_0440 to YARN : Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:bigdata-next, Ident: (HDFS_DELEGATION_TOKEN token 10947 for falcon)
    at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:456)
    at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:440)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1139)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1293)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:250)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:64)
    at org.apache.oozie.command.XCommand.call(XCommand.java:286)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:321)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:250)
    at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1467203595416_0440 to YARN : Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:bigdata-next, Ident: (HDFS_DELEGATION_TOKEN token 10947 for falcon)
    at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:306)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1124)
    ... 10 more
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1467203595416_0440 to YARN : Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:bigdata-next, Ident: (HDFS_DELEGATION_TOKEN token 10947 for falcon)
    at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:271)
    at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:291)
    at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:290)
    ... 25 more