Support Questions
Find answers, ask questions, and share your expertise

[RESOLVED] [FALCON] : failed to schedule a Feed

Explorer

Hi all,

I've got an error when i've tried to schedule a feed

Caused by: org.apache.falcon.FalconException: E0501 : E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie/clusterA@FTI.NET from IP XXXXXXX

Feed :

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="my-example-feed" description="my-example-feed" xmlns="uri:falcon:feed:0.1">
 <frequency>minutes(5)</frequency>
 <timezone>UTC</timezone>
 <clusters>
 <cluster name="next-rec-cluster" type="source">
 <validity start="2015-05-01T12:00Z" end="2099-12-31T23:00Z"/>
 <retention limit="months(9999)" action="delete"/>
 <locations>
 <location type="data" path="/tmp/falcon/"/>
 </locations>
 </cluster>
 <cluster name="current-rec-cluster" type="target">
 <validity start="2015-05-01T12:00Z" end="2099-12-31T23:00Z"/>
 <retention limit="months(9999)" action="delete"/>
 <locations>
 <location type="data" path="/tmp/falcon/"/>
 </locations>
 </cluster>
 </clusters>
 <locations>
 <location type="data" path="/tmp/falcon/"/>
 <location type="stats" path="/none"/>
 <location type="meta" path="/none"/>
 </locations>
 <ACL owner="falcon" group="hadoop" permission="0755"/>
 <schema location="/none" provider="none"/>
 <properties><property name="queueName" value="oozie-launcher"/></properties>
</feed>

Proxy user FALCON :

falcon.service.ProxyUserService.proxyuser.falcon.groups=*
falcon.service.ProxyUserService.proxyuser.falcon.hosts=*

HDFS :

hadoop.proxyuser.oozie.groups = * 
hadoop.proxyuser.oozie.hosts = *

hadoop.proxyuser.falcon.groups = * 
hadoop.proxyuser.falcon.hosts = *

OOZIE :

oozie.service.ProxyUserService.proxyuser.falcon.groups = *
oozie.service.ProxyUserService.proxyuser.falcon.hosts = *

FALCON application.log :

2016-05-17 10:39:46,071 INFO  - [888060389@qtp-1738446583-0 - a5bdcd78-a7f4-4fc0-8c3f-214e8bdfb091:falcon:POST//entities/schedule/feed/my-example-feed] ~ Memory lock released for (feed) my-example-fe
ed (AbstractSchedulableEntityManager:100)
2016-05-17 10:39:46,071 ERROR - [888060389@qtp-1738446583-0 - a5bdcd78-a7f4-4fc0-8c3f-214e8bdfb091:falcon:POST//entities/schedule/feed/my-example-feed] ~ Unable to schedule workflow (AbstractSchedula
bleEntityManager:76)
org.apache.falcon.FalconException: Entity schedule failed for feed: my-example-feed
        at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:96)
        at org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:73)
        at org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$9.doExecute(SchedulableEntityManagerProxy.java:403)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:577)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody12(SchedulableEntityManagerProxy.java:405)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure13.run(SchedulableEntityManagerProxy.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:388)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
:1        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:108)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:188)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:615)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:574)
        at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:197)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.HostnameFilter.doFilter(HostnameFilter.java:82)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.falcon.FalconException: E0501 : E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie/clusterA@FTI.NET from IP XXXXXXX
        at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:234)
        at org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:172)
        at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:94)
        ... 61 more
Caused by: E0501 : E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie/clusterA@FTI.NET from IP XXXXXXXXXXXX
        at org.apache.oozie.client.OozieClient.handleError(OozieClient.java:612)
        at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:695)
        at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:665)
        at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:560)
        at org.apache.oozie.client.OozieClient.dryrun(OozieClient.java:829)
        at org.apache.oozie.client.ProxyOozieClient.access$401(ProxyOozieClient.java:48)
        at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:177)
        at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:174)
        at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:244)
        at org.apache.oozie.client.ProxyOozieClient.dryrun(ProxyOozieClient.java:174)
        at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:232)
        ... 63 more
2016-05-17 10:39:46,073 ERROR - [888060389@qtp-1738446583-0 - a5bdcd78-a7f4-4fc0-8c3f-214e8bdfb091:falcon:POST//entities/schedule/feed/my-example-feed] ~ Action failed: Bad Request
Error: Entity schedule failed for feed: my-example-feed (FalconWebException:83)
2016-05-17 10:39:46,073 ERROR - [888060389@qtp-1738446583-0 - a5bdcd78-a7f4-4fc0-8c3f-214e8bdfb091:falcon:POST//entities/schedule/feed/my-example-feed] ~ Action failed: Bad Request
Error: default/org.apache.falcon.FalconWebException::org.apache.falcon.FalconException: Entity schedule failed for feed: my-example-feed
 (FalconWebException:83)
2016-05-17 10:39:46,074 INFO  - [888060389@qtp-1738446583-0 - a5bdcd78-a7f4-4fc0-8c3f-214e8bdfb091:falcon:POST//entities/schedule/feed/my-example-feed] ~ {Action:schedule, Dimensions:{entityType=feed
, colo=*, entityName=my-example-feed}, Status: FAILED, Time-taken:1332154913 ns} (METRIC:38)
2016-05-17 10:39:46,074 DEBUG - [888060389@qtp-1738446583-0 - a5bdcd78-a7f4-4fc0-8c3f-214e8bdfb091:] ~ Audit: falcon/XXXXXXXXX performed request http://clusterA:15000/api/entities/schedule/feed/my-ex
ample-feed?colo=* (XXXXXXXXXXXXXX) at time 2016-05-17T08:39Z (FalconAuditFilter:86)

11 REPLIES 11

Re: [RESOLVED] [FALCON] : failed to schedule a Feed

Rising Star

The root cause of this issue is that the user you are using to run the workflow does not have permission to read the workflow.xml. If you submitted the feed entity as user "ABC", try to schedule it as user "ABC" and not some other user who does not have permissions to read the workflow.

Re: [RESOLVED] [FALCON] : failed to schedule a Feed

Explorer

@Balu : thanks, but clusters entities, feed were created with falcon user. so i've tried schedule with falcon user.

as described in falcon documentation :

drwxrwxrwx   - falcon hadoop          0 2016-05-12 14:01 /apps/falcon/current-rec-cluster/staging
drwxr-xr-x   - falcon hadoop          0 2016-05-12 11:15 /apps/falcon/current-rec-cluster/working

drwxrwxr-x  - falcon hadoop  0 2016-05-12 14:01 /apps/falcon/current-rec-cluster/staging/falcon
drwxr-xr-x  - falcon hadoop  0 2016-05-17 11:10 /apps/falcon/current-rec-cluster/working/lib

User falcon can read the remote workflow.xml (from clusterA to clusterB) :

[falcon@clusterA logs]$ hadoop fs -cat hdfs://clusterB:8020/apps/falcon/current-rec-cluster/staging/falcon/workflows/feed/my-example-feed/a73ecbea81cf12de12df39702ad95d0f_1463066728543/REPLICATION/next-rec-cluster/workflow.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<workflow-app xmlns="uri:oozie:workflow:0.3" name="FALCON_FEED_REPLICATION_my-example-feed">
    <start to="replication"/>
    <action name="replication">
        <java>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>oozie.launcher.mapreduce.job.user.classpath.first</name>
                    <value>true</value>
                </property>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>oozie.launcher.mapred.job.priority</name>
                    <value>${jobPriority}</value>
                </property>
                <property>

Re: [RESOLVED] [FALCON] : failed to schedule a Feed

Rising Star

@mayki wogno : You did all the steps correct. The only possible explanation for you to continue seeing this error is that hdfs, yarn and oozie were not restarted after the changes were made to configs. If you did restart hdfs, yarn and oozie - I am not sure why you are continuing to see the problem.

Based on the logs, I am assuming you have enabled Kerberos on your cluster. I see that the job is scheduled from clusterA, but the workflow is on clusterB. Oozie user oozie/clusterA@EXAMPLE.COM is unable to connect to clusterB and read the workflow.xml. This seems like kerberos configuration issue.

Make sure, you have the config "oozie.authentication.kerberos.name.rules" set up correctly to handle these situations, and that kerberos server on clusterB can authenticate oozie/clusterA@EXAMPLE.COM

Re: [RESOLVED] [FALCON] : failed to schedule a Feed

Rising Star
@mayki wogno

Please share the relevant Oozie log for this workflow. This seems like auth issue on Oozie and not Falcon, and I will help you debug.

Re: [RESOLVED] [FALCON] : failed to schedule a Feed

Explorer

Here log from oozie , it seems a problem is impersonate user but i don't know why because all proxyusers are ok.

09:16:10,147  INFO SLACalculatorMemory:520 - SERVER[clusterA] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Running periodic SLA check
2016-05-18 09:16:13,168  INFO BaseJobServlet:520 - SERVER[clusterA] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] AuthorizationException
org.apache.oozie.service.AuthorizationException: E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie/clusterA@EXAMPLE.COM from IP XXXXXXXXXXX
        at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:406)
        at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:263)
        at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:99)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Unauthorized connection for super-user: oozie/clusterA@EXAMPLE.COM from IP XXXXXXXXXXX
        at org.apache.hadoop.ipc.Client.call(Client.java:1427)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1311)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
        at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:378)
        ... 26 more
2016-05-18 09:16:13,169  WARN V1JobsServlet:523 - SERVER[clusterA] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://clusterA:11000/oozie/v2/jobs?action=dryrun&doAs=falcon&user.name=falcon/clusterA@EXAMPLE.COM] error[E0501], E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie/clusterA@EXAMPLE.COM from IP XXXXXXXXXXX
org.apache.oozie.servlet.XServletException: E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie/clusterA@EXAMPLE.COM from IP XXXXXXXXXXX
        at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:271)
        at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:99)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.oozie.service.AuthorizationException: E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie/clusterA@EXAMPLE.COM from IP XXXXXXXXXXX
        at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:406)
        at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:263)
        ... 25 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Unauthorized connection for super-user: oozie/clusterA@EXAMPLE.COM from IP XXXXXXXXXXX
        at org.apache.hadoop.ipc.Client.call(Client.java:1427)
        at org.apache.hadoop.ipc.Client.call(Client.java:1358)

Re: [RESOLVED] [FALCON] : failed to schedule a Feed

Explorer

@balu : it is weird, all authentication and authorization are set.

Here oozie from clusterA enable to read workflow on HDFS clusterB

[oozie@clusterA ~]$ hdfs dfs -cat hdfs://clusterB:8020/apps/falcon/current-rec-cluster/staging/falcon/workflows/feed/my-example-feed/a73ecbea81cf12de12df39702ad95d0f_1463736136013/REPLICATION/next-rec-cluster/workflow.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<workflow-app xmlns="uri:oozie:workflow:0.3" name="FALCON_FEED_REPLICATION_my-example-feed">
    <start to="replication"/>
    <action name="replication">
        <java>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>oozie.launcher.mapreduce.job.user.classpath.first</name>
                    <value>true</value>
                </property>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>oozie.launcher.mapred.job.priority</name>
                    <value>${jobPriority}</value>
                </property>
                <property>
                    <name>oozie.action.sharelib.for.java</name>
                    <value>distcp</value>
                </property>
                <property>
                    <name>oozie.launcher.oozie.libpath</name>
                    <value>${wf:conf("falcon.libpath")}</value>
                </property>

It is exact what workflow is schedule from clusterA to clusterB. How can I scheduled the workflow on clusterA but keep clusterB as target in the flow ?

Re: [RESOLVED] [FALCON] : failed to schedule a Feed

Explorer

@Balu: do you think that the UID oozie, falcon user need to be the same on clusterA and clusterB ?

Re: [RESOLVED] [FALCON] : failed to schedule a Feed

Explorer

Some news logs, but i don't know where's looking .. 😞

here targer cluster and log when i want to submit

<cluster xmlns='uri:falcon:cluster:0.1' name='current-rec-cluster' description='undefined' colo='nextRecColo'>
  <interfaces>
    <interface type='readonly' endpoint='hftp://clusterB:50070' version='2.2.0'/>
    <interface type='write' endpoint='hdfs://clusterB' version='2.2.0'/>
    <interface type='execute' endpoint='clusterB:8050' version='2.2.0'/>
    <interface type='workflow' endpoint='http://clusterB:11000/oozie/' version='4.0.0'/>
    <interface type='messaging' endpoint='tcp://clusterA:61616?daemon=true' version='5.1.6'/>

  </interfaces>
  <locations>
    <location name='staging' path='/apps/falcon/current-rec-cluster/staging'/>
    <location name='temp' path='/apps/falcon/tmp'/>
    <location name='working' path='/apps/falcon/current-rec-cluster/working'/>
  </locations>
  <ACL owner='falcon' group='hadoop' permission='0755'/>
  <properties>
    <property name='dfs.namenode.kerberos.principal' value='nn/_HOST@FTI.NET'/>
    <property name='hive.metastore.kerberos.principal' value='hive/_HOST@FTI.NET'/>
    <property name="hive.metastore.uris" value="thrift://clusterB:9083"/>
    <property name='queueName' value='oozie-launcher'/>
    <property name="hive.metastore.sasl.enabled" value="true"/>

  </properties>
</cluster>
~


2016-05-20 17:05:51,752 INFO  - [1222181367@qtp-1738446583-6 - efbbf7f7-b1a1-47f9-9fa1-cc8af8348ac0:falcon:POST//entities/schedule/feed/replication-feed] ~ dryRun with properties {user.name=falcon, oozie.coord.application.path=${nameNode}/apps/falcon/current-rec-cluster/staging/falcon/workflows/feed/replication-feed/7b0cae7c8eb435ebb1615700d8e6cfaa_1463756750973/RETENTION/coordinator.xml, ENTITY_NAME=FALCON_FEED_RETENTION_replication-feed, ENTITY_PATH=/apps/falcon/current-rec-cluster/staging/falcon/workflows/feed/replication-feed/7b0cae7c8eb435ebb1615700d8e6cfaa_1463756750973/RETENTION/coordinator.xml, nameNode=hdfs://clusterB} (OozieWorkflowEngine:231)
2016-05-20 17:05:51,908 INFO  - [1222181367@qtp-1738446583-6 - efbbf7f7-b1a1-47f9-9fa1-cc8af8348ac0:falcon:POST//entities/schedule/feed/replication-feed] ~ dryRun with properties {jobTracker=clusterB:8050, user.name=falcon, userWorkflowName=replication-policy, oozie.coord.application.path=${nameNode}/apps/falcon/current-rec-cluster/staging/falcon/workflows/feed/replication-feed/7b0cae7c8eb435ebb1615700d8e6cfaa_1463756750973/REPLICATION/next-rec-cluster/coordinator.xml, maxMaps=5, queueName=oozie-launcher, workflowEngineUrl=http://clusterB:11000/oozie/, hive.metastore.uris=thrift://clusterB:9083, falcon.libpath=/apps/falcon/current-rec-cluster/working/lib, jobPriority=NORMAL, dfs.namenode.kerberos.principal=nn/_HOST@FTI.NET, ENTITY_PATH=/apps/falcon/current-rec-cluster/staging/falcon/workflows/feed/replication-feed/7b0cae7c8eb435ebb1615700d8e6cfaa_1463756750973/REPLICATION/next-rec-cluster/coordinator.xml, brokerImplClass=org.apache.activemq.ActiveMQConnectionFactory, entityName=replication-feed, userBrokerUrl=tcp://clusterA:61616?daemon=true, colo.name=nextRecColo, hive.metastore.sasl.enabled=true, oozie.use.system.libpath=true, entityType=FEED, mapBandwidth=100, ENTITY_NAME=FALCON_FEED_REPLICATION_replication-feed_next-rec-cluster, hive.metastore.kerberos.principal=hive/_HOST@FTI.NET, userBrokerImplClass=org.apache.activemq.ActiveMQConnectionFactory, brokerTTL=4320, availabilityFlag=NA, cluster=current-rec-cluster, nameNode=hdfs://clusterB, brokerUrl=tcp://clusterA:61616, falconDataOperation=REPLICATE, shouldRecord=false, logDir=${nameNode}/apps/falcon/current-rec-cluster/staging/falcon/workflows/feed/replication-feed/logs, userWorkflowVersion=0.6.1.2.3.4.0-3485-r85c561b36ee49153a0757076444ee3755bd41ea6, userWorkflowEngine=falcon} (OozieWorkflowEngine:231)
2016-05-20 17:05:52,074 INFO  - [1222181367@qtp-1738446583-6 - efbbf7f7-b1a1-47f9-9fa1-cc8af8348ac0:falcon:POST//entities/schedule/feed/replication-feed] ~ Successfully released lock for (feed) replication-feed by 1222181367@qtp-1738446583-6 - efbbf7f7-b1a1-47f9-9fa1-cc8af8348ac0 (MemoryLocks:70)
2016-05-20 17:05:52,075 INFO  - [1222181367@qtp-1738446583-6 - efbbf7f7-b1a1-47f9-9fa1-cc8af8348ac0:falcon:POST//entities/schedule/feed/replication-feed] ~ Memory lock released for (feed) replication-feed (AbstractSchedulableEntityManager:100)
2016-05-20 17:05:52,075 ERROR - [1222181367@qtp-1738446583-6 - efbbf7f7-b1a1-47f9-9fa1-cc8af8348ac0:falcon:POST//entities/schedule/feed/replication-feed] ~ Unable to schedule workflow (AbstractSchedulableEntityManager:76)
org.apache.falcon.FalconException: Entity schedule failed for feed: replication-feed
        at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:96)
        at org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:73)
        at org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$9.doExecute(SchedulableEntityManagerProxy.java:403)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:577)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody12(SchedulableEntityManagerProxy.java:405)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure13.run(SchedulableEntityManagerProxy.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:388)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:108)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:188)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:615)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:574)
        at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:197)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.HostnameFilter.doFilter(HostnameFilter.java:82)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.falcon.FalconException: E0803 : E0803: IO error, null
        at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:234)
        at org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:172)
        at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:94)
        ... 61 more
Caused by: E0803 : E0803: IO error, null
        at org.apache.oozie.client.OozieClient.handleError(OozieClient.java:612)
        at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:695)
        at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:665)
        at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:560)
        at org.apache.oozie.client.OozieClient.dryrun(OozieClient.java:829)
        at org.apache.oozie.client.ProxyOozieClient.access$401(ProxyOozieClient.java:48)
        at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:177)
        at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:174)
        at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:244)
        at org.apache.oozie.client.ProxyOozieClient.dryrun(ProxyOozieClient.java:174)
        at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:232)
        ... 63 more
2016-05-20 17:05:52,078 ERROR - [1222181367@qtp-1738446583-6 - efbbf7f7-b1a1-47f9-9fa1-cc8af8348ac0:falcon:POST//entities/schedule/feed/replication-feed] ~ Action failed: Bad Request
Error: Entity schedule failed for feed: replication-feed (FalconWebException:83)
2016-05-20 17:05:52,078 ERROR - [1222181367@qtp-1738446583-6 - efbbf7f7-b1a1-47f9-9fa1-cc8af8348ac0:falcon:POST//entities/schedule/feed/replication-feed] ~ Action failed: Bad Request
Error: default/org.apache.falcon.FalconWebException::org.apache.falcon.FalconException: Entity schedule failed for feed: replication-feed
 (FalconWebException:83)
2016-05-20 17:05:52,079 INFO  - [1222181367@qtp-1738446583-6 - efbbf7f7-b1a1-47f9-9fa1-cc8af8348ac0:falcon:POST//entities/schedule/feed/replication-feed] ~ {Action:schedule, Dimensions:{entityType=feed, colo=*, entityName=replication-feed}, Status: FAILED, Time-taken:1222303023 ns} (METRIC:38)
2016-05-20 17:05:52,080 DEBUG - [1222181367@qtp-1738446583-6 - efbbf7f7-b1a1-47f9-9fa1-cc8af8348ac0:] ~ Audit: falcon/XX.XX.XX.XX performed request http://clusterA:15000/api/entities/schedule/feed/replication-feed?colo=* (XX.XX.XX.XX) at time 2016-05-20T15:05Z (FalconAuditFilter:86)
2016-05-20 17:07:37,952 INFO  - [Thread-11:] ~ config.location is set, using: /usr/hdp/current/falcon-server/conf/runtime.properties (ApplicationProperties:108)
2016-05-20 17:07:37,952 INFO  - [Thread-11:] ~ Initializing org.apache.falcon.util.RuntimeProperties properties with domain falcon (ApplicationProperties:145)
2016-05-20 17:07:37,953 DEBUG - [Thread-11:] ~ log.cleanup.frequency.hours.retention=minutes(1) (ApplicationProperties:151)
2016-05-20 17:07:37,953 DEBUG - [Thread-11:] ~ log.cleanup.frequency.months.retention=months(3) (ApplicationProperties:151)
2016-05-20 17:07:37,953 DEBUG - [Thread-11:] ~ log.cleanup.frequency.minutes.retention=hours(6) (ApplicationProperties:151)

Re: [RESOLVED] [FALCON] : failed to schedule a Feed

Super Guru

Have any components been updated and not restarted?