Support Questions
Find answers, ask questions, and share your expertise

[RESOLVED] [FALCON] : failed to schedule a Feed

Explorer

Hi all,

I've got an error when i've tried to schedule a feed

Caused by: org.apache.falcon.FalconException: E0501 : E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie/clusterA@FTI.NET from IP XXXXXXX

Feed :

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="my-example-feed" description="my-example-feed" xmlns="uri:falcon:feed:0.1">
 <frequency>minutes(5)</frequency>
 <timezone>UTC</timezone>
 <clusters>
 <cluster name="next-rec-cluster" type="source">
 <validity start="2015-05-01T12:00Z" end="2099-12-31T23:00Z"/>
 <retention limit="months(9999)" action="delete"/>
 <locations>
 <location type="data" path="/tmp/falcon/"/>
 </locations>
 </cluster>
 <cluster name="current-rec-cluster" type="target">
 <validity start="2015-05-01T12:00Z" end="2099-12-31T23:00Z"/>
 <retention limit="months(9999)" action="delete"/>
 <locations>
 <location type="data" path="/tmp/falcon/"/>
 </locations>
 </cluster>
 </clusters>
 <locations>
 <location type="data" path="/tmp/falcon/"/>
 <location type="stats" path="/none"/>
 <location type="meta" path="/none"/>
 </locations>
 <ACL owner="falcon" group="hadoop" permission="0755"/>
 <schema location="/none" provider="none"/>
 <properties><property name="queueName" value="oozie-launcher"/></properties>
</feed>

Proxy user FALCON :

falcon.service.ProxyUserService.proxyuser.falcon.groups=*
falcon.service.ProxyUserService.proxyuser.falcon.hosts=*

HDFS :

hadoop.proxyuser.oozie.groups = * 
hadoop.proxyuser.oozie.hosts = *

hadoop.proxyuser.falcon.groups = * 
hadoop.proxyuser.falcon.hosts = *

OOZIE :

oozie.service.ProxyUserService.proxyuser.falcon.groups = *
oozie.service.ProxyUserService.proxyuser.falcon.hosts = *

FALCON application.log :

2016-05-17 10:39:46,071 INFO  - [888060389@qtp-1738446583-0 - a5bdcd78-a7f4-4fc0-8c3f-214e8bdfb091:falcon:POST//entities/schedule/feed/my-example-feed] ~ Memory lock released for (feed) my-example-fe
ed (AbstractSchedulableEntityManager:100)
2016-05-17 10:39:46,071 ERROR - [888060389@qtp-1738446583-0 - a5bdcd78-a7f4-4fc0-8c3f-214e8bdfb091:falcon:POST//entities/schedule/feed/my-example-feed] ~ Unable to schedule workflow (AbstractSchedula
bleEntityManager:76)
org.apache.falcon.FalconException: Entity schedule failed for feed: my-example-feed
        at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:96)
        at org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:73)
        at org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$9.doExecute(SchedulableEntityManagerProxy.java:403)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:577)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody12(SchedulableEntityManagerProxy.java:405)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure13.run(SchedulableEntityManagerProxy.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:388)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
:1        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:108)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:188)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:615)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:574)
        at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:197)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.HostnameFilter.doFilter(HostnameFilter.java:82)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.falcon.FalconException: E0501 : E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie/clusterA@FTI.NET from IP XXXXXXX
        at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:234)
        at org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:172)
        at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:94)
        ... 61 more
Caused by: E0501 : E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie/clusterA@FTI.NET from IP XXXXXXXXXXXX
        at org.apache.oozie.client.OozieClient.handleError(OozieClient.java:612)
        at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:695)
        at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:665)
        at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:560)
        at org.apache.oozie.client.OozieClient.dryrun(OozieClient.java:829)
        at org.apache.oozie.client.ProxyOozieClient.access$401(ProxyOozieClient.java:48)
        at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:177)
        at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:174)
        at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:244)
        at org.apache.oozie.client.ProxyOozieClient.dryrun(ProxyOozieClient.java:174)
        at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:232)
        ... 63 more
2016-05-17 10:39:46,073 ERROR - [888060389@qtp-1738446583-0 - a5bdcd78-a7f4-4fc0-8c3f-214e8bdfb091:falcon:POST//entities/schedule/feed/my-example-feed] ~ Action failed: Bad Request
Error: Entity schedule failed for feed: my-example-feed (FalconWebException:83)
2016-05-17 10:39:46,073 ERROR - [888060389@qtp-1738446583-0 - a5bdcd78-a7f4-4fc0-8c3f-214e8bdfb091:falcon:POST//entities/schedule/feed/my-example-feed] ~ Action failed: Bad Request
Error: default/org.apache.falcon.FalconWebException::org.apache.falcon.FalconException: Entity schedule failed for feed: my-example-feed
 (FalconWebException:83)
2016-05-17 10:39:46,074 INFO  - [888060389@qtp-1738446583-0 - a5bdcd78-a7f4-4fc0-8c3f-214e8bdfb091:falcon:POST//entities/schedule/feed/my-example-feed] ~ {Action:schedule, Dimensions:{entityType=feed
, colo=*, entityName=my-example-feed}, Status: FAILED, Time-taken:1332154913 ns} (METRIC:38)
2016-05-17 10:39:46,074 DEBUG - [888060389@qtp-1738446583-0 - a5bdcd78-a7f4-4fc0-8c3f-214e8bdfb091:] ~ Audit: falcon/XXXXXXXXX performed request http://clusterA:15000/api/entities/schedule/feed/my-ex
ample-feed?colo=* (XXXXXXXXXXXXXX) at time 2016-05-17T08:39Z (FalconAuditFilter:86)

11 REPLIES 11

Explorer

Only all components of clusterA have restarted, falcon server is on clusterA.

Explorer

I found the bug when schedule the feed, the option -skipDryRun must be set.

[falcon@clusterA ~]$ falcon entity -type feed -submit -file replication-next-current.xml
log4j:WARN No appenders could be found for logger (org.apache.hadoop.security.authentication.client.KerberosAuthenticator).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
falcon/default/Submit successful (feed) replication-next-current

[falcon@clusterA ~]$ falcon entity  -type feed -schedule -name replication-next-current
log4j:WARN No appenders could be found for logger (org.apache.hadoop.security.authentication.client.KerberosAuthenticator).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
ERROR: Bad Request;default/org.apache.falcon.FalconWebException::org.apache.falcon.FalconException: Entity schedule failed for feed: replication-next-current

[falcon@clusterA ~]$ falcon entity  -type feed -schedule -name replication-next-current -skipDryRun
log4j:WARN No appenders could be found for logger (org.apache.hadoop.security.authentication.client.KerberosAuthenticator).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
default/replication-next-current(feed) scheduled successfully

But I've another issue :

On TargetCluester (clusterB), the workflow has failed due to permission denied :

          Parent FALCON_FEED_RETENTION_replication-next-current :
        Workflow FALCON_FEED_RETENTION_replication-next-current
      GraphActionsDetailsConfigurationLogDefinition
            
              0009963-160510161955685-oozie-oozi-W@eviction
           
            START_MANUAL
            JA009             JA009: Permission denied: user=falcon, access=WRITE, inode="/user/falcon/oozie-oozi/0009963-160510161955685-oozie-oozi-W/eviction--java.tmp":hdfs:hdfs:drwxr-xr-x
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213)
	at org.apache.hadoop.hdfs.serv

Here the right permission on /usr/falcon (clusterA) :

drwxrwxr-x   - falcon hadoop          0 2016-05-24 15:02 /user/falcon/oozie-oozi

drwxrwxr-x   - falcon hadoop          0 2016-05-20 11:56 /user/falcon/oozie-oozi/0000000-160520114312397-oozie-oozi-W
drwxrwxr-x   - falcon hadoop          0 2016-05-24 15:02 /user/falcon/oozie-oozi/0000340-160520114312397-oozie-oozi-W
drwxrwxr-x   - falcon hadoop          0 2016-05-24 14:19 /user/falcon/oozie-oozi/0009847-160510161955685-oozie-oozi-W

the hdfs user can write on /user/falcon from clusterB :

hdfs dfs -touchz  hdfs://clusterA:8020/user/falcon/oozie-oozi/0009847-160510161955685-oozie-oozi-W/test
[hdfs@clusterB ~]$ hdfs dfs -ls  hdfs://clusterA:8020/user/falcon/oozie-oozi/*/*

-rw-rw-r--   3 hdfs   hadoop          0 2016-05-24 14:19 hdfs://clusterA:8020/user/falcon/oozie-oozi/0009847-160510161955685-oozie-oozi-W/test