Problem while Mirroring Dataset between hadoop cluster using Apache Falcon

[#1]

Hello I am new to Apache Falcon & I am mirroring a dataset as shown here Mirroring Dataset between Hadoop Clusters using Apache Falcon ...... I saw the logs of Apache Falcon in "falcon.application.log" & it is throwing this exception ::

org.apache.falcon.FalconException: Entity schedule failed for PROCESS: MirrorTest
	at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:96)
	at org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:73)
	at org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:131)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49)
	at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$9.doExecute(SchedulableEntityManagerProxy.java:403)
	at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:575)
	at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody12(SchedulableEntityManagerProxy.java:405)
	at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure13.run(SchedulableEntityManagerProxy.java:1)
	at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
	at org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)
	at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:388)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
	at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:106)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:184)
	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:595)
	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:554)
	at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:193)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)


Caused by: org.apache.falcon.FalconException: E0501 : E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie 
	at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:234)
	at org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:172)
	at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:94)


Am I missing any configuration for oozie or falcon in any configuration files ?? And also how should we deal with this following exception ?? 

[#2]

@Akshay Shingote

Causedby: org.apache.falcon.FalconException: E0501 : E0501:Couldnot perform authorization operation,Unauthorized connection forsuper-user: oozie

Check oozie proxyuser settings.

If more lax security is preferred, the wildcard value * may be used to allow impersonation from any host or of any user. For example, by specifying as below in core-site.xml, user named oozieaccessing from any host can impersonate any user belonging to any group.

  <property>
    <name>hadoop.proxyuser.oozie.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.oozie.groups</name>
    <value>*</value>
  </property>

[#3]

@Neeraj Sabharwal

:Can you please specify or mention how should my oozie-site.xml & core-site.xml be ?? Do I need to make any changes in oozie-site.xml for Falcon ??

[#4]

@Akshay Shingote

You should use ambari to manage those files.

[#5 · Accepted solution]

@Akshay Shingote See this question. This issue is not caused by how your workflow.xml is configured but the permissions on it.

The root cause of this issue is that the user you are using to run the workflow does not have permission to read the workflow.xml. Change the permissions on workflow.xml to 777 or 755 and try again.

Also make sure that the directories (absolute path) that contains the workflow.xml also has at least 755, so that the user is able to get to the file and then read it

Here is the method that is generating this error.

[#6]

Hi,

I set the permission to 777 to the workflow.xml but i still have the same error. Is there something else to do to start the mirroring job.

drwxrwxrwx - falcon users 0 2016-02-09 15:06 /apps/data-mirroring/workflows

-rwxrwxrwx 3 falcon users 5010 2016-02-09 15:06 /apps/data-mirroring/workflows/hdfs-replication-workflow.xml

org.apache.falcon.FalconException: Entity schedule failed for PROCESS: MirrorTest

[#7]

you did right by setting the permission to 777 for /apps/data-mirroring/workflows/* but apart from that when you create the MirrorTest, a directory in the name of MirrorTest gets created at this location where the workflow of MirrorTest i.e.workflow.xml gets generated at location /apps/falcon/backupCluster/staging/falcon/workflows/process/MirrorTest .Same applies for primaryCluster also.. Now make sure that the path from staging/falcon/workflows/process/MirrorTest should be set as 777/755 so that the workflow.xml has permissions 777 & ownership=ambari-qa

[#8]

@Akshay Shingote has this been resolved? Can you post your solution or accept best answer?

[#9]

Yes, this has been resolved...thanks for the help....The solution which you said that the permissions to run workflow should be 777 or 755 that does work....also apart from that the permissions & ownership to the path /apps/data-mirroring/ should be 777/755 or & its ownership should be granted to falcon....there is another location where workflow.xml gets defined i.e. at location /apps/falcon/backupCluster/staging/falcon/workflows/process/MirrorTest ,so its permissions should be either 777 or 755

[#10]

Hi, I am having similar issue with falcon replication not working. The permissions, ownership, oozie wildcard are all as per this page in my environment. But still replication is not going through. Any insights?