question Re: Problem while Mirroring Dataset between hadoop cluster using Apache Falcon in Archives of Support Questions (Read Only)

Problem while Mirroring Dataset between hadoop cluster using Apache Falcon

mrakki3110 — Thu, 24 Dec 2015 17:30:35 GMT

Hello I am new to Apache Falcon & I am mirroring a dataset as shown here Mirroring Dataset between Hadoop Clusters using Apache Falcon ...... I saw the logs of Apache Falcon in "falcon.application.log" & it is throwing this exception ::

org.apache.falcon.FalconException: Entity schedule failed for PROCESS: MirrorTest
	at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:96)
	at org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:73)
	at org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:131)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49)
	at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$9.doExecute(SchedulableEntityManagerProxy.java:403)
	at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:575)
	at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody12(SchedulableEntityManagerProxy.java:405)
	at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure13.run(SchedulableEntityManagerProxy.java:1)
	at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
	at org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)
	at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:388)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
	at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:106)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:184)
	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:595)
	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:554)
	at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:193)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)


Caused by: org.apache.falcon.FalconException: E0501 : E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie 
	at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:234)
	at org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:172)
	at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:94)


Am I missing any configuration for oozie or falcon in any configuration files ?? And also how should we deal with this following exception ??

Re: Problem while Mirroring Dataset between hadoop cluster using Apache Falcon

nsabharwal — Thu, 24 Dec 2015 20:57:02 GMT

@Akshay Shingote

Causedby: org.apache.falcon.FalconException: E0501 : E0501:Couldnot perform authorization operation,Unauthorized connection forsuper-user: oozie

Check oozie proxyuser settings.

If more lax security is preferred, the wildcard value * may be used to allow impersonation from any host or of any user. For example, by specifying as below in core-site.xml, user named oozieaccessing from any host can impersonate any user belonging to any group.

  <property>
    <name>hadoop.proxyuser.oozie.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.oozie.groups</name>
    <value>*</value>
  </property>

Re: Problem while Mirroring Dataset between hadoop cluster using Apache Falcon

mrakki3110 — Mon, 04 Jan 2016 17:09:58 GMT

@Neeraj Sabharwal

:Can you please specify or mention how should my oozie-site.xml & core-site.xml be ?? Do I need to make any changes in oozie-site.xml for Falcon ??

Re: Problem while Mirroring Dataset between hadoop cluster using Apache Falcon

bsaini — Thu, 14 Jan 2016 21:32:13 GMT

@Akshay Shingote See this question. This issue is not caused by how your workflow.xml is configured but the permissions on it.

The root cause of this issue is that the user you are using to run the workflow does not have permission to read the workflow.xml. Change the permissions on workflow.xml to 777 or 755 and try again.

Also make sure that the directories (absolute path) that contains the workflow.xml also has at least 755, so that the user is able to get to the file and then read it

Here is the method that is generating this error.

Re: Problem while Mirroring Dataset between hadoop cluster using Apache Falcon

nsabharwal — Mon, 19 Aug 2019 12:25:46 GMT

@Akshay Shingote

You should use ambari to manage those files.

Re: Problem while Mirroring Dataset between hadoop cluster using Apache Falcon

aervits — Tue, 02 Feb 2016 22:30:36 GMT

@Akshay Shingote has this been resolved? Can you post your solution or accept best answer?

Re: Problem while Mirroring Dataset between hadoop cluster using Apache Falcon

mrakki3110 — Thu, 04 Feb 2016 12:50:43 GMT

Yes, this has been resolved...thanks for the help....The solution which you said that the permissions to run workflow should be 777 or 755 that does work....also apart from that the permissions & ownership to the path /apps/data-mirroring/ should be 777/755 or & its ownership should be granted to falcon....there is another location where workflow.xml gets defined i.e. at location /apps/falcon/backupCluster/staging/falcon/workflows/process/MirrorTest ,so its permissions should be either 777 or 755

Re: Problem while Mirroring Dataset between hadoop cluster using Apache Falcon

arthur_grevin — Wed, 10 Feb 2016 22:05:55 GMT

Hi,

I set the permission to 777 to the workflow.xml but i still have the same error. Is there something else to do to start the mirroring job.

drwxrwxrwx - falcon users 0 2016-02-09 15:06 /apps/data-mirroring/workflows

-rwxrwxrwx 3 falcon users 5010 2016-02-09 15:06 /apps/data-mirroring/workflows/hdfs-replication-workflow.xml

org.apache.falcon.FalconException: Entity schedule failed for PROCESS: MirrorTest

Re: Problem while Mirroring Dataset between hadoop cluster using Apache Falcon

mrakki3110 — Fri, 12 Feb 2016 13:35:18 GMT

you did right by setting the permission to 777 for /apps/data-mirroring/workflows/* but apart from that when you create the MirrorTest, a directory in the name of MirrorTest gets created at this location where the workflow of MirrorTest i.e.workflow.xml gets generated at location /apps/falcon/backupCluster/staging/falcon/workflows/process/MirrorTest .Same applies for primaryCluster also.. Now make sure that the path from staging/falcon/workflows/process/MirrorTest should be set as 777/755 so that the workflow.xml has permissions 777 & ownership=ambari-qa

Re: Problem while Mirroring Dataset between hadoop cluster using Apache Falcon

vtpcnk — Thu, 09 Mar 2017 21:55:02 GMT

Hi, I am having similar issue with falcon replication not working. The permissions, ownership, oozie wildcard are all as per this page in my environment. But still replication is not going through. Any insights?