Member since
03-21-2017
4
Posts
0
Kudos Received
0
Solutions
02-09-2019
06:42 PM
Hi Natus, thanks for your reply. Spark would be good idea to batch load periodically. We are looking for an near real time/online solution.
... View more
01-04-2019
09:01 PM
HBase Indexer from HDPSearch helps push all updates in HBase to Solr. With recent release of HDPSearch 4.0 does not seem to support HBase Indexer anymore. We have huge data in HBase that needs to be indexed. We are mainly looking for a way to index data in HBase near real time. If not HDPSearch, what are the other options to index data in HBase 2.0?
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Solr
04-19-2017
09:02 PM
I have setup Hive mirror using falcon between two clusters. I have configured this job to run on destination cluster as source cluster doesn't have oozie or falcon installed. The job would get submitted successfully but, it would fail during scheduling it. I have configured oozie as proxy user for both groups and hosts in destination cluster(where the job is running). Anyone dealt with this kind of issue before? Thanks in advance. org.apache.falcon.FalconException: Entity schedule failed for PROCESS: hivemirrortest
at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:96)
at org.apache.falcon.resource.AbstractSchedulableEntityManager.schedule(AbstractSchedulableEntityManager.java:73)
at org.apache.falcon.resource.SchedulableEntityManager.schedule(SchedulableEntityManager.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$9.doExecute(SchedulableEntityManagerProxy.java:403)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:577)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule_aroundBody12(SchedulableEntityManagerProxy.java:405)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure13.run(SchedulableEntityManagerProxy.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)
at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.schedule(SchedulableEntityManagerProxy.java:388)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:108)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:188)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:615)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:574)
at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:197)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.falcon.security.HostnameFilter.doFilter(HostnameFilter.java:82)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.falcon.FalconException: E0501 : E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie from IP X.X.X.X(ip address of falcon & oozie at destination cluster)
at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:234)
at org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:172)
at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:94)
... 61 more
Caused by: E0501 : E0501: Could not perform authorization operation, Unauthorized connection for super-user: oozie from IP X.X.X.X(ip address of falcon&oozie at destination cluster)
at org.apache.oozie.client.OozieClient.handleError(OozieClient.java:612)
at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:695)
at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:665)
at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:560)
at org.apache.oozie.client.OozieClient.dryrun(OozieClient.java:829)
at org.apache.oozie.client.ProxyOozieClient.access$401(ProxyOozieClient.java:48)
at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:177)
at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:174)
at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:244)
at org.apache.oozie.client.ProxyOozieClient.dryrun(ProxyOozieClient.java:174)
at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:232)
... 63 more
... View more
Labels:
03-21-2017
10:27 PM
Hi, We are planning to create a new cluster intended for different set of users. Currently we have data coming into our existing cluster in multiple ways. Mainly through Kafka and other batch jobs pushing data in regular intervals. Also, data can be modified after replicated to new cluster. In this case, we want copy only delta instead of copy everything. What would be an ideal tool for this job? Thanks in Advance!
... View more
Labels:
- Labels:
-
Apache Falcon
-
Apache Hadoop
-
Apache NiFi