Support Questions
Find answers, ask questions, and share your expertise

[FALCON] cluster entity creation with HA cluster

Explorer

Hi all,

Why it is unable to submit a cluster entity with the name for cluster HDFS instead of hostname ?

this xml file is ok, submit successfully :

<cluster xmlns='uri:falcon:cluster:0.1' name='next-rec-cluster-bigdata' description='undefined' colo='nextRecColo'>
  <interfaces>
    <interface type='readonly' endpoint='hftp://server01:50070' version='2.2.0'/>
    <interface type='write' endpoint='hdfs://bigdata-next:8020' version='2.2.0'/>
    <interface type='execute' endpoint='server01:8050' version='2.2.0'/>
    <interface type='workflow' endpoint='http://server01:11000/oozie/' version='4.0.0'/>
  </interfaces>
  <locations>
    <location name='staging' path='/apps/falcon/next-rec-cluster/staging'/>
    <location name='temp' path='/apps/falcon/tmp'/>
    <location name='working' path='/apps/falcon/next-rec-cluster/working'/>
  </locations>
  <ACL owner='falcon' group='hadoop' permission='0755'/>
</cluster>

and here it is unable to submit :

<cluster xmlns='uri:falcon:cluster:0.1' name='next-rec-cluster-bigdata' description='undefined' colo='nextRecColo'>
  <interfaces>
    <interface type='readonly' endpoint='hftp://bigdata:50070' version='2.2.0'/>
    <interface type='write' endpoint='hdfs://bigdata:8020' version='2.2.0'/>
    <interface type='execute' endpoint='server01:8050' version='2.2.0'/>
    <interface type='workflow' endpoint='http://server01:11000/oozie/' version='4.0.0'/>
  </interfaces>
  <locations>
    <location name='staging' path='/apps/falcon/next-rec-cluster/staging'/>
    <location name='temp' path='/apps/falcon/tmp'/>
    <location name='working' path='/apps/falcon/next-rec-cluster/working'/>
  </locations>
  <ACL owner='falcon' group='hadoop' permission='0755'/>
</cluster>
28 REPLIES 28

Re: [FALCON] cluster entity creation with HA cluster

Expert Contributor

@mayki wogno Can you please share the exception message. Also please attach the Falcon application log.

Re: [FALCON] cluster entity creation with HA cluster

Explorer

@peeyush here exception and log:

it seems what "cluster hadoop name" could not be use

2016-05-31 09:03:02,663 ERROR - [307630071@qtp-794075965-25 - 3e184491-93f8-4fc4-afa7-b5c104c6ad64:falcon:POST//entities/submit/cluster] ~ Unable to persist entity object (AbstractEntityManager:198)
java.lang.IllegalArgumentException: java.net.UnknownHostException: bigdata
        at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:406)
        at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:424)
        at org.apache.hadoop.hdfs.web.HftpFileSystem.initialize(HftpFileSystem.java:191)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
        at org.apache.falcon.hadoop.HadoopClientFactory.createFileSystem(HadoopClientFactory.java:193)
        at org.apache.falcon.hadoop.HadoopClientFactory.createProxiedFileSystem(HadoopClientFactory.java:153)
        at org.apache.falcon.hadoop.HadoopClientFactory.createProxiedFileSystem(HadoopClientFactory.java:127)
        at org.apache.falcon.entity.parser.ClusterEntityParser.validateFileSystem(ClusterEntityParser.java:136)
        at org.apache.falcon.entity.parser.ClusterEntityParser.validateReadInterface(ClusterEntityParser.java:112)
        at org.apache.falcon.entity.parser.ClusterEntityParser.validate(ClusterEntityParser.java:84)
        at org.apache.falcon.entity.parser.ClusterEntityParser.validate(ClusterEntityParser.java:55)
        at org.apache.falcon.resource.AbstractEntityManager.validate(AbstractEntityManager.java:490)
        at org.apache.falcon.resource.AbstractEntityManager.submitInternal(AbstractEntityManager.java:413)
        at org.apache.falcon.resource.AbstractEntityManager.submit(AbstractEntityManager.java:195)
        at org.apache.falcon.resource.ConfigSyncService.submit(ConfigSyncService.java:50)
        at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$1.doExecute(SchedulableEntityManagerProxy.java:139)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:577)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit_aroundBody0(SchedulableEntityManagerProxy.java:141)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure1.run(SchedulableEntityManagerProxy.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit(SchedulableEntityManagerProxy.java:119)
        at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:108)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:188)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:615)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:574)
        at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:197)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.HostnameFilter.doFilter(HostnameFilter.java:82)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.net.UnknownHostException: bigdata
        ... 76 more
2016-05-31 09:03:02,664 ERROR - [307630071@qtp-794075965-25 - 3e184491-93f8-4fc4-afa7-b5c104c6ad64:falcon:POST//entities/submit/cluster] ~ Action failed: Bad Request
Error: java.net.UnknownHostException: bigdata (FalconWebException:83)
2016-05-31 09:03:02,665 ERROR - [307630071@qtp-794075965-25 - 3e184491-93f8-4fc4-afa7-b5c104c6ad64:falcon:POST//entities/submit/cluster] ~ Action failed: Bad Request
Error: default/org.apache.falcon.FalconWebException::org.apache.falcon.FalconException: java.net.UnknownHostException: bigdata
 (FalconWebException:83)
2016-05-31 09:03:02,665 INFO  - [307630071@qtp-794075965-25 - 3e184491-93f8-4fc4-afa7-b5c104c6ad64:falcon:POST//entities/submit/cluster] ~ {Action:submit, Dimensions:{colo=NULL, entityType=cluster}, Status: FAILED, Time-taken:65496087 ns} (METRIC:38)

Re: [FALCON] cluster entity creation with HA cluster

Expert Contributor

@mayki wogno On looking around the exception, I have found that issue happened while validating read Interface of HA cluster entity. You can use hdfs instead of hftp as workaround and also recreate the cluster entity again. For the endpoint you have to put your nameservice like "bigdata" or "bigdata:8020"

<interface type="readonly" endpoint="hdfs://bigdata" version="2.2.0"/>

Re: [FALCON] cluster entity creation with HA cluster

Explorer

@peeyush : failed with these two parametersand I'm suppose it is the same for "interface EXECUTE"

Caused by: java.io.IOException: Incomplete HDFS URI, no host: hdfs://bigdata

or

Caused by: java.io.IOException: Incomplete HDFS URI, no host: hdfs://bigdata:8020

Re: [FALCON] cluster entity creation with HA cluster

Expert Contributor
@mayki wogno

Can you please explain how it is same for "interface EXECUTE".

For "no host", is "bigdata" nameservice for HA or HDFS namenode name.

I initially assumed that you are trying to run in HA environment , so you require nameservice for that.

Re: [FALCON] cluster entity creation with HA cluster

Explorer

@peeyush

I have le cluster in HA mode.

Hostname : clustera001, clustera002

Hadoop cluster name = bigdata

Nameservice : nn1, nn2

Re: [FALCON] cluster entity creation with HA cluster

@mayki wogno

Have you configured your cluster with all name services?

What does your dfs.nameservices looks like?

Re: [FALCON] cluster entity creation with HA cluster

Explorer

@rahul : I've follow this url to configure multi dfs.nameservices but failed to restart HDFS, my stack is over than HDP2.3

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_HDP_RelNotes/content/behav-changes-230.h...

dfs.nameservices = clusterA,clusterB

dfs.internal.nameservices = clusterA

Re: [FALCON] cluster entity creation with HA cluster

@mayki wogno

Just check.

It should be dfs.internal.nameservices, note trailing s in nameservices

Also there should not be space between clusterA, clusterB

It should be something like below:

  1. dfs.nameservices=clusterA,clusterB
  2. dfs.internal.nameservices=clusterA

Also ensure similar settings are done on other cluster.