question Re: Falcon UI not working in Support Questions

Falcon UI not working

sushil61 — Wed, 27 Jan 2016 08:25:25 GMT

I restarted Falcon Server and getting this error:

HTTP ERROR: 503 Problem accessing /index.html. Reason: SERVICE_UNAVAILABLE

Re: Falcon UI not working

aervits — Wed, 27 Jan 2016 08:51:19 GMT

Please provde Falcon, HDP and Ambari versions, also provide logs. @Sushil Saxena

Re: Falcon UI not working

sushil61 — Wed, 27 Jan 2016 09:04:34 GMT

Falcon: 0.6.1.2.3, HDP: 2.3.2.0-2950, Ambari: 2.1.2.1 Which log I need to provide?

Extract from application log:

2016-01-27 01:00:50,873 ERROR - [main:] ~ Failed to initialize service org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:49)
org.apache.falcon.FalconException: Unable to restore configurations for entity type PROCESS
        at org.apache.falcon.entity.store.ConfigurationStore.loadEntity(ConfigurationStore.java:189)
        at org.apache.falcon.entity.store.ConfigurationStore.init(ConfigurationStore.java:152)
        at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47)
        at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56)
        at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:550)
        at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
        at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
        at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:519)
        at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
        at org.mortbay.jetty.Server.doStart(Server.java:224)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:57)
        at org.apache.falcon.Main.main(Main.java:83)
2016-01-27 01:00:50,875 ERROR - [main:] ~ Failed startup of context org.mortbay.jetty.webapp.WebAppContext@4759d881{/,/usr/hdp/current/falcon-server/server/webapp/falcon} (log:87)
java.lang.RuntimeException: org.apache.falcon.FalconException: org.apache.falcon.FalconException: Unable to restore configurations for entity type PROCESS
        at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:59)
        at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:550)
        at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
        at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
        at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:519)
        at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
        at org.mortbay.jetty.Server.doStart(Server.java:224)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:57)
        at org.apache.falcon.Main.main(Main.java:83)
Caused by: org.apache.falcon.FalconException: org.apache.falcon.FalconException: Unable to restore configurations for entity type PROCESS
        at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:50)
        at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56)
        ... 11 more
Caused by: org.apache.falcon.FalconException: Unable to restore configurations for entity type PROCESS
        at org.apache.falcon.entity.store.ConfigurationStore.loadEntity(ConfigurationStore.java:189)
        at org.apache.falcon.entity.store.ConfigurationStore.init(ConfigurationStore.java:152)
        at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47)
        ... 12 more
2016-01-27 01:00:50,879 INFO  - [main:] ~ Started SocketConnector@0.0.0.0:15000 (log:67)

Re: Falcon UI not working

sushil61 — Wed, 27 Jan 2016 09:21:42 GMT

Please see the log in my previous response. The output of ps -ef | grep falcon

falcon 13996 1 3 01:00 ? 00:00:31 /usr/jdk64/jdk1.7.0_67/bin/java -Xmx1024m -noverify -Dfalcon.embeddedmq=True -Dfalcon.emeddedmq.port=61616 -Dfalcon.log.dir=/var/log/falcon -Dfalcon.embeddedmq.data=/hadoop/falcon/embeddedmq/data -Dfalcon.home=/usr/hdp/current/falcon-server -Dconfig.location=/usr/hdp/current/falcon-server/conf -Dfalcon.app.type=falcon -Dfalcon.catalog.service.enabled= -cp /usr/hdp/current/falcon-server/conf:/usr/hdp/2.3.2.0-2950/hadoop/conf:/usr/hdp/2.3.2.0-2950/hadoop/lib/*:/usr/hdp/2.3.2.0-2950/hadoop/.//*:/usr/hdp/2.3.2.0-2950/hadoop-hdfs/./:/usr/hdp/2.3.2.0-2950/hadoop-hdfs/lib/*:/usr/hdp/2.3.2.0-2950/hadoop-hdfs/.//*:/usr/hdp/2.3.2.0-2950/hadoop-yarn/lib/*:/usr/hdp/2.3.2.0-2950/hadoop-yarn/.//*:/usr/hdp/2.3.2.0-2950/hadoop-mapreduce/lib/*:/usr/hdp/2.3.2.0-2950/hadoop-mapreduce/.//*:::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/hdp/current/falcon-server/server/webapp/falcon/WEB-INF/classes:/usr/hdp/current/falcon-server/server/webapp/falcon/WEB-INF/lib/*:/usr/hdp/current/falcon-server/libext/* org.apache.falcon.Main -app /usr/hdp/current/falcon-server/server/webapp/falcon -port 15000

On Ambari, there is not error on Falcon Server. But when I tried to run Falcon UI, I am getting: HTTP ERROR: 503 Problem accessing /index.html. Reason: SERVICE_UNAVAILABLE

Please Note: I need to Update pig-action.xml to have hive in the share lib config and repackage

falcon-oozie-adaptor-0.6.1.2.3.2.0-2950.jar and replace jar at "/usr/hdp/current/falcon-server/webapp/falcon/WEB-INF/lib” and restart Falcon.

Re: Falcon UI not working

sushil61 — Thu, 28 Jan 2016 00:47:33 GMT

The issue has been resolved. The entity was corrupt and we restored the falcon Store and WebApp.

Re: Falcon UI not working

dconnolly — Sat, 10 Sep 2016 19:52:47 GMT

Hi, having same issue, followed steps in jira https://issues.apache.org/jira/browse/AMBARI-18046?jql=project%20%3D%20AMBARI and still getting This host-level alert is triggered if the Falcon Server Web UI is unreachable - 503. Any insights on what you did to get around this. I am running ambari 2.4 on HDP 2.5 tech preview. Steps from Jira listed here.

STR: 1) Upgrade Ambari from 2.2.1 to 2.4.0 2) Delete Falcon 3) Add Falcon

Result: Falcon UI is unavailable. From Falcon logs: java.lang.RuntimeException: org.apache.falcon.FalconException: Unable to get instance for org.apache.falcon.atlas.service.AtlasService

Re: Falcon UI not working

msertacturkel — Thu, 29 Dec 2016 23:32:18 GMT

Hi all,

I am getting a warning from Falcon like yours. I am using HDP-2.5.3.0, Ambari Version2.4.2.0 and Falcon 0.10.0. There is no valuable log into falcon.application.log. Falcon warns me like this:

HTTP 503 response from http://hostname:15000 in 0.000s (HTTP Error 503: Service Unavailable)

Do you have any ideas?

Re: Falcon UI not working

peter_braken — Wed, 04 Jan 2017 18:14:27 GMT

As HortonWorks partner we have tested this an confirm the error.

HDP-2.5.3.0, Ambari Version2.4.2.0 and Falcon 0.10.0 using distro Ubuntu 14.0.4

HDP-2.5.3.0, Ambari Version2.4.1.0 and Falcon 0.10.0 using distro CentOS 7

If i am not mistaken this problem did NOT occur on HDP 2.5.0.0-1245

During install i get warning on one of the slave nodes

resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/hdp/current/falcon-client/bin/falcon admin -version' returned 255. ERROR: Unable to initialize Falcon Client object. Cause : Could not authenticate, Authentication failed, URL: http://master2.localhost:15000/api/options?user.name=ambari-qa&user.name=ambari-qa, status: 503, message: Service Unavailable

Re: Falcon UI not working

zachary_marsh — Thu, 05 Jan 2017 03:44:39 GMT

I am also encountering the same issues (HTTP Error 503 when trying to access the Falcon web UI and Ambari Service Check for Falcon is consistently failing).

HDP 2.5.3.0-37, Ambari 2.4.2.0-136, Falcon 0.10.0.2.5.3.0-37 installed on SLES 11 SP3

This problem didn't occur when the cluster was installed with HDP 2.5.0.2 and Ambari 2.4.1.0.

I've opened the following Apache JIRA for the Ambari project for this issue:

https://issues.apache.org/jira/browse/AMBARI-19370

Excerpts from the falcon.application.log:

2017-01-04 14:08:56,436 ERROR - [main:] ~ Failed to initialize service org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:49)
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].

2017-01-04 14:08:56,438 ERROR - [main:] ~ Failed startup of context org.mortbay.jetty.webapp.WebAppContext@96a75da{/,/usr/hdp/current/falcon-server/server/webapp/falcon} (log:87) 
java.lang.RuntimeException: org.apache.falcon.FalconException: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].

Re: Falcon UI not working

bandalora — Fri, 06 Jan 2017 07:43:52 GMT

As of HDP 2.5.3, you need to install the Berkeley DB prior to upgrading or installing Falcon. That might be the problem for some of you. You might try the following: [Updated content below]

Download the required Berkeley DB implementation file.
wget –O je-5.0.73.jar http://search.maven.org/remotecontent?filepath=com/sleepycat/je/5.0.73/je-5.0.73.jar
Log in to the Ambari server with administrator privileges.
su – root
Copy the file to the Ambari server share folder.
cp je-5.0.73.jar /usr/share/
Set permissions on the file to owner=read/write, group=read, other=read.
chmod 644 /usr/share/je-5.0.73.jar
Configure the Ambari server to use the Berkeley DB driver.
ambari-server setup --jdbc-db=bdb --jdbc-driver=/usr/share/je-5.0.73.jar
Restart the Ambari server.
ambari-server restart
Restart the Falcon service from the Ambari UI.
You need to have administrator privileges in Ambari to restart a service.
1. In the Ambari web UI, click the Services tab and select the Falcon service in the left Services pane.
2. From the Falcon Summary page, click Service Actions > Restart All.
3. Click Confirm Restart All.

When the service is available, the Falcon status displays as Started on the Summary page.

Further information and manual install instructions are available in an article at https://community.hortonworks.com/articles/78274/prerequisite-to-installing-or-upgrading-falcon.html.

Re: Falcon UI not working

andy1 — Mon, 09 Jan 2017 23:23:54 GMT

Cut'n'pasteable / idempotent:

su - falcon
l_hdp_ver=$(ls -d /usr/hdp/[0-9]* | head -n 1 | sed -e 's#^.*/\([0-9].*\)#\1#')
wget http://download.oracle.com/maven/com/sleepycat/je/5.0.73/je-5.0.73.jar
yes | cp je-5.0.73.jar /usr/hdp/$l_hdp_ver/falcon/webapp/falcon/WEB-INF/lib
chmod 644 /usr/hdp/$l_hdp_ver/falcon/webapp/falcon/WEB-INF/lib/je-5.0.73.jar

Re: Falcon UI not working

mlucero — Tue, 10 Jan 2017 03:58:54 GMT

I also just hit this issue after upgrading to 2.5.3.0-37

Re: Falcon UI not working

vranganathan — Wed, 11 Jan 2017 02:08:29 GMT

You need to register download and register the bdb jar with ambari server before starting Falcon server. This is required from HDP 2.5 or later now The following steps would help

wget -O je-5.0.73.jar http://search.maven.org/remotecontent?filepath=com/sleepycat/je/5.0.73/je-5.0.73.jar
cp je-5.0.73.jar /usr/share/
chmod 644 /usr/share/je-5.0.73.jar
ambari-server setup --jdbc-db=bdb --jdbc-driver=/usr/share/je-5.0.73.jar
ambari-server restart
Restart Falcon service

Re: Falcon UI not working

peter_braken — Thu, 12 Jan 2017 19:09:59 GMT

Great answer @Venkat Ranganathan your answer fixed our problem

Re: Falcon UI not working

narenreddy — Fri, 27 Jan 2017 12:14:52 GMT

It worked Perfectly Venkat R...

Re: Falcon UI not working

rafal_fojcik — Wed, 12 Jul 2017 18:24:20 GMT

Hi Venkat, I did the above, but the issue still persists on my cluster. Do you have any idea what else might be wrong or what's worth checking? Basically I get the same error whether I try to run a service check or see the falcon version

ERROR: Unable to initialize Falcon Client object. Cause : Could not authenticate, Authentication failed, URL: <a href="http://<host>:15000/api/options?user.name=falcon&user.name=falcon,">http://:15000/api/options?user.name=falcon&user.name=falcon,>:15000/api/options?user.name=falcon&user.name=falcon,</a>><host>:15000/api/options?user.name=falcon&user.name=falcon, status: 503, message: Service Unavailable

In the falcon.application.log there are the following errors. From what I understood, adding the je-5.0.73.jar file should've solved this problem, but it persists and I run out of ideas.

2017-07-12 09:07:42,568 ERROR - [main:] ~ Failed to initialize service org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:49)
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].
        at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:50)
        at org.apache.falcon.metadata.MetadataMappingService.initializeGraphDB(MetadataMappingService.java:146)
        at org.apache.falcon.metadata.MetadataMappingService.init(MetadataMappingService.java:113)
        at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47)
        at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56)
        at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:549)
        at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
        at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
        at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
        at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
        at org.mortbay.jetty.Server.doStart(Server.java:224)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:58)
        at org.apache.falcon.FalconServer.main(FalconServer.java:118)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:45)
        ... 15 more
Caused by: java.lang.NoClassDefFoundError: com/sleepycat/je/LockMode
        at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.<clinit>(BerkeleyJEStoreManager.java:47)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at com.thinkaurelius.titan.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:42)
        at com.thinkaurelius.titan.diskstorage.Backend.getImplementationClass(Backend.java:421)
        at com.thinkaurelius.titan.diskstorage.Backend.getStorageManager(Backend.java:361)
        at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1275)
        at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:93)
        at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:73)
        ... 20 more
Caused by: java.lang.ClassNotFoundException: com.sleepycat.je.LockMode
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 29 more
2017-07-12 09:07:42,569 ERROR - [main:] ~ Failed startup of context org.mortbay.jetty.webapp.WebAppContext@351f2244{/,/usr/hdp/current/falcon-server/server/webapp/falcon} (log:87)
java.lang.RuntimeException: org.apache.falcon.FalconException: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].
        at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:59)
...

Re: Falcon UI not working

guntaka — Thu, 05 Apr 2018 01:16:57 GMT

Currently my ambari server is setup using mysql DB and corresponding jdbc driver. If I do the setup again to specify a different DB for Falcon will that cause inconsistency? Or I can just setup mysql for ambari and bdb for falcon?

Re: Falcon UI not working

Shelton — Thu, 05 Apr 2018 01:45:06 GMT

@Swaapnika Guntaka

As of HDP 2.5.3, you need to install the Berkeley DB prior to upgrading or installing Falcon. That might be the problem you are encountering. You might try the following: [Updated content below]

1.Download the required Berkeley DB implementation file.

wget –O je-5.0.73.jar http://search.maven.org/remotecontent?filepath=com/sleepycat/je/5.0.73/je-5.0.73.jar

2.Log in to the Ambari server with administrator privileges.

su – root

3.Copy the file to the Ambari server share folder.

cp je-5.0.73.jar /usr/share/

4.Set permissions on the file to owner=read/write, group=read, other=read.

chmod 644 /usr/share/je-5.0.73.jar

5.Configure the Ambari server to use the Berkeley DB driver.

ambari-server setup --jdbc-db=bdb --jdbc-driver=/usr/share/je-5.0.73.jar

6.Restart the Ambari server.

ambari-server restart

7.Restart the Falcon service from the Ambari UI.

You need to have administrator privileges in Ambari to restart a service.

1.In the Ambari web UI, click the Services tab and select the Falcon service in the left Services pane.

2.From the Falcon Summary page, click Service Actions > Restart All.

3.Click Confirm Restart All.

When the service is available, the Falcon status displays as Started on the Summary page.

Further information and manual install instructions are available in an article at https://community.hortonworks.com/articles/78274/prerequisite-to-installing-or-upgrading-falcon.html.

Re: Falcon UI not working

guntaka — Thu, 05 Apr 2018 02:26:21 GMT

@Geoffrey Shelton Okot

I have downloaded the jar already. The question I have is since I’ve already setup the ambari server using mysql DB and mysql jdbc driver, would doing a setup again with bdb and it’s jar cause a conflict? Or can I set up ambari using any number of databases like for eg mysql for ambari, derby for oozie, bdb for falcon etc

Re: Falcon UI not working

Shelton — Thu, 05 Apr 2018 02:41:34 GMT

@Swaapnika Guntaka

That will run independently it won't interfere with the Mysql or any other database.Falcon doesn't use the classic db's like run. Before you proceed make sure Falcon is not running ,kill any rogue falcon process