Support Questions

Find answers, ask questions, and share your expertise

Falcon UI not working

Contributor

I restarted Falcon Server and getting this error:

HTTP ERROR: 503 Problem accessing /index.html. Reason: SERVICE_UNAVAILABLE

1 ACCEPTED SOLUTION

Contributor

The issue has been resolved. The entity was corrupt and we restored the falcon Store and WebApp.

View solution in original post

19 REPLIES 19

Mentor

Please provde Falcon, HDP and Ambari versions, also provide logs. @Sushil Saxena

Contributor

Falcon: 0.6.1.2.3, HDP: 2.3.2.0-2950, Ambari: 2.1.2.1 Which log I need to provide?

Extract from application log:

2016-01-27 01:00:50,873 ERROR - [main:] ~ Failed to initialize service org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:49)
org.apache.falcon.FalconException: Unable to restore configurations for entity type PROCESS
        at org.apache.falcon.entity.store.ConfigurationStore.loadEntity(ConfigurationStore.java:189)
        at org.apache.falcon.entity.store.ConfigurationStore.init(ConfigurationStore.java:152)
        at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47)
        at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56)
        at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:550)
        at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
        at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
        at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:519)
        at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
        at org.mortbay.jetty.Server.doStart(Server.java:224)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:57)
        at org.apache.falcon.Main.main(Main.java:83)
2016-01-27 01:00:50,875 ERROR - [main:] ~ Failed startup of context org.mortbay.jetty.webapp.WebAppContext@4759d881{/,/usr/hdp/current/falcon-server/server/webapp/falcon} (log:87)
java.lang.RuntimeException: org.apache.falcon.FalconException: org.apache.falcon.FalconException: Unable to restore configurations for entity type PROCESS
        at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:59)
        at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:550)
        at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
        at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
        at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:519)
        at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
        at org.mortbay.jetty.Server.doStart(Server.java:224)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:57)
        at org.apache.falcon.Main.main(Main.java:83)
Caused by: org.apache.falcon.FalconException: org.apache.falcon.FalconException: Unable to restore configurations for entity type PROCESS
        at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:50)
        at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56)
        ... 11 more
Caused by: org.apache.falcon.FalconException: Unable to restore configurations for entity type PROCESS
        at org.apache.falcon.entity.store.ConfigurationStore.loadEntity(ConfigurationStore.java:189)
        at org.apache.falcon.entity.store.ConfigurationStore.init(ConfigurationStore.java:152)
        at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47)
        ... 12 more
2016-01-27 01:00:50,879 INFO  - [main:] ~ Started SocketConnector@0.0.0.0:15000 (log:67)

Contributor

Please see the log in my previous response. The output of ps -ef | grep falcon

falcon 13996 1 3 01:00 ? 00:00:31 /usr/jdk64/jdk1.7.0_67/bin/java -Xmx1024m -noverify -Dfalcon.embeddedmq=True -Dfalcon.emeddedmq.port=61616 -Dfalcon.log.dir=/var/log/falcon -Dfalcon.embeddedmq.data=/hadoop/falcon/embeddedmq/data -Dfalcon.home=/usr/hdp/current/falcon-server -Dconfig.location=/usr/hdp/current/falcon-server/conf -Dfalcon.app.type=falcon -Dfalcon.catalog.service.enabled= -cp /usr/hdp/current/falcon-server/conf:/usr/hdp/2.3.2.0-2950/hadoop/conf:/usr/hdp/2.3.2.0-2950/hadoop/lib/*:/usr/hdp/2.3.2.0-2950/hadoop/.//*:/usr/hdp/2.3.2.0-2950/hadoop-hdfs/./:/usr/hdp/2.3.2.0-2950/hadoop-hdfs/lib/*:/usr/hdp/2.3.2.0-2950/hadoop-hdfs/.//*:/usr/hdp/2.3.2.0-2950/hadoop-yarn/lib/*:/usr/hdp/2.3.2.0-2950/hadoop-yarn/.//*:/usr/hdp/2.3.2.0-2950/hadoop-mapreduce/lib/*:/usr/hdp/2.3.2.0-2950/hadoop-mapreduce/.//*:::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/hdp/current/falcon-server/server/webapp/falcon/WEB-INF/classes:/usr/hdp/current/falcon-server/server/webapp/falcon/WEB-INF/lib/*:/usr/hdp/current/falcon-server/libext/* org.apache.falcon.Main -app /usr/hdp/current/falcon-server/server/webapp/falcon -port 15000

On Ambari, there is not error on Falcon Server. But when I tried to run Falcon UI, I am getting: HTTP ERROR: 503 Problem accessing /index.html. Reason: SERVICE_UNAVAILABLE

Please Note: I need to Update pig-action.xml to have hive in the share lib config and repackage

falcon-oozie-adaptor-0.6.1.2.3.2.0-2950.jar and replace jar at "/usr/hdp/current/falcon-server/webapp/falcon/WEB-INF/lib” and restart Falcon.

Contributor

The issue has been resolved. The entity was corrupt and we restored the falcon Store and WebApp.

Contributor

Hi, having same issue, followed steps in jira https://issues.apache.org/jira/browse/AMBARI-18046?jql=project%20%3D%20AMBARI and still getting This host-level alert is triggered if the Falcon Server Web UI is unreachable - 503. Any insights on what you did to get around this. I am running ambari 2.4 on HDP 2.5 tech preview. Steps from Jira listed here.

STR: 1) Upgrade Ambari from 2.2.1 to 2.4.0 2) Delete Falcon 3) Add Falcon

Result: Falcon UI is unavailable. From Falcon logs: java.lang.RuntimeException: org.apache.falcon.FalconException: Unable to get instance for org.apache.falcon.atlas.service.AtlasService

Hi all,

I am getting a warning from Falcon like yours. I am using HDP-2.5.3.0, Ambari Version2.4.2.0 and Falcon 0.10.0. There is no valuable log into falcon.application.log. Falcon warns me like this:

HTTP 503 response from http://hostname:15000 in 0.000s (HTTP Error 503: Service Unavailable)

Do you have any ideas?

New Contributor

As HortonWorks partner we have tested this an confirm the error.

HDP-2.5.3.0, Ambari Version2.4.2.0 and Falcon 0.10.0 using distro Ubuntu 14.0.4

HDP-2.5.3.0, Ambari Version2.4.1.0 and Falcon 0.10.0 using distro CentOS 7

If i am not mistaken this problem did NOT occur on HDP 2.5.0.0-1245

During install i get warning on one of the slave nodes

resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/hdp/current/falcon-client/bin/falcon admin -version' returned 255. ERROR: Unable to initialize Falcon Client object. Cause : Could not authenticate, Authentication failed, URL: http://master2.localhost:15000/api/options?user.name=ambari-qa&user.name=ambari-qa, status: 503, message: Service Unavailable

New Contributor

I am also encountering the same issues (HTTP Error 503 when trying to access the Falcon web UI and Ambari Service Check for Falcon is consistently failing).

HDP 2.5.3.0-37, Ambari 2.4.2.0-136, Falcon 0.10.0.2.5.3.0-37 installed on SLES 11 SP3

This problem didn't occur when the cluster was installed with HDP 2.5.0.2 and Ambari 2.4.1.0.

I've opened the following Apache JIRA for the Ambari project for this issue:

https://issues.apache.org/jira/browse/AMBARI-19370

Excerpts from the falcon.application.log:

2017-01-04 14:08:56,436 ERROR - [main:] ~ Failed to initialize service org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:49)
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].
2017-01-04 14:08:56,438 ERROR - [main:] ~ Failed startup of context org.mortbay.jetty.webapp.WebAppContext@96a75da{/,/usr/hdp/current/falcon-server/server/webapp/falcon} (log:87) 
java.lang.RuntimeException: org.apache.falcon.FalconException: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].

Rising Star

As of HDP 2.5.3, you need to install the Berkeley DB prior to upgrading or installing Falcon. That might be the problem for some of you. You might try the following: [Updated content below]

  1. Download the required Berkeley DB implementation file.

    wget –O je-5.0.73.jar http://search.maven.org/remotecontent?filepath=com/sleepycat/je/5.0.73/je-5.0.73.jar

  2. Log in to the Ambari server with administrator privileges.

    su – root

  3. Copy the file to the Ambari server share folder.

    cp je-5.0.73.jar /usr/share/

  4. Set permissions on the file to owner=read/write, group=read, other=read.

    chmod 644 /usr/share/je-5.0.73.jar

  5. Configure the Ambari server to use the Berkeley DB driver.

    ambari-server setup --jdbc-db=bdb --jdbc-driver=/usr/share/je-5.0.73.jar

  6. Restart the Ambari server.

    ambari-server restart

  7. Restart the Falcon service from the Ambari UI.

    You need to have administrator privileges in Ambari to restart a service.

    1. In the Ambari web UI, click the Services tab and select the Falcon service in the left Services pane.
    2. From the Falcon Summary page, click Service Actions > Restart All.
    3. Click Confirm Restart All.

When the service is available, the Falcon status displays as Started on the Summary page.

Further information and manual install instructions are available in an article at https://community.hortonworks.com/articles/78274/prerequisite-to-installing-or-upgrading-falcon.html.

New Contributor

Cut'n'pasteable / idempotent:

su - falcon
l_hdp_ver=$(ls -d /usr/hdp/[0-9]* | head -n 1 | sed -e 's#^.*/\([0-9].*\)#\1#')
wget http://download.oracle.com/maven/com/sleepycat/je/5.0.73/je-5.0.73.jar
yes | cp je-5.0.73.jar /usr/hdp/$l_hdp_ver/falcon/webapp/falcon/WEB-INF/lib
chmod 644 /usr/hdp/$l_hdp_ver/falcon/webapp/falcon/WEB-INF/lib/je-5.0.73.jar

Explorer

I also just hit this issue after upgrading to 2.5.3.0-37

Rising Star

You need to register download and register the bdb jar with ambari server before starting Falcon server. This is required from HDP 2.5 or later now The following steps would help

  1. wget -O je-5.0.73.jar http://search.maven.org/remotecontent?filepath=com/sleepycat/je/5.0.73/je-5.0.73.jar
  2. cp je-5.0.73.jar /usr/share/
  3. chmod 644 /usr/share/je-5.0.73.jar
  4. ambari-server setup --jdbc-db=bdb --jdbc-driver=/usr/share/je-5.0.73.jar
  5. ambari-server restart
  6. Restart Falcon service

Explorer

It worked Perfectly Venkat R...

Hi Venkat, I did the above, but the issue still persists on my cluster. Do you have any idea what else might be wrong or what's worth checking? Basically I get the same error whether I try to run a service check or see the falcon version

ERROR: Unable to initialize Falcon Client object. Cause : Could not authenticate, Authentication failed, URL: <a href="http://<host>:15000/api/options?user.name=falcon&user.name=falcon,">http://:15000/api/options?user.name=falcon&user.name=falcon,>:15000/api/options?user.name=falcon&user.name=falcon,</a>><host>:15000/api/options?user.name=falcon&user.name=falcon, status: 503, message: Service Unavailable

In the falcon.application.log there are the following errors. From what I understood, adding the je-5.0.73.jar file should've solved this problem, but it persists and I run out of ideas.

2017-07-12 09:07:42,568 ERROR - [main:] ~ Failed to initialize service org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:49)
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].
        at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:50)
        at org.apache.falcon.metadata.MetadataMappingService.initializeGraphDB(MetadataMappingService.java:146)
        at org.apache.falcon.metadata.MetadataMappingService.init(MetadataMappingService.java:113)
        at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47)
        at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56)
        at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:549)
        at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
        at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
        at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
        at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
        at org.mortbay.jetty.Server.doStart(Server.java:224)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:58)
        at org.apache.falcon.FalconServer.main(FalconServer.java:118)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:45)
        ... 15 more
Caused by: java.lang.NoClassDefFoundError: com/sleepycat/je/LockMode
        at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.<clinit>(BerkeleyJEStoreManager.java:47)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at com.thinkaurelius.titan.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:42)
        at com.thinkaurelius.titan.diskstorage.Backend.getImplementationClass(Backend.java:421)
        at com.thinkaurelius.titan.diskstorage.Backend.getStorageManager(Backend.java:361)
        at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1275)
        at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:93)
        at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:73)
        ... 20 more
Caused by: java.lang.ClassNotFoundException: com.sleepycat.je.LockMode
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 29 more
2017-07-12 09:07:42,569 ERROR - [main:] ~ Failed startup of context org.mortbay.jetty.webapp.WebAppContext@351f2244{/,/usr/hdp/current/falcon-server/server/webapp/falcon} (log:87)
java.lang.RuntimeException: org.apache.falcon.FalconException: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].
        at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:59)
...

New Contributor

Great answer @Venkat Ranganathan your answer fixed our problem

Explorer

Currently my ambari server is setup using mysql DB and corresponding jdbc driver. If I do the setup again to specify a different DB for Falcon will that cause inconsistency? Or I can just setup mysql for ambari and bdb for falcon?

Mentor

@Swaapnika Guntaka

As of HDP 2.5.3, you need to install the Berkeley DB prior to upgrading or installing Falcon. That might be the problem you are encountering. You might try the following: [Updated content below]

1.Download the required Berkeley DB implementation file.

wget –O je-5.0.73.jar http://search.maven.org/remotecontent?filepath=com/sleepycat/je/5.0.73/je-5.0.73.jar

2.Log in to the Ambari server with administrator privileges.

su – root

3.Copy the file to the Ambari server share folder.

cp je-5.0.73.jar /usr/share/

4.Set permissions on the file to owner=read/write, group=read, other=read.

chmod 644 /usr/share/je-5.0.73.jar

5.Configure the Ambari server to use the Berkeley DB driver.

ambari-server setup --jdbc-db=bdb --jdbc-driver=/usr/share/je-5.0.73.jar

6.Restart the Ambari server.

ambari-server restart

7.Restart the Falcon service from the Ambari UI.

You need to have administrator privileges in Ambari to restart a service.

1.In the Ambari web UI, click the Services tab and select the Falcon service in the left Services pane.

2.From the Falcon Summary page, click Service Actions > Restart All.

3.Click Confirm Restart All.

When the service is available, the Falcon status displays as Started on the Summary page.

Further information and manual install instructions are available in an article at https://community.hortonworks.com/articles/78274/prerequisite-to-installing-or-upgrading-falcon.html.

Explorer

@Geoffrey Shelton Okot

I have downloaded the jar already. The question I have is since I’ve already setup the ambari server using mysql DB and mysql jdbc driver, would doing a setup again with bdb and it’s jar cause a conflict? Or can I set up ambari using any number of databases like for eg mysql for ambari, derby for oozie, bdb for falcon etc

Mentor

@Swaapnika Guntaka

That will run independently it won't interfere with the Mysql or any other database.Falcon doesn't use the classic db's like run. Before you proceed make sure Falcon is not running ,kill any rogue falcon process