Faclon Web UI failing with HTTP 503 service unavailable

[#1]

Falcon webUi not working From http://xxxxx.com:15000 in 0.000s (HTTP Error 503: Service Unavailable).

Service check fails with SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] ERROR: Unable to initialize Falcon Client object

falcon admin -version command also fails with the same error .

Any help appreciated .

[#2 · Accepted solution]

@Shihab - This might occur due to presence of corrupt entities in falcon store . Please check for any such exception in falcon.applciation.log . falcon would fail to restore such entities. You can test these by temporarily moving entities to a backup location.

Default location is /hadoop/falcon/store/<entity_type>/

[#3]

If you installed HDP 2.5 or later, you must manually install the Berkeley DB file required for Falcon. Below are the instructions for Ambari installations. See if this helps.

1. Download the required Berkeley DB implementation file.

wget –O je-5.0.73.jar http://search.maven.org/remotecontent?filepath=com/sleepycat/je/5.0.73/je-5.0.73.jar

2. Log in to the Ambari server with administrator privileges.

su – root

3. Copy the file to the Ambari server share folder.

cp je-5.0.73.jar /usr/share/

4. Set permissions on the file to owner=read/write, group=read, other=read.

chmod 644 /usr/share/je-5.0.73.jar

5. Configure the Ambari server to use the Berkeley DB driver.

ambari-server setup --jdbc-db=bdb --jdbc-driver=/usr/share/je-5.0.73.jar

6. Restart the Ambari server.

ambari-server restart

7. Restart the Falcon service from the Ambari UI.

You need to have administrator privileges in Ambari to restart a service.

a) In the Ambari web UI, click the Services tab and select the Falcon service in the left Services pane.

b) From the Falcon Summary page, click Service Actions > Restart All.

c) Click Confirm Restart All.

When the service is available, the Falcon status displays as Started on the Summary page.

[#4]

@Shihab It looks like that Falcon server did not come up properly due to which Falcon webUI and client is having issues. Can you please share the falcon application log from Falcon logs directory to analyze the Falcon server issue.

[#5]

@Beverley Andalora Thank you. This helped my case.

I have provided the error which I receiving below (before making your recommended changes) as it may help others.

java.lang.RuntimeException: org.apache.falcon.FalconException: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].
	at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:59)
	at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:549)
	at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
	at org.mortbay.jetty.Server.doStart(Server.java:224)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:58)
	at org.apache.falcon.FalconServer.main(FalconServer.java:118)
Caused by: org.apache.falcon.FalconException: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].
	at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:50)
	at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56)
	... 11 more
Caused by: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].
	at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:50)
	at org.apache.falcon.metadata.MetadataMappingService.initializeGraphDB(MetadataMappingService.java:146)
	at org.apache.falcon.metadata.MetadataMappingService.init(MetadataMappingService.java:113)
	at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47)
	... 12 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:45)
	... 15 more
Caused by: java.lang.NoClassDefFoundError: com/sleepycat/je/LockMode
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.<clinit>(BerkeleyJEStoreManager.java:47)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at com.thinkaurelius.titan.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:42)
	at com.thinkaurelius.titan.diskstorage.Backend.getImplementationClass(Backend.java:421)
	at com.thinkaurelius.titan.diskstorage.Backend.getStorageManager(Backend.java:361)
	at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1275)
	at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:93)
	at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:73)
	... 20 more
Caused by: java.lang.ClassNotFoundException: com.sleepycat.je.LockMode
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 29 more
2017-06-23 20:51:14,687 INFO  - [main:] ~ Started SocketConnector@0.0.0.0:15000 (log:67)