Created 01-27-2016 12:25 AM
I restarted Falcon Server and getting this error:
HTTP ERROR: 503 Problem accessing /index.html. Reason: SERVICE_UNAVAILABLE
Created 01-27-2016 04:47 PM
The issue has been resolved. The entity was corrupt and we restored the falcon Store and WebApp.
Created 01-27-2016 12:51 AM
Please provde Falcon, HDP and Ambari versions, also provide logs. @Sushil Saxena
Created 01-27-2016 01:04 AM
Falcon: 0.6.1.2.3, HDP: 2.3.2.0-2950, Ambari: 2.1.2.1 Which log I need to provide?
Extract from application log:
2016-01-27 01:00:50,873 ERROR - [main:] ~ Failed to initialize service org.apache.falcon.entity.store.ConfigurationStore (ServiceInitializer:49) org.apache.falcon.FalconException: Unable to restore configurations for entity type PROCESS at org.apache.falcon.entity.store.ConfigurationStore.loadEntity(ConfigurationStore.java:189) at org.apache.falcon.entity.store.ConfigurationStore.init(ConfigurationStore.java:152) at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47) at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56) at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:550) at org.mortbay.jetty.servlet.Context.startContext(Context.java:136) at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282) at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:519) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) at org.mortbay.jetty.Server.doStart(Server.java:224) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:57) at org.apache.falcon.Main.main(Main.java:83) 2016-01-27 01:00:50,875 ERROR - [main:] ~ Failed startup of context org.mortbay.jetty.webapp.WebAppContext@4759d881{/,/usr/hdp/current/falcon-server/server/webapp/falcon} (log:87) java.lang.RuntimeException: org.apache.falcon.FalconException: org.apache.falcon.FalconException: Unable to restore configurations for entity type PROCESS at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:59) at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:550) at org.mortbay.jetty.servlet.Context.startContext(Context.java:136) at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282) at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:519) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) at org.mortbay.jetty.Server.doStart(Server.java:224) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:57) at org.apache.falcon.Main.main(Main.java:83) Caused by: org.apache.falcon.FalconException: org.apache.falcon.FalconException: Unable to restore configurations for entity type PROCESS at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:50) at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56) ... 11 more Caused by: org.apache.falcon.FalconException: Unable to restore configurations for entity type PROCESS at org.apache.falcon.entity.store.ConfigurationStore.loadEntity(ConfigurationStore.java:189) at org.apache.falcon.entity.store.ConfigurationStore.init(ConfigurationStore.java:152) at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47) ... 12 more 2016-01-27 01:00:50,879 INFO - [main:] ~ Started SocketConnector@0.0.0.0:15000 (log:67)
Created 01-27-2016 01:21 AM
Please see the log in my previous response. The output of ps -ef | grep falcon
falcon 13996 1 3 01:00 ? 00:00:31 /usr/jdk64/jdk1.7.0_67/bin/java -Xmx1024m -noverify -Dfalcon.embeddedmq=True -Dfalcon.emeddedmq.port=61616 -Dfalcon.log.dir=/var/log/falcon -Dfalcon.embeddedmq.data=/hadoop/falcon/embeddedmq/data -Dfalcon.home=/usr/hdp/current/falcon-server -Dconfig.location=/usr/hdp/current/falcon-server/conf -Dfalcon.app.type=falcon -Dfalcon.catalog.service.enabled= -cp /usr/hdp/current/falcon-server/conf:/usr/hdp/2.3.2.0-2950/hadoop/conf:/usr/hdp/2.3.2.0-2950/hadoop/lib/*:/usr/hdp/2.3.2.0-2950/hadoop/.//*:/usr/hdp/2.3.2.0-2950/hadoop-hdfs/./:/usr/hdp/2.3.2.0-2950/hadoop-hdfs/lib/*:/usr/hdp/2.3.2.0-2950/hadoop-hdfs/.//*:/usr/hdp/2.3.2.0-2950/hadoop-yarn/lib/*:/usr/hdp/2.3.2.0-2950/hadoop-yarn/.//*:/usr/hdp/2.3.2.0-2950/hadoop-mapreduce/lib/*:/usr/hdp/2.3.2.0-2950/hadoop-mapreduce/.//*:::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/hdp/current/falcon-server/server/webapp/falcon/WEB-INF/classes:/usr/hdp/current/falcon-server/server/webapp/falcon/WEB-INF/lib/*:/usr/hdp/current/falcon-server/libext/* org.apache.falcon.Main -app /usr/hdp/current/falcon-server/server/webapp/falcon -port 15000
On Ambari, there is not error on Falcon Server. But when I tried to run Falcon UI, I am getting: HTTP ERROR: 503 Problem accessing /index.html. Reason: SERVICE_UNAVAILABLE
Please Note: I need to Update pig-action.xml to have hive in the share lib config and repackage
falcon-oozie-adaptor-0.6.1.2.3.2.0-2950.jar and replace jar at "/usr/hdp/current/falcon-server/webapp/falcon/WEB-INF/lib” and restart Falcon.
Created 01-27-2016 04:47 PM
The issue has been resolved. The entity was corrupt and we restored the falcon Store and WebApp.
Created 09-10-2016 12:52 PM
Hi, having same issue, followed steps in jira https://issues.apache.org/jira/browse/AMBARI-18046?jql=project%20%3D%20AMBARI and still getting This host-level alert is triggered if the Falcon Server Web UI is unreachable - 503. Any insights on what you did to get around this. I am running ambari 2.4 on HDP 2.5 tech preview. Steps from Jira listed here.
STR: 1) Upgrade Ambari from 2.2.1 to 2.4.0 2) Delete Falcon 3) Add Falcon
Result: Falcon UI is unavailable. From Falcon logs: java.lang.RuntimeException: org.apache.falcon.FalconException: Unable to get instance for org.apache.falcon.atlas.service.AtlasService
Created 12-29-2016 03:32 PM
Hi all,
I am getting a warning from Falcon like yours. I am using HDP-2.5.3.0, Ambari Version2.4.2.0 and Falcon 0.10.0. There is no valuable log into falcon.application.log. Falcon warns me like this:
HTTP 503 response from http://hostname:15000 in 0.000s (HTTP Error 503: Service Unavailable)
Do you have any ideas?
Created 01-04-2017 10:14 AM
As HortonWorks partner we have tested this an confirm the error.
HDP-2.5.3.0, Ambari Version2.4.2.0 and Falcon 0.10.0 using distro Ubuntu 14.0.4
HDP-2.5.3.0, Ambari Version2.4.1.0 and Falcon 0.10.0 using distro CentOS 7
If i am not mistaken this problem did NOT occur on HDP 2.5.0.0-1245
During install i get warning on one of the slave nodes
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/hdp/current/falcon-client/bin/falcon admin -version' returned 255. ERROR: Unable to initialize Falcon Client object. Cause : Could not authenticate, Authentication failed, URL: http://master2.localhost:15000/api/options?user.name=ambari-qa&user.name=ambari-qa, status: 503, message: Service Unavailable
Created 01-04-2017 07:44 PM
I am also encountering the same issues (HTTP Error 503 when trying to access the Falcon web UI and Ambari Service Check for Falcon is consistently failing).
HDP 2.5.3.0-37, Ambari 2.4.2.0-136, Falcon 0.10.0.2.5.3.0-37 installed on SLES 11 SP3
This problem didn't occur when the cluster was installed with HDP 2.5.0.2 and Ambari 2.4.1.0.
I've opened the following Apache JIRA for the Ambari project for this issue:
https://issues.apache.org/jira/browse/AMBARI-19370
Excerpts from the falcon.application.log:
2017-01-04 14:08:56,436 ERROR - [main:] ~ Failed to initialize service org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:49) java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].
2017-01-04 14:08:56,438 ERROR - [main:] ~ Failed startup of context org.mortbay.jetty.webapp.WebAppContext@96a75da{/,/usr/hdp/current/falcon-server/server/webapp/falcon} (log:87) java.lang.RuntimeException: org.apache.falcon.FalconException: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory].
Created 01-05-2017 11:43 PM
As of HDP 2.5.3, you need to install the Berkeley DB prior to upgrading or installing Falcon. That might be the problem for some of you. You might try the following: [Updated content below]
wget –O je-5.0.73.jar http://search.maven.org/remotecontent?filepath=com/sleepycat/je/5.0.73/je-5.0.73.jar
su – root
cp je-5.0.73.jar /usr/share/
chmod 644 /usr/share/je-5.0.73.jar
ambari-server setup --jdbc-db=bdb --jdbc-driver=/usr/share/je-5.0.73.jar
ambari-server restart
You need to have administrator privileges in Ambari to restart a service.
When the service is available, the Falcon status displays as Started on the Summary page.
Further information and manual install instructions are available in an article at https://community.hortonworks.com/articles/78274/prerequisite-to-installing-or-upgrading-falcon.html.
Created 01-09-2017 03:23 PM
Cut'n'pasteable / idempotent:
su - falcon l_hdp_ver=$(ls -d /usr/hdp/[0-9]* | head -n 1 | sed -e 's#^.*/\([0-9].*\)#\1#') wget http://download.oracle.com/maven/com/sleepycat/je/5.0.73/je-5.0.73.jar yes | cp je-5.0.73.jar /usr/hdp/$l_hdp_ver/falcon/webapp/falcon/WEB-INF/lib chmod 644 /usr/hdp/$l_hdp_ver/falcon/webapp/falcon/WEB-INF/lib/je-5.0.73.jar
Created 01-09-2017 07:58 PM
I also just hit this issue after upgrading to 2.5.3.0-37
Created 01-10-2017 06:08 PM
You need to register download and register the bdb jar with ambari server before starting Falcon server. This is required from HDP 2.5 or later now The following steps would help
Created 01-27-2017 04:14 AM
It worked Perfectly Venkat R...
Created 07-12-2017 11:24 AM
Hi Venkat, I did the above, but the issue still persists on my cluster. Do you have any idea what else might be wrong or what's worth checking? Basically I get the same error whether I try to run a service check or see the falcon version
ERROR: Unable to initialize Falcon Client object. Cause : Could not authenticate, Authentication failed, URL: <a href="http://<host>:15000/api/options?user.name=falcon&user.name=falcon,">http://:15000/api/options?user.name=falcon&user.name=falcon,>:15000/api/options?user.name=falcon&user.name=falcon,</a>><host>:15000/api/options?user.name=falcon&user.name=falcon, status: 503, message: Service Unavailable
In the falcon.application.log there are the following errors. From what I understood, adding the je-5.0.73.jar file should've solved this problem, but it persists and I run out of ideas.
2017-07-12 09:07:42,568 ERROR - [main:] ~ Failed to initialize service org.apache.falcon.metadata.MetadataMappingService (ServiceInitializer:49) java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory]. at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:50) at org.apache.falcon.metadata.MetadataMappingService.initializeGraphDB(MetadataMappingService.java:146) at org.apache.falcon.metadata.MetadataMappingService.init(MetadataMappingService.java:113) at org.apache.falcon.service.ServiceInitializer.initialize(ServiceInitializer.java:47) at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:56) at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:549) at org.mortbay.jetty.servlet.Context.startContext(Context.java:136) at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282) at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) at org.mortbay.jetty.Server.doStart(Server.java:224) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.apache.falcon.util.EmbeddedServer.start(EmbeddedServer.java:58) at org.apache.falcon.FalconServer.main(FalconServer.java:118) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.tinkerpop.blueprints.GraphFactory.open(GraphFactory.java:45) ... 15 more Caused by: java.lang.NoClassDefFoundError: com/sleepycat/je/LockMode at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.<clinit>(BerkeleyJEStoreManager.java:47) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.thinkaurelius.titan.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:42) at com.thinkaurelius.titan.diskstorage.Backend.getImplementationClass(Backend.java:421) at com.thinkaurelius.titan.diskstorage.Backend.getStorageManager(Backend.java:361) at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1275) at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:93) at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:73) ... 20 more Caused by: java.lang.ClassNotFoundException: com.sleepycat.je.LockMode at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 29 more 2017-07-12 09:07:42,569 ERROR - [main:] ~ Failed startup of context org.mortbay.jetty.webapp.WebAppContext@351f2244{/,/usr/hdp/current/falcon-server/server/webapp/falcon} (log:87) java.lang.RuntimeException: org.apache.falcon.FalconException: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory]. at org.apache.falcon.listener.ContextStartupListener.contextInitialized(ContextStartupListener.java:59) ...
Created 01-12-2017 11:09 AM
Great answer @Venkat Ranganathan your answer fixed our problem
Created 04-04-2018 06:16 PM
Currently my ambari server is setup using mysql DB and corresponding jdbc driver. If I do the setup again to specify a different DB for Falcon will that cause inconsistency? Or I can just setup mysql for ambari and bdb for falcon?
Created 04-04-2018 06:45 PM
As of HDP 2.5.3, you need to install the Berkeley DB prior to upgrading or installing Falcon. That might be the problem you are encountering. You might try the following: [Updated content below]
1.Download the required Berkeley DB implementation file.
wget –O je-5.0.73.jar http://search.maven.org/remotecontent?filepath=com/sleepycat/je/5.0.73/je-5.0.73.jar
2.Log in to the Ambari server with administrator privileges.
su – root
3.Copy the file to the Ambari server share folder.
cp je-5.0.73.jar /usr/share/
4.Set permissions on the file to owner=read/write, group=read, other=read.
chmod 644 /usr/share/je-5.0.73.jar
5.Configure the Ambari server to use the Berkeley DB driver.
ambari-server setup --jdbc-db=bdb --jdbc-driver=/usr/share/je-5.0.73.jar
6.Restart the Ambari server.
ambari-server restart
7.Restart the Falcon service from the Ambari UI.
You need to have administrator privileges in Ambari to restart a service.
1.In the Ambari web UI, click the Services tab and select the Falcon service in the left Services pane.
2.From the Falcon Summary page, click Service Actions > Restart All.
3.Click Confirm Restart All.
When the service is available, the Falcon status displays as Started on the Summary page.
Further information and manual install instructions are available in an article at https://community.hortonworks.com/articles/78274/prerequisite-to-installing-or-upgrading-falcon.html.
Created 04-04-2018 07:26 PM
I have downloaded the jar already. The question I have is since I’ve already setup the ambari server using mysql DB and mysql jdbc driver, would doing a setup again with bdb and it’s jar cause a conflict? Or can I set up ambari using any number of databases like for eg mysql for ambari, derby for oozie, bdb for falcon etc
Created 04-04-2018 07:41 PM
That will run independently it won't interfere with the Mysql or any other database.Falcon doesn't use the classic db's like run. Before you proceed make sure Falcon is not running ,kill any rogue falcon process