Falcon server takes more than 20 minutes to start

[#1]

SYMPTOMS:

Starting Falcon server from Ambari is successful, but when user tries to access Falcon web UI.

Logs show one or more cluster entities trying to initialize:

INFO  - [main:] ~ Initializing FS: hdfs://fc1.support.com:8020 for cluster: falconcluster1

INFO  - [main:] ~ Initializing FS: hdfs://ft2.support.com:8020 for cluster: falcontest2 

Start up is delayed with numerous attempts to contact cluster entity.

2016-12-05 13:49:17,240 INFO  - [main:] ~ Retrying connect to server: ft2.support.com/xxx.xxx.xxx.xx:8020. Already tried 0 time(s); maxRetries=45 (Client:835)
....
....
2016-12-05 14:19:52,971 INFO  - [main:] ~ Retrying connect to server:  ft2.support.com/xxx.xxx.xxx.xx:8020. Already tried 44 time(s); maxRetries=45 (Client:835)
2016-12-05 14:20:13,000 ERROR - [main:] ~ Failed to initialize FS for cluster :  (SharedLibraryHostingService:200)
org.apache.falcon.FalconException: Failed to initialize FS for cluster : falcontest2 
    at org.apache.falcon.service.SharedLibraryHostingService.addLibsTo(SharedLibraryHostingService.java:85)

ROOT CAUSE: Falcon is working as expected. A fix was introduced in HDP 2.3.0 to address:

https://issues.apache.org/jira/browse/FALCON-1165

Previously, if Falcon could not access a cluster entity, it would not restart successfully. Newer versions

WORKAROUND:

Verify that the cluster specified in that particular entity is accessible.

If cluster is not accessible or no longer available, remove cluster entity definition from the falcon store (default location: /hadoop/falcon/store/CLUSTER)