Failed on: Service Check Hive Falcon then deleted "Falcon service" while upgrade

[#1]

While upgrade to 2.5.3.40 I am getting Failed on:Service Check Falcon error with below error log

File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 294, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/hdp/current/falcon-client/bin/falcon admin -version' returned 255. ERROR: Unable to initialize Falcon Client object. Cause : Could not authenticate, Authentication failed, URL: http://zld00736.vci.att.com:15000/api/options?user.name=ambariqa&user.name=ambariqa, status: 503, message: Service Unavailable

[#2 · Accepted solution]

@sivareddy akkala

- It is really not a good idea to delete a service once the Upgrade is started. As it might create DB corruption and pending item in the scheduled upgrade tasks items/upgrade groups.

- Is there any specific reason you deleted that service?

- Also which kind of upgrade are you performing? (Rolling or Express Upgrade)?

- Have you run the pre-upgrade service checks (Ambari usually asks when we start performing upgrade) ?

- Can you also share the output of the following URLs?

http://<ambari_server>:8080/api/v1/clusters/<cluster_name>/upgrades

In the above URL whatever is the latest <upgrade_id> take that value and form the following kind of URL and then share that output as well:

http://<ambari_server>:8080/api/v1/clusters/<cluster_name>/upgrades/<upgrade_id>?fields=upgrade_grou...

.

[#3]

@sivareddy akkala

Please check the Falcon logs because it is 503 (Internal Server Error), means Falcon is having some internal server issues.

Check in the logs if you find any ERROR and please share the same.

Also please check the following directory permissions and ownership is correct?

[root@sandbox ~]#  ls -ld /hadoop
drwxr-xr-x 10 root root 110 Apr  6 07:16 /hadoop

[root@sandbox ~]# ls -ld /hadoop/falcon/
drwxr-xr-x 5 falcon root 46 Sep 15  2016 /hadoop/falcon/

[root@sandbox ~]# ls -ld /hadoop/falcon/data
drwxr-xr-x 3 root root 20 Sep 15  2016 /hadoop/falcon/data

[root@sandbox ~]# ls -ld /hadoop/falcon/data/lineage/
drwxrwxr-x 3 falcon hadoop 20 Sep 15  2016 /hadoop/falcon/data/lineage/

[root@sandbox ~]# ls -ld /hadoop/falcon/data/lineage/graphdb/
drwxrwxr-x 2 falcon hadoop 54 Sep 15  2016 /hadoop/falcon/data/lineage/graphdb/

[root@sandbox ~]# ll /hadoop/falcon/data/lineage/graphdb/je.lck
-rw-r--r-- 1 falcon hadoop 0 Sep 13  2016 /hadoop/falcon/data/lineage/graphdb/je.lck

.

[#4]

HI Jay,

I have deleted Falcon since I am not using this component 3 hours back, but I delete falcon while doing upgrade now it not allowing me to moving forward or backward.

And It showing below message from last 3 hours.

Upgrade in Progress 96% Waiting to execute the next task...

[#5 · Accepted solution]

@sivareddy akkala

- It is really not a good idea to delete a service once the Upgrade is started. As it might create DB corruption and pending item in the scheduled upgrade tasks items/upgrade groups.

- Is there any specific reason you deleted that service?

- Also which kind of upgrade are you performing? (Rolling or Express Upgrade)?

- Have you run the pre-upgrade service checks (Ambari usually asks when we start performing upgrade) ?

- Can you also share the output of the following URLs?

http://<ambari_server>:8080/api/v1/clusters/<cluster_name>/upgrades

In the above URL whatever is the latest <upgrade_id> take that value and form the following kind of URL and then share that output as well:

http://<ambari_server>:8080/api/v1/clusters/<cluster_name>/upgrades/<upgrade_id>?fields=upgrade_grou...

.

[#6 · Accepted solution]

HI Jay,

I Fond the issue with given two URLS and get request_id and stage_id values for "status" : "PENDING" then fixed the issue.

- Connected ambari DB and issued the below command then continued the Upgrade.

update host_role_command set status='COMPLETED' where request_id='523' and stage_id='107';