How to Abort a hung ambari operation

[#1]

In a browser put in:

http://<AMBARI-HOST>:8080/api/v1/clusters/<CLUSTER-NAME>/requests/

Find the highest reqest number in the output. Now, we can go to each request by pasting in the number of the request, an example would be:

http://<AMBARI-HOST>:8080/api/v1/clusters/<CLUSTER-NAME>/requests/110

The output should look like:

{ 
"href" : "http://mon-orlan:8080/api/v1/clusters/test/requests/110", 
"Requests" : { 
"aborted_task_count" : 0, 
"cluster_name" : "test", 
"completed_task_count" : 3, 
"create_time" : 1444340395849, 
"end_time" : 1444340449936, 
"exclusive" : false, 
"failed_task_count" : 0, 
"id" : 110, 
"inputs" : "{}", 
"operation_level" : "SERVICE", 
"progress_percent" : 100.0, 
"queued_task_count" : 0, 
"request_context" : "Restart all components with Stale Configs for HIVE", 
..... 

If that is not your hung operation, go back one by one from the highest request number. What we are looking for is the request context matches your hung Operation, that is the one we want to abort. Once you find that request ID, then go to the ambari server command line as root and do:

curl -u admin:admin -i -H "X-RequestedBy:ambari" -X PUT -d '{"Requests":{"request_status":"ABORTED","abort_reason":"Aborted by user"}}' http://localhost:8080/api/v1/clusters/<CLUSTER_NAME>/requests/<REQUEST_ID>;

[#2]

We are trying to abort one request from ambari-server with this command but it is not working it seems.

Getting following exception in ambari-server:

11:28:25,579  WARN [Thread-20] ActionScheduler:189 - Exception received
javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.4.0.v20120608-r11652): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.BatchUpdateException: Batch entry 0 UPDATE request SET end_time = 1463135305576 WHERE (request_id = 433) was aborted.  Call getNextException to see the cause.
Error Code: 0
Call: UPDATE request SET end_time = ? WHERE (request_id = ?)
   bind => [2 parameters bound]
   at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commitInternal(EntityTransactionImpl.java:102)
   at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:63)
   at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:91)
   at org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.endRequest(ActionDBAccessorImpl.java:302)
   at org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.abortOperation(ActionDBAccessorImpl.java:150)
   at org.apache.ambari.server.actionmanager.ActionScheduler.abortOperationsForStage(ActionScheduler.java:636)
   at org.apache.ambari.server.actionmanager.ActionScheduler.processCancelledRequestsList(ActionScheduler.java:829)
   at org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:203)
   at org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:184)
   at java.lang.Thread.run(Thread.java:745)
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.4.0.v20120608-r11652): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.BatchUpdateException: Batch entry 0 UPDATE request SET end_time = 1463135305576 WHERE (request_id = 433) was aborted.  Call getNextException to see the cause.
Error Code: 0

[#3]

In case ambari API doesn't help. You can use following SQL queries to abort hung/queued/pending operations directly from ambari database.

1) Stop Ambari-Server

ambari-server stop

2) Take backup of ambari database before making any changes to ambari db

3) Check IN_PROGRESS, QUEUED and PENDING operations.

ambari=> select task_id,role,role_command from host_role_command where status='IN_PROGRESS';
ambari=> select task_id,role,role_command from host_role_command where status='QUEUED' limit 100;
ambari=> select task_id,role,role_command from host_role_command where status='PENDING'; 

4) Abort Operations

ambari=> update host_role_command set status='ABORTED' where status='QUEUED';
ambari=> update host_role_command set status='ABORTED' where status='PENDING';
ambari=> update host_role_command set status='ABORTED' where status='IN_PROGRESS';

5) Start ambari-server

ambari-server start