Removing the Hive MySQL component from Ambari

[#1]

In this case, after we set up Hive and other MySQL-dependent services, we decided to hand MySQL responsibilities over to the DBA team so they could carry out specialized management including configuring an HA set-up. We therefore no longer wanted Ambari to run the MySQL server.

The MySQL server was shut down and uninstalled from the host outside of Ambari, before we could do this cleanly via Ambari. Because of this, we had to carry out some extra steps to delete the component from Ambari.

The following is mostly based on this HCC article by @Kuldeep Kulkarni. The version of Ambari in this case is 2.2.2.1.

Deletion of the component from the host appears to have worked.

localhost:~ nl$ curl -u admin:admin -X DELETE -H 'X-Requested-By:admin' http://localhost:8080/api/v1/clusters/mycluster/hosts/host005.hadoop.example.com/host_components/MYS...
localhost:~ nl$

However, MySQL being stopped and uninstalled outside of Ambari appears to have caused an inconsistency, preventing the second API call from succeeding.

localhost:~ nl$ curl -u admin:admin -X DELETE -H 'X-Requested-By:admin' http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER
{
  "status" : 500,
  "message" : "org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Could not delete service component from cluster. To remove service component, it must be in DISABLED/INIT/INSTALLED/INSTALL_FAILED/UNKNOWN/UNINSTALLED/INSTALLING state., clusterName=mycluster, serviceName=HIVE, componentName=MYSQL_SERVER, current state=STARTED."
}localhost:~ nl$

Checking the status via the API shows the following:

localhost:~ nl$ curl -u admin:admin -H "X-Requested-By:ambari" -X GET  http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER
{
  "href" : "http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER",
  "ServiceComponentInfo" : {
    "category" : "MASTER",
    "cluster_name" : "mycluster",
    "component_name" : "MYSQL_SERVER",
    "display_name" : "MySQL Server",
    "installed_count" : 0,
    "service_name" : "HIVE",
    "started_count" : 0,
    "state" : "STARTED",
    "total_count" : 0
  },
  "host_components" : [ ]
}localhost:~ nl$

The other article suggests that you can manually update the hostcomponentstate table to allow the DELETE call to succeed. However, in this case there was no such component in that table.

ambari=> select * from hostcomponentstate where component_name='MYSQL_SERVER';
 id | cluster_id | component_name | version | current_stack_id | current_state |
 host_id | service_name | upgrade_state | security_state 
----+------------+----------------+---------+------------------+---------------+
---------+--------------+---------------+----------------
(0 rows)

ambari=>

There was another table servicecomponentdesiredstate that had an entry for this component.

ambari=> select * from servicecomponentdesiredstate where component_name='MYSQL_SERVER';
 component_name | cluster_id | desired_stack_id | desired_state | service_name 
----------------+------------+------------------+---------------+--------------
 MYSQL_SERVER   |          2 |                4 | STARTED       | HIVE
(1 row)

ambari=>

Rather than deleting the row, I updated the desired_state cell to UNINSTALLED.

ambari=> update servicecomponentdesiredstate set desired_state='UNINSTALLED' where component_name='MYSQL_SERVER';
UPDATE 1
ambari=> select * from servicecomponentdesiredstate where component_name='MYSQL_SERVER';
 component_name | cluster_id | desired_stack_id | desired_state | service_name 
----------------+------------+------------------+---------------+--------------
 MYSQL_SERVER   |          2 |                4 | UNINSTALLED   | HIVE
(1 row)

ambari=>

When I queried the Ambari server again to check on the state, it had not changed.

localhost:~ nl$ curl -u admin:admin -H "X-Requested-By:ambari" -X GET  http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER
{
  "href" : "http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER",
  "ServiceComponentInfo" : {
    "category" : "MASTER",
    "cluster_name" : "mycluster",
    "component_name" : "MYSQL_SERVER",
    "display_name" : "MySQL Server",
    "installed_count" : 0,
    "service_name" : "HIVE",
    "started_count" : 0,
    "state" : "STARTED",
    "total_count" : 0
  },
  "host_components" : [ ]
}localhost:~ nl$

However, after restarting the Ambari server

localhost:~ nl$ sudo /sbin/service ambari-server restart
Using python  /usr/bin/python
Restarting ambari-server
Using python  /usr/bin/python
Stopping ambari-server
Ambari Server stopped
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start....................
Ambari Server 'start' completed successfully.
localhost:~ nl$

the state had changed to UNINSTALLED.

localhost:~ nl$ curl -u admin:admin -H "X-Requested-By:ambari" -X GET  http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER
{
  "href" : "http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER",
  "ServiceComponentInfo" : {
    "category" : "MASTER",
    "cluster_name" : "mycluster",
    "component_name" : "MYSQL_SERVER",
    "display_name" : "MySQL Server",
    "installed_count" : 0,
    "service_name" : "HIVE",
    "started_count" : 0,
    "state" : "UNINSTALLED",
    "total_count" : 0
  },
  "host_components" : [ ]
}localhost:~ nl$

I was then able to use the API to delete the component without any error.

localhost:~ nl$ curl -u admin:doc2.toaster -X DELETE -H 'X-Requested-By:admin' http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER
localhost:~ nl$