The HBase Book says this about enabling wire-encryption in HBase: Once HBase is configured for secure RPC it is possible to optionally configure encrypted communication. To do so, add the following to the hbase-site.xml file on every client: <property> <name>hbase.rpc.protection</name> <value>privacy</value> </property> What is/are the encryption algorithm(s) used when HBase wire-encryption is enabled? ... View more

In many cases, it's sufficient or even desirable to use self-signed, or company-internal signed certificates to provide TLS encryption, for example between your users' browsers and Ambari and also between Ambari and an LDAP service (LDAPS). If you're not careful with your management of the certificates for each component you're working with, it's possible that you can end up with a broken Ambari views installation. In such cases, when you try to load Ambari views, you might instead see a stack trace which on the surface may appear cryptic, with an error code such as "RA040". If you do see javax.net.ssl.SSLHandshakeException in the trace somewhere, it's likely that your problem lies with the custom trust store in Ambari. Consider this scenario: 1. You set up HTTPS for Ambari using a private CA - e.g. your corporate IT team to sign your certificates so that IT-managed browsers which already have imported the CA certificates do not generate warnings. You used "ambari-server setup-security" for this, choosing option 1 "Enable HTTPS for Ambari Server" 2. You then enabled LDAPS-based authentication, but during the process, unwittingly switch to using a custom keystore because the team that runs LDAP have signed their own certificates has CA-certificate associated with the CA that signed your LDAP server's certificate. You might have done something like this mkdir /etc/ambari-server/keys $JAVA_HOME/bin/keytool -import -trustcacerts -alias root -file /etc/openldap/cacerts/ldap-ca.crt -keystore /etc/ambari-server/keys/custom-keystore.jks Then during "ambari-server setup-ldap" you may have answered as follows Do you want to provide custom TrustStore for Ambari [y/n] (n)?y TrustStore type [jks/jceks/pkcs12] (jks): Path to TrustStore file :/etc/ambari-server/keys/custom-keystore.jks Password for TrustStore: Re-enter password: By switching to the custom keystore, you'll lose the previously established trust that you set up when you set up HTTPS for Ambari. Ambari needs to be able to connect to itself over HTTPS for Views to work and if it can't trust it's own connection you will see the SSL handshake errors. To fix this problem you need to re-import the corporate IT team's CA back into your custom keystore in Ambari. Place a copy of your corporate CA's certificates in a path that's accessible to you, then run "ambari-server setup-security" again, but this time choose option 5 - Import certificate to truststore. Ambari is already configured to use the custom-keystore.jks file, so all you need to do is provide the path to the certificate. [00:48]:[root@ambariserver:~]# ambari-server setup-security Using python /usr/bin/python Security setup options... =========================================================================== Choose one of the following options: [1] Enable HTTPS for Ambari server. [2] Encrypt passwords stored in ambari.properties file. [3] Setup Ambari kerberos JAAS configuration. [4] Setup truststore.<br> [5] Import certificate to truststore. =========================================================================== Enter choice, (1-5): 5 Do you want to configure a truststore [y/n] (y)? y Do you want to import a certificate [y/n] (y)? y Please enter an alias for the certificate: CORP-CA Enter path to certificate: /tmp/CORP-CA.pem Ambari Server 'setup-security' completed successfully. [00:49]:[root@ambariserver:~]# Restart the Ambari server and with a bit of luck your Ambari Views will be working again! ... View more

On Kerberos-secured clusters managed by Ambari versions prior to 2.0.0, the hdfs.headless.keytab file is deployed on all the nodes in the cluster. From Ambari version 2.0.0 onwards, this keytab file is not deployed by default on every node. If the Ranger HBase plugin is enabled and set up to write to HDFS, Ambari's RegionServer start up script tries to access the hdfs.headless.keytab file. If the host that the RegionServer is running on does not also have the HDFS client role assigned to it, the hdfs.headless.keytab file will be absent and the RegionServer will fail to start with an error such as this in Ambari: resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-examplecluster@KRBEXAMPLE.COM' returned 1. kinit: Key table file '/etc/security/keytabs/hdfs.headless.keytab' not found while getting initial credentials This problem is expected to be resolved with the release of Ambari 2.4.0, but until then, the work-around is to install the HDFS client role on all of your RegionServers. There are two challenges to implement this work-around using the Ambari Web UI. The web UI does not allow you to only install the HDFS client role, it forces you to install all the service clients together. The web UI does not allow you to carry out a bulk install of the client role on all of your regionservers in one operation. The Ambari REST API can be used to deploy the HDFS_CLIENT role on all your RegionServer nodes, even if your KDC admin credentials are not stored in Ambari. This article describes how to load your Ambari REST API session with KDC credentials. We'll use the same technique here. The cluster is called examplecluster and the Kerberos realm is KRBEXAMPLE.COM. The first example node that we install the HDFS_CLIENT role on is slavenode01.hdp.example.com. The Ambari administrator username is admin. We are running all the curl commands on the Ambari server itself (ambnode), so we can use localhost in the URL for curl and none of the credentials are traversing the network. Authenticate as an Ambari administrator and get an Ambari session ID (AMBARISESSIONID). [19:11]:[nl@ambnode:~]$ curl -i -u admin -H 'X-Requested-By: ambari' -X GET 'http://localhost:8080/api/v1/clusters' Enter host password for user 'admin': HTTP/1.1 200 OK X-Frame-Options: DENY X-XSS-Protection: 1; mode=block User: admin Set-Cookie: AMBARISESSIONID=xqptqwi8qs4s13wcikxtu5gnb;Path=/;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Content-Type: text/plain Vary: Accept-Encoding, User-Agent Content-Length: 242 Server: Jetty(8.1.17.v20150415) { "href" : "http://localhost:8080/api/v1/clusters", "items" : [ { "href" : "http://localhost:8080/api/v1/clusters/examplecluster", "Clusters" : { "cluster_name" : "examplecluster", "version" : "HDP-2.4" } } ] }[19:11]:[nl@ambnode:~]$ Check the status of KDC administrator credentials in the session. [19:12]:[nl@ambnode:~]$ curl -H 'Cookie: AMBARISESSIONID=xqptqwi8qs4s13wcikxtu5gnb' -H 'X-Requested-By: ambari' -X GET 'http://localhost:8080/api/v1/clusters/examplecluster/services/KERBEROS?fields=Services/attributes/kdc_validation_result,Services/attributes/kdc_validation_failure_details' { "href" : "http://localhost:8080/api/v1/clusters/examplecluster/services/KERBEROS?fields=Services/attributes/kdc_validation_result,Services/attributes/kdc_validation_failure_details", "ServiceInfo" : { "cluster_name" : "examplecluster", "service_name" : "KERBEROS" }, "Services" : { "attributes" : { "kdc_validation_failure_details" : "Missing KDC administrator credentials.\nThe KDC administrator credentials must be set as a persisted or temporary credential resource.This may be done by issuing a POST to the /api/v1/clusters/:clusterName/credentials/kdc.admin.credential API entry point with the following payload:\n{\n \"Credential\" : {\n \"principal\" : \"(PRINCIPAL)\", \"key\" : \"(PASSWORD)\", \"type\" : \"(persisted|temporary)\"}\n }\n}", "kdc_validation_result" : "MISSING_CREDENTIALS" } } }[19:13]:[nl@ambnode:~]$ Load the KDC administrator credentials into the session and check again. [19:14]:[nl@ambnode:~]$ curl -H 'Cookie: AMBARISESSIONID=xqptqwi8qs4s13wcikxtu5gnb' -H 'X-Requested-By: ambari' -X PUT 'http://localhost:8080/api/v1/clusters/examplecluster' -d '[{"session_attributes":{"kerberos_admin":{"principal":"kadmin/admin@KRBEXAMPLE.COM","password":"XXXXXXXXX"}}}]' [19:15]:[nl@ambnode:~]$ [19:15]:[nl@ambnode:~]$ curl -H 'Cookie: AMBARISESSIONID=xqptqwi8qs4s13wcikxtu5gnb' -H 'X-Requested-By: ambari' -X GET 'http://localhost:8080/api/v1/clusters/examplecluster/services/KERBEROS?fields=Services/attributes/kdc_validation_result,Services/attributes/kdc_validation_failure_details' { "href" : "http://localhost:8080/api/v1/clusters/examplecluster/services/KERBEROS?fields=Services/attributes/kdc_validation_result,Services/attributes/kdc_validation_failure_details", "ServiceInfo" : { "cluster_name" : "examplecluster", "service_name" : "KERBEROS" }, "Services" : { "attributes" : { "kdc_validation_failure_details" : "", "kdc_validation_result" : "OK" } } }[19:15]:[nl@ambnode:~]$ Create the HDFS_CLIENT component on the node. [19:19]:[nl@amb-host:~]$ curl -H 'X-Requested-By: ambari' -H 'Cookie: AMBARISESSIONID=xqptqwi8qs4s13wcikxtu5gnb' -X POST "http://localhost:8080/api/v1/clusters/examplecluster/hosts/slavenode01.hdp.example.com/host_components/HDFS_CLIENT" [19:19]:[nl@ambnode:~]$ Check the Ambari web UI - on the node you will see that the HDFS client is added but there will be an exclamation mark next to it. You can add the HDFS_CLIENT component on the rest of the RegionServer nodes using a for loop such as this (where /var/tmp/regionservers contains a list of nodes, one per line, that you want to install the HDFS client role on): for i in $(cat /var/tmp/regionservers) ; do echo $i ; curl -H 'X-Requested-By: ambari' -H 'Cookie: AMBARISESSIONID=xqptqwi8qs4s13wcikxtu5gnb' -X POST "http://localhost:8080/api/v1/clusters/examplecluster/hosts/${i}/host_components/HDFS_CLIENT" ; echo ; done Install the HDFS_CLIENT component on the node. [19:20]:[nl@ambnode:~]$ curl -H 'X-Requested-By: ambari' -H 'Cookie: AMBARISESSIONID=xqptqwi8qs4s13wcikxtu5gnb' -X PUT "http://localhost:8080/api/v1/clusters/examplecluster/hosts/slavenode01.hdp.example.com/host_components/HDFS_CLIENT" -d '[{"RequestInfo": {"context" :"Install HDFS_CLIENT component via REST"}, "Body": {"HostRoles":{"state":"INSTALLED"}}}]' { "href" : "http://localhost:8080/api/v1/clusters/examplecluster/requests/572", "Requests" : { "id" : 572, "status" : "Accepted" } }[19:20]:[nl@ambnode:~]$ Check the Ambari web UI. If you completely reload the page for that host, you should see that the exclamation mark has gone from next to the HDFS Client listing. You will also see in the background operations view of Ambari a Install HDFS_CLIENT component via REST operation. If the operation was successful, you will find that you can now start the RegionServer on the node. Each of these background operations takes some time for the KDC operations to complete so if you use a for loop, it might be advisable to insert a pause after each one. for i in $(cat /var/tmp/regionservers) ; do echo $i ; curl -H 'X-Requested-By: ambari' -H 'Cookie: AMBARISESSIONID=xqptqwi8qs4s13wcikxtu5gnb' -X PUT "http://localhost:8080/api/v1/clusters/examplecluster/hosts/${i}/host_components/HDFS_CLIENT" -d '[{"RequestInfo": {"context" :"Install HDFS_CLIENT component via REST"}, "Body": {"HostRoles":{"state":"INSTALLED"}}}]' ; echo ; echo ; sleep 20 ; done At this point you should find that all HBase components can be started up successfully, with Ranger-based authorization in place. ... View more

In this case, after we set up Hive and other MySQL-dependent services, we decided to hand MySQL responsibilities over to the DBA team so they could carry out specialized management including configuring an HA set-up. We therefore no longer wanted Ambari to run the MySQL server. The MySQL server was shut down and uninstalled from the host outside of Ambari, before we could do this cleanly via Ambari. Because of this, we had to carry out some extra steps to delete the component from Ambari. The following is mostly based on this HCC article by @Kuldeep Kulkarni. The version of Ambari in this case is 2.2.2.1. Deletion of the component from the host appears to have worked. localhost:~ nl$ curl -u admin:admin -X DELETE -H 'X-Requested-By:admin' http://localhost:8080/api/v1/clusters/mycluster/hosts/host005.hadoop.example.com/host_components/MYSQL_SERVER localhost:~ nl$ However, MySQL being stopped and uninstalled outside of Ambari appears to have caused an inconsistency, preventing the second API call from succeeding. localhost:~ nl$ curl -u admin:admin -X DELETE -H 'X-Requested-By:admin' http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER { "status" : 500, "message" : "org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Could not delete service component from cluster. To remove service component, it must be in DISABLED/INIT/INSTALLED/INSTALL_FAILED/UNKNOWN/UNINSTALLED/INSTALLING state., clusterName=mycluster, serviceName=HIVE, componentName=MYSQL_SERVER, current state=STARTED." }localhost:~ nl$ Checking the status via the API shows the following: localhost:~ nl$ curl -u admin:admin -H "X-Requested-By:ambari" -X GET http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER { "href" : "http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER", "ServiceComponentInfo" : { "category" : "MASTER", "cluster_name" : "mycluster", "component_name" : "MYSQL_SERVER", "display_name" : "MySQL Server", "installed_count" : 0, "service_name" : "HIVE", "started_count" : 0, "state" : "STARTED", "total_count" : 0 }, "host_components" : [ ] }localhost:~ nl$ The other article suggests that you can manually update the hostcomponentstate table to allow the DELETE call to succeed. However, in this case there was no such component in that table. ambari=> select * from hostcomponentstate where component_name='MYSQL_SERVER'; id | cluster_id | component_name | version | current_stack_id | current_state | host_id | service_name | upgrade_state | security_state ----+------------+----------------+---------+------------------+---------------+ ---------+--------------+---------------+---------------- (0 rows) ambari=> There was another table servicecomponentdesiredstate that had an entry for this component. ambari=> select * from servicecomponentdesiredstate where component_name='MYSQL_SERVER'; component_name | cluster_id | desired_stack_id | desired_state | service_name ----------------+------------+------------------+---------------+-------------- MYSQL_SERVER | 2 | 4 | STARTED | HIVE (1 row) ambari=> Rather than deleting the row, I updated the desired_state cell to UNINSTALLED. ambari=> update servicecomponentdesiredstate set desired_state='UNINSTALLED' where component_name='MYSQL_SERVER'; UPDATE 1 ambari=> select * from servicecomponentdesiredstate where component_name='MYSQL_SERVER'; component_name | cluster_id | desired_stack_id | desired_state | service_name ----------------+------------+------------------+---------------+-------------- MYSQL_SERVER | 2 | 4 | UNINSTALLED | HIVE (1 row) ambari=> When I queried the Ambari server again to check on the state, it had not changed. localhost:~ nl$ curl -u admin:admin -H "X-Requested-By:ambari" -X GET http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER { "href" : "http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER", "ServiceComponentInfo" : { "category" : "MASTER", "cluster_name" : "mycluster", "component_name" : "MYSQL_SERVER", "display_name" : "MySQL Server", "installed_count" : 0, "service_name" : "HIVE", "started_count" : 0, "state" : "STARTED", "total_count" : 0 }, "host_components" : [ ] }localhost:~ nl$ However, after restarting the Ambari server localhost:~ nl$ sudo /sbin/service ambari-server restart Using python /usr/bin/python Restarting ambari-server Using python /usr/bin/python Stopping ambari-server Ambari Server stopped Using python /usr/bin/python Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start.................... Ambari Server 'start' completed successfully. localhost:~ nl$ the state had changed to UNINSTALLED. localhost:~ nl$ curl -u admin:admin -H "X-Requested-By:ambari" -X GET http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER { "href" : "http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER", "ServiceComponentInfo" : { "category" : "MASTER", "cluster_name" : "mycluster", "component_name" : "MYSQL_SERVER", "display_name" : "MySQL Server", "installed_count" : 0, "service_name" : "HIVE", "started_count" : 0, "state" : "UNINSTALLED", "total_count" : 0 }, "host_components" : [ ] }localhost:~ nl$ I was then able to use the API to delete the component without any error. localhost:~ nl$ curl -u admin:doc2.toaster -X DELETE -H 'X-Requested-By:admin' http://localhost:8080/api/v1/clusters/mycluster/services/HIVE/components/MYSQL_SERVER localhost:~ nl$ ... View more