Migrate Ranger Components Admin, UserSync, tagSync to different host in Ambari

[#1]

Scenarios: There are few scenarios where you may need to migrate ranger components from one host to another host in Ambari.

• Server that is hosting Ranger master components is faulty & down or need to be replaced for newer hardware.
• Reduce the administrative implementation of enabling Ranger HA with load balancer and still have some recovery mechanism to get ranger back online as quickly as possible should you encounter above sitaution.
  
  • Design of Ranger has HA builtin where the policies are cached in every plugin such as namenode will have a copy of policies cached on disk in the event ranger admin is not available. If ranger admin/usersync is down administrators will only lose access to add/modify/delete policies and users. In many of the hadoop clusters, users may not need to have high availability access to Ranger UI. It may be ok to survive for few hours without ranger ui and usersync components as existing policies and users should continue to work.

Method:

Stop Ranger service, stop ambari, back up ambari & ranger database, start ambari, use ambari rest api to remove ranger_admin, usersync host components from old host and change one ranger config and add components back using restapi to new host, Click reinstall in ambari, start admin/usersync, restart rest of the ranger dependent services.

Pre-setup:

• Choose Failover node for admin/usersync
• Ensure firewalls are open between new host and all nodes in the clusters, ldap system and administrators desktop
• Ensure ranger backend database is configured to be accessed from new host. i.e mysql permissions - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_security/content/configuring_database_in...
• If using AD unix account for ranger , ensure it will available for the new host
• Ensure you have Kerberos admin credentials are available
• If using ldaps, Ensure the root and intermediate certs for ldaps are in the truststore for new host. You should be able to use the same trustore jks files from older host.
• If Ranger SSL is enabled, Ensure relevant ranger admin and usersync keystore and truststore jks files are ready to use in new host in the same locations. You won't be able to use the same jks files from older host as the common name would be different for the new host. Ensure the root and intermediate CA(if applicable) is also available in java cacerts. Review Ranger SSL Ambari guide.
• You can choose to have packages ranger_<Version>-admin & ranger_<Version>-usersync are installed ahead to ensure faster migration and prevent any installation issues during exercising failover.
  

  Ex. yum install ranger_2_4_3_0_227-admin ranger_2_4_3_0_227-usersync
   

Migration Exercise :

Below are commands straight from my test cluster exercise. Test Cluster is HDP-2.4.3 with MIT Kerberos Enabled and Ranger SSL enabled. Other HDP versions should work using the same method. Will update/comment again if have tested on higher versions.

Ranger KMS is out of scope of this exercise. We might be able to follow similar steps for it.

Test Cluster Values

export AMBARI_USER=admin
export AMBARI_PASSWD=admin
export AMBARI_URL=http://wg01.wg.com:8080
export CLUSTER_NAME=WG243
export SERVICE=RANGER
export COMPONENT_HOST=wg03.wg.com
export COMPONENT_NEW_HOST=wg01.wg.com

• Ensure Ranger Service is stopped. You may use below command should you need.

curl -ik -u $AMBARI_USER:$AMBARI_PASSWD -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop Ranger via REST"}, "Body": {"ServiceInfo": {"state": "INSTALLED"}}}' "$AMBARI_URL/api/v1/clusters/$CLUSTER_NAME/services/$SERVICE"

• BackUp Ranger Database
• Stop ambari and BackUp Ambari Database

Migrating Ranger Admin Component :

• DELETE RangerAdmin COMPONENT ON the current HOST

• curl -ik -u $AMBARI_USER:$AMBARI_PASSWD -H 'X-Requested-By: ambari' -X DELETE "$AMBARI_URL/api/v1/clusters/$CLUSTER_NAME/hosts/$COMPONENT_HOST/host_components/RANGER_ADMIN"
   

• Add RangerAdmin COMPONENT ON the new HOST

curl -ik -u $AMBARI_USER:$AMBARI_PASSWD -H 'X-Requested-By: ambari' -X POST -d '{"host_components" : [{"HostRoles":{"component_name":"RANGER_ADMIN"}}] }' "$AMBARI_URL/api/v1/clusters/$CLUSTER_NAME/hosts?Hosts/host_name=$COMPONENT_NEW_HOST"

• Head to ambari UI - to the new host page and click Re-install under Ranger admin as shown below:
• Update Configuration Ambari ->Ranger->Configs>Advanced > "External URL" to reflect new hostname
• Start Ranger admin from Ambari UI
• Check ranger admin logs /var/log/ranger/admin/xa_portal.log for any errors.
  • Until HDP 2.4, Due to bug RANGER-1073, you may need to do chown ranger:ranger /etc/ranger/admin/.rangeradmin.jceks.crc on new ranger admin server and restart ranger admin.
• Ensure Ranger Admin UI is accessible then proceed to other components.

Migrating Ranger UserSync Component:

• Delete ranger UserSync component on the current host:

curl -ik -u $AMBARI_USER:$AMBARI_PASSWD -H 'X-Requested-By: ambari' -X DELETE "$AMBARI_URL/api/v1/clusters/$CLUSTER_NAME/hosts/$COMPONENT_HOST/host_components/RANGER_USERSYNC"

• Add RANGER UserSync COMPONENT ON the new HOST

curl -ik -u $AMBARI_USER:$AMBARI_PASSWD -H 'X-Requested-By: ambari' -X POST -d '{"host_components" : [{"HostRoles":{"component_name":"RANGER_USERSYNC"}}] }' "$AMBARI_URL/api/v1/clusters/$CLUSTER_NAME/hosts?Hosts/host_name=$COMPONENT_NEW_HOST"

• Start Ranger UserSync from Ambari UI
• Ensure /var/log/ranger/usersync/usersync.log does not throw any unusual errors and syncs to ldap system every 5 to 10mins.
  • Until HDP 2.4, Due to RANGER-1073, you may need to do chown ranger:ranger /usr/hdp/current/ranger-usersync/conf/.ugsync.jceks.crc on new ranger usersync server and restart
  • One issue I encountered was with following error

  • UnixAuthenticationService [main] - ERROR: Service: UnixAuthenticationService
    java.io.IOException: Keystore was tampered with, or password was incorrect 

  • I moved out the jceks files under usersync conf, it usersync started fine with no errors.
  • mv /etc/ranger/usersync/conf/ugsync.jceks /etc/ranger/usersync/conf/unixauthservice.jks /tmp/

Migrating Ranger TagSync Component

• This component is available starting from HDP 2.5 and can be deleted from Ambari UI under the host page of current host that has tagsync. and then be installed on a different host using "Add" option.
• You may also use this command to remove it using rest api should you need to

• curl -ik -u $AMBARI_USER:$AMBARI_PASSWD -H 'X-Requested-By: ambari' -X DELETE "$AMBARI_URL/api/v1/clusters/$CLUSTER_NAME/hosts/$COMPONENT_HOST/host_components/RANGER_TAGSYNC"

Restart all dependent services such as HDFS, YARN, HBASE, KAFKA etc

• You may start/test with one component restart to ensure it is checking in for policies.
• For example, check namenode log for any errors connecting to new ranger admin host. and confirm from ranger ui audit page, plugin section for latest entry.

Please provide any feedback/comment or even better Vote if this article helped you.

[#2]

For secured/kerberized cluster, we may need to push kdc admin credentials by creating a temp credential store before deleting Ranger Admin component

ADD KDC CREDENTIALS:

# curl -ivk -H "X-Requested-By: ambari" -u $AMBARI_USER:$AMBARI_PASSWD -X POST -d '{ "Credential" : { "principal" : "admin/admin@REALM", "key" : "hadoop", "type" : "temporary" } }' "$AMBARI_URL/api/v1/clusters/$CLUSTER_NAME/credentials/kdc.admin.credential"

GET CREDENTIALS

# curl -ivk -H "X-Requested-By: ambari" -u $AMBARI_USER:$AMBARI_PASSWD -X GET "$AMBARI_URL/api/v1/clusters/$CLUSTER_NAME/credentials/kdc.admin.credential"