this Article explain the additional steps required to configure wire encryption by exporting/importing the certificates across the cluster for distcp to work on wire encrypted multi cluster . Problem: on wire encrypted multi cluster environment distcp fails if steps given in this article are not performed, we may see ssl error as follows: javax.net.ssl.SSLHandshakeException: DestHost:destPort <KMS_HOST>:9393 , LocalHost:localPort null:0. Failed on local exception: javax.net.ssl.SSLHandshakeException: Error while authenticating with endpoint: https://<KMS_HOST>e:9393/kms/v1/?op=GETDELEGATIONTOKEN&rene.   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) Prerequisites: 1) two cluster should be setup with Ranger 2) wire encryption should be enabled on the both clusters already. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/enabling-ssl-for-components.html 3) if Ranger kms is installed then wire encryption should be enabled for ranger kms too in both the clusters. steps to configure SSL for distcp to work in multi cluster: 1) export the certificate from Hadoop server key store file on all the host part of the cluster1 and cluster2. cd <server_hadoop_key_location>;keytool -export -alias hadoop_cert_<host_name> -keystore <keystore_file_path> -rfc -file hadoop_cert_<host_name> -storepass <keystore_password> Note: if you don't know the location of the key store, then you can search for config "ssl.server.keystore.location" in the hdfs config 2) copy all the certificates generated for cluster1 in previous step from cluster1 hosts to client key location on all the hosts part of cluster2. and similarly copy all the certificates generated for cluster2, from cluster2 hosts to client key location on all the host part of cluster1 3) Import the all the cluster1 certificates to the hadoop client trustore on all the host of cluster2 and vice versa. cd <client_hadoop_key_location>;keytool import -noprompt -alias hadoop_cert_<host_name> -file hadoop_cert_<host_name> -keystore <truststore_file_path> -storepass <truststore_password> Note: if you don't know the location of the truststore, then you can search for config "ssl.client.truststore.location" in the hdfs config Additional steps if Ranger Kms is installed: if ranger kms is installed then we need to export the ranger kms certificate from ranger kms hosts of cluster1 to Hadoop client trust store of cluster2 1) export the certificate from Ranger kms server key store file on kms hosts part of the cluster1 and cluster2. cd <kms_key_store_location>;keytool -export -alias kms_cert_<host_name> -keystore <kms_keystore_file_path> -rfc -file kms_cert_<host_name> -storepass <kms_keystore_password> Note: if you don't know the location of the kms key store, then you can search for config "ranger.https.attrib.keystore.file" in the kms config 2) copy all the certificates generated for kms in cluster1 in previous step from cluster1 kms hosts to client key location on all the hosts part of cluster2. and similarly copy all the certificates generated for kms in cluster2, from cluster2 kms hosts to client key location on all the host part of cluster1 3) Import all the cluster1 kms certificates to the Hadoop client trust store on all the host of cluster2 and vice versa. cd <client_hadoop_key_location>;keytool import -noprompt -alias kms_cert_<host_name> -file kms_cert_<host_name> -keystore <truststore_file_path> -storepass <truststore_password> Now restart Hdfs, Yarn, Mapreduce and Ranger KMS on both the cluster and once both the services successfully started, try distcp it should work fine. hadoop distcp -Dmapreduce.job.hdfs-servers.token-renewal.exclude=cluster1 -skipcrccheck -update /distcp_cluster1 hdfs://cluster2/distcp_cluster2/ ... View more

prerequisites: 1) You must have an MySQL Server database instance running to be used by Ranger. 2) Execute the following command on the Ambari Server host. Replace database-type with mysql|oracle|postgres|mssql|sqlanywhere and /jdbc/driver/path based on the location of corresponding JDBC driver: ambari-server setup --jdbc-db={database-type} --jdbc-driver={/jdbc/driver/path} 3) make sure root user have access to the db from ranger host and ranger kms host. eg: your ranger host is ranger.host.com & ranger kms host is ranger.kms.host.com then you should run following command on the mysql database: GRANT ALL ON *.* TO 'root'@'ranger.host.com' IDENTIFIED BY '<root_password>' WITH GRANT OPTION; GRANT ALL ON *.* TO 'root'@'ranger.kms.host.com' IDENTIFIED BY '<root_password>' WITH GRANT OPTION; flush privileges; Steps: 1) go to the ambari and go to the add service wizard 2) select the ranger and ranger kms host 3) make sure you fill following properties carefully in Ranger & Ranger KMS both: DB_FLAVOR = mysql Ranger DB host = <db_host> eg: test.mysql.com Setup Database and Database User = yes Database Administrator (DBA) username = root Database Administrator (DBA) password = <root_password> 4) any name can be configured to following db or user properties as they will be created by root user as fresh database or user since we selected "Setup Database and Database User" to yes Ranger DB name = ranger Ranger DB username = rangeradmin Ranger DB password = rangeradmin 5) Ranger KMS has an additional property: KMS master key password: <kms_password> , this is also newly configured password of your choice. 6)) audit can configured based on your choice if you want audit for service operations, Note: I have not given details of all the properties in this article because these are the important properties where people make mistakes. , ... View more

Background: when it comes to rely completely on ranger, and if you are specific about configuring authorization for a resource to an end user then you have to create one policy for each resource. There should be a way to configure a policy that provide access to specific resources based on the User who is making call. Soultion: {USER} Support: {USER} support solves this problem , It allows us to create a policy where we can configure resource as {USER} Eg. /user/{USER} and select user also as {USER}. that means all users will get access of their corresponding home directory. Eg. Hdfs: resource: /user/{USER} user1 will have access to /user/user1 user2 will have access to /user/user2 Hive: resource: database:database_{USER} user1 will have access to database database_user1 user2 will have access to database database_user2 resource may contrain {USER} partially or fully. delimiter can be customised also , Steps to configure {USER}: 1) go to ranger admin, and create policy page, there on resource give {USER} as input. 2) in user type {USER} and {USER} will populate , just select it and add the policy more details can be found at https://cwiki.apache.org/confluence/display/RANGER/Support+for+%24username+variable. ... View more

Background: prior to HDP2.5 release , people used to look for a solution to sync the users from multiple organisation unit. but this was not possible, so in HDP2.5 this feature is released to support syncing the users from multiple OU. How to configure the multiple OU: OU can be configured in the same way as it was earlier , but if you want to configure multiple OU then it should semicolon(;) separated as follows: ou=Executives,DC=abc,DC=com;ou=Engineering,DC=abc,DC=com Sample User setting: Sample Group setting: ... View more

we should follow the given steps for remote debugging of the ranger: 1. clone the incubator ranger code to your local machine & Import the Apache Ranger project into eclipse. http://ranger.apache.org/quick_start_guide.html 2. Now lets say you want to debug the Ranger Admin process then go to the machine where Ranger Admin process is running. edit the /usr/bin/ranger-admin and add the entry as follows: JAVA_OPTS="$JAVA_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9000" keep suspend to n , if nothing need to be debug in startup flow. keep suspend to y, if ranger startup flow need to be debug. in this case process will wait for eclipse to connect to the debug port before it starts 3. check whether debug port 9000 is open for ranger admin process netstat -tanlp | grep 16776 tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 16776/java 4. go to eclipse and in your project Debug as -> Debug Configurations -> Remote Java Applications and there provide the host and port. host should be the node ip/hostname where ranger admin is running port is the debug port we have configured in JAVA_OPTS ( in our case it is 9000) and click on debug. now eclipse client will be connected to the running process. 5. Now lets say we want to debug the search user flow in ranger. then we can put debug point at the respective point and perform the user search operation on ranger UI then we can see flow will come to the debug point we had put. please see the screenshot. ranger user search operation: Debug point: then we can further step over/into and debug the flow. Note: same way we can debug other ranger process such as usersync. - that is it in Remote debugging Ranger process, please comment your questions if any. ... View more