Knox Implementation with AD/LDAP to access HDFS commands , Hive via beeline and different Web UI's.

[#1]

Note: First made your topology file. Please find an attached example. knox-topology-file.xml

knox-ad-ldap-upgraded-docus.pdf

Above PDF file covered all practical concepts and some theory part.

Step 1:- Install Knox on edge node or any node on the cluster.

Step 2:- Start Knox service from Ambari,make sure your Ambari Server is already sync with LDAP.

Step3:- Search your LDAP Server via below command

ldapsearch -W -H ldap://ad2012.ansari.net -D binduser@ansari.net -b "dc=ansari,dc=net"

ldapsearch -W -H ldaps://ad2012.ansari.net -D [email protected] -b "dc=ansari,dc=net"

Step 4:- Create a master password for Knox:

/usr/hdp/current/knox-server/data/security/keystores/gateway.jks

/usr/hdp/2.6.4.0-91/knox/bin/knoxcli.sh create-master --force

enter password then verify it

Note:-

(2.6.4.0-91 is my HDP versions select your hdp version /usr/hdp/XXXXXXX/)

Step 5: Validate your topology file (your cluster name and toplogy file name should be same):-

/usr/hdp/2.6.0.3-8/knox/bin/knoxcli.sh validate-topology --cluster walhdp

Stpe 6: Validate your auth users:-

sudo /usr/hdp/2.6.4.0-91/knox/bin/knoxcli.sh --d system-user-auth-test --cluster walhdp

Step 7:- Change all below property and restart required services:-

HDFS:- Core-site.xml:

hadoop.proxyuser.knox.groups=*

hadoop.proxyuser.knox.hosts=*

HIVE:-

webhcat.proxyuser.knox.groups=*

webhcat.proxyuser.knox.hosts=*

hive.server2.allow.user.substitution=true

hive.server2.transport.mode=http

hive.server2.thrift.http.port=10001

hive.server2.thrift.http.path=cliservice

Oozie

oozie.service.ProxyUserService.proxyuser.knox.groups=*

oozie.service.ProxyUserService.proxyuser.knox.hosts=*

Step 7 :- Try to access HDFS list status:-

curl -vvv -i -k -u binduser -X GET https://hdp-node1.ansari.net:8443/gateway/walhdp/webhdfs/v1?op=LISTSTATUS

curl -vvv -i -k -u binduser -X GET https://namenodehost:8443/gateway/walhdp(clustername)/webhdfs/v1?op=LISTSTATUS

Step 8:- Try to access hive beeline

!connect jdbc:hive2://hdp node1.ansari.net:8443/;ssl=true;sslTrustStore=/home/faheem/gateway.jks;trustStorePassword=bigdata;transportMode=http;httpPath=gateway/walhdp/hive

entery username: binduser

password for binduser: XXXXXXXXXX

Step 9: To access Web UI’s via knox using below lines:-

Ambari Ui access

https://ambari-server-fqdn-or ambari-server-ip:8443/gateway/walhdp/ambari/

HDFS UI's access

https://namenode-fqdn:8443/gateway/walhdp/hdfs/

HBase access

https://hbase-master-fqdn:8443/gateway/walhdp/hbase/webui/

YARN UI's

https://yarn-master-fqdn:8443/gateway/walhdp/yarn/cluster/apps/RUNNING

Resource Manager:-

https://resource-manager-fqdn:8443/gateway/walhdp/resourcemanager/v1/cluster

curl -ivk -u binduser:Ansari123 " https://hdp-node3.ansari.net:8443/gateway/walhdp/resourcemanager/v1/cluster"

curl -ivk -u binduser:Ansari123" https://localhost:8443/gateway/walhdp/resourcemanager/v1/cluster"

Ranger Web UI's

https://ranger-admin-fqdn:8443/gateway/walhdp/ranger/index.html

OOzie UI's

https://oozie-server-fqdn:8443/gateway/walhdp/oozie/

Zeppline

https://zeppline-fqdn:8443/gateway/walhdp/zeppelin/

Thanks

Ansari Faheem Ahmed

HDPCA Certified