About nhgodwal

falbani · ‎05-28-2018

@GN_Exp If the above answer helped addressed your question, please take a moment to login and click the "accept" link on the answer.

vperiasamy · ‎05-15-2018

Yes, and having the DB on its own host is recommended if possible.

nhgodwal · ‎05-14-2018

I am looking to write few files in HDFS via knox. There is one solution I came across i.e. nifi to create dataflow to write files to hdfs. But as per my requirement, I have to write into HDFS via knox. Is there any suggestion how would i do this? And if it is not possible to write into HDFS in Nifi via Knox then is there any alternative solution?

nhgodwal · ‎05-17-2018

@Mohsin Aqee I am also facing similar issue. PutHDFS processor is writing empty files. My HDFS is running in Kubernetes cluster and namenode and datanodes running on different pods in cluster. I am able to connect to namenode with external hostname address for namnenode with this -> hdfs://<Kubernetes-ip>:9000 in core-site.xml. PutHDFS processor not giving me any error if I have this property dfs.client.use.datanode.hostname=true but if it is false then I would get IOException as below: Caused by: org.apache.hadoop.ipc.RemoteException: File /.test.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and 2 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1547) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:724) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) I think this means its not able to connect to internal hostname in cluster,. Hence I gave external address to datanode port in hdfs-site.xml but still didn't work. I have knox gateway in my cluster too. Do you know if I can write files with webhdfs via knox using Nifi?

jsensharma · ‎04-16-2018

@GN_Exp Can you please try this: 1. Create a file as following: "/etc/knox/conf/topologies/ui.xml". and add the following contents to it. Make sure to change the "rangeradmin.example.com:6080" with your Ranger Hostname <topology> <gateway> <provider> <role>authentication</role> <name>Anonymous</name> <enabled>true</enabled> </provider> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>false</enabled> </provider> </gateway> <service> <role>AMBARI</role> <url>http://ambariserver.example.com:8080</url> </service> <service> <role>AMBARIUI</role> <url>http://ambariserver.example.com:8080</url> </service> <service> <role>RANGER</role> <url>http://rangeradmin.example.com:6080/</url> </service> <service> <role>RANGERUI</role> <url>http://rangeradmin.example.com:6080/</url> </service> </topology> . 2. Then restart "KNOX" service and then try accessing the following URL to access Ranger Admin UI. Please change the "knoxserver.example.com" with your own knox host URL. https://knoxserver.example.com:8443/gateway/ui/ranger/index.html . Additionally if you want to To make Knox use Ranger authorization then please refer to the following article: https://community.hortonworks.com/articles/114601/how-to-configure-and-troubleshoot-a-knox-topology.html

vperiasamy · ‎04-27-2018

Yes, the java patches are required to accommodate the changes in new versions.

massoudm · ‎05-16-2019

Did this issue get resolved? I'm using HDP 3.1 with Ranger 1.2.0, and I have the correct Unlimited JCE, but still get this error when using the test connection button.

nhgodwal · ‎03-12-2018

I am able to create with the below rest call: curl -ivu admin:admin -H "Content-Type: application/json" -d '{"name":"hdfs-test-service","description":"testing","repositoryType":"hdfs","config":"{\"username\":\"admin\",\"password\":\"admin\",\"fs.default.name\":\"hdfs://<namenode-host>:9000\",\"hadoop.security.authorization\":true,\"hadoop.security.authentication\":\"simple\"}","isActive":true}' -X POST http://<ranger-host>:6080/service/public/api/repository

bkosaraju · ‎03-06-2018

Hi @GN_Exp, you can update the user configuration from the command line utilities using the scripts under ranger-admin for example to update the password of any user ( including the admin user ) Password Utility : python changepasswordutil.py <loginID> <currentPassword> <newPassword> for admin Account : python changepasswordutil.py admin <currentPassword> <new_password> to Update the User Name : python changeusernameutil.py <loginID> <currentPassword> <newUserName> Hope this help !!

myoung · ‎03-02-2018

@GN_Exp Ranger is the component that enforces access policies to Hadoop resources in the cluster. Ranger must be configured to sync against local UNIX groups or against and AD/LDAP to have a list users and groups for which you will be creating policies. Ranger has no way of knowing which users and/or groups exist without that configuration. When a user attempts to access a resource like Hive, Ranger will use the current user running the command (say hive from the command line) to see if policies allow or deny the requesting user access. A properly secure cluster would use Kerberos for non-repudiation.

VR46 · ‎03-15-2018

That's correct @GN_Exp. If you want to do SLA in Knox via Ranger plugin then you'd need kerberos too.

spolavarapu · ‎02-22-2018

Ranger doens't support sync from sql database. But one more option that Ranger usersync supports is to sync from file (which can be of csv, json, or text file format). One option is to dump the users and groups from sql to a file and sync from file. https://community.hortonworks.com/questions/1372/how-to-configure-ranger-usersync-to-sync-users-fro.html

spolavarapu · ‎02-22-2018

Currently ranger usersync doesn't support multiple LDAP. But once the users and groups are sync'd to Ranger, they are mainly used for configuring policies in Ranger. Ranger doesn't contact AD/LDAP during authorization or policy enforcement. Mostly it relies on hadoop group mapping for during authorization. I am not sure what exactly is the use case you are trying to solve. Can you share some details?

nhgodwal · ‎02-27-2018

Thanks @spolavarapu. This worked for me.

nhgodwal · ‎12-12-2017

I am trying to write java program using Knox shell classes. I am interested to knox how SSO can be possible in hadoop sessions or tokens can be passed instead of credentials in Hadoop session ?

nhgodwal · ‎12-07-2017

I am trying to understand complete HDFS file read workflow over http in case of webhdfs. When HTTP client request to read a file, the request goes to Namenode. Namenode responds back to client with the datanode address(block location) with block access token for client's authentication. This reponse of Namenode to client is in the form of redirect and with this redirect client send request to datanodes to read required blocks. Now my question is, how data streams are happening from datanode to client?

nhgodwal · ‎12-01-2017

I am installing Ranger through Ambari but its failing while "Install,Start and Test" as shown in attached picture. Below is the log for the same: stdOUT: 2017-12-01 12:46:53,991 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://mastervm/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='J10007' and active='N' and updated_by='mastervm';"2017-12-01 12:46:55,100 [I] java patch PatchForHiveServiceDefUpdate_J10007 is applied.. 2017-12-01 12:46:55,100 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://mastervm/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'J10008' and active = 'Y';"2017-12-01 12:46:56,163 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://mastervm/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'J10008' and active = 'N';"2017-12-01 12:46:57,237 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://mastervm/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('J10008', now(), 'Ranger 0.7.0.2.6.2.0-205', now(), 'mastervm','N') ;"2017-12-01 12:46:58,431 [I] java patch PatchForTagServiceDefUpdate_J10008 is being applied..2017-12-01 12:47:33,831 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://mastervm/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='J10008' and active='N' and updated_by='mastervm';"2017-12-01 12:47:34,820 [I] java patch PatchForTagServiceDefUpdate_J10008 is applied.. 2017-12-01 12:47:34,821 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://mastervm/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'J10009' and active = 'Y';" Command failed after 1 tries STDError : Python script has been killed due to timeout after waiting 600 secs

pchaudhari · ‎01-12-2018

It's similar to session maintenance in web - post login, browser receives a cookie which it sends with every request so that it's authenticity session is maintained. So, with knox it should be possible same way (if you can manage to send cookie with each request post the first one). Kerberos is another authentication mechanism so don't think adding the 2 together will help nullify one of it.

smore · ‎10-25-2017

This is because 'guest' user in the Demo Ldap does not belong to group "users". For this to work you should create a group "users" in the demo ldap (users.ldif) and add guest user to it. You could try to impersonate 'guest' to a user who belongs to 'users' group on the machine.

smore · ‎10-25-2017

Hello @Neha G, you cannot access "webhdfs://" protocol using Knox, Knox only supports Http, Https and WS protocols.

Shelton · ‎10-17-2017

@Neha G In a kerberized cluster there are 2 types of keytabs or principals headless and service principals. Headless principals are not bound to a specific host or node and are presented like @ SRV.COM Service principals are bound to a specific service and host or node, and are presented like with syntax: /@ SRV.COM So when you initialize the hdfs.headless.keytab is as DoAs so the user will take hdfs permissions

smore · ‎10-17-2017

Hello @Neha G Sure, this guide should help you http://knox.apache.org/books/knox-0-13-0/user-guide.html#Secure+Clusters

mqureshi · ‎10-18-2017

@Neha G The answer to both your questions is Yes (for Active Directory integration and installing Kerberos client). You need to first understand how Kerberos works and how it integrates with Hadoop before attempting to create users and connecting to cluster with users authenticated by Keberos. The following link does a really good job explaining how to setup Kerberos and integrate with Active Directory. https://hortonworks.com/blog/enabling-kerberos-hdp-active-directory-integration/

asirna · ‎10-16-2017

@Neha G You can use the basic hdfs commands to achieve this. hdfs dfs -copyFromLocal <local file system path> <hdfs path> - to copy files from local to hdfs hdfs dfs -copyToLocal <hdfs path> <local fs path> - copy from hdfs to local hdfs dfs -put <local file system path> <hdfs path> - to copy files from local to hdfs hdfs dfs -get <hdfs path> <local fs path> - copy from hdfs to local hdfs dfs -ls <path> - list the files. Check this link for more commands. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html Make sure to run kinit with hdfs keytab before running the above commands as it is a secure enviroment ( kinit -kt /etc/security/keytabs/hdfs.headless.keytab <principal>) If you don't know the value of principal , you can run the below command and get it [root@xxxxx ~]# klist -kte /etc/security/keytabs/hdfs.headless.keytab Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 1 09/10/17 14:06:14 hdfs@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 1 09/10/17 14:06:14 hdfs@EXAMPLE.COM (arcfour-hmac) 1 09/10/17 14:06:14 hdfs@EXAMPLE.COM (des-cbc-md5) 1 09/10/17 14:06:14 hdfs@EXAMPLE.COM (des3-cbc-sha1) 1 09/10/17 14:06:14 hdfs@EXAMPLE.COM (aes256-cts-hmac-sha1-96) In the above output hdfs@EXAMPLE.COM is the principal, so your kinit command would be kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs@EXAMPLE.COM Note : You can also use ambari files view to do these operations. GUI would be easy. Thanks, Aditya

asirna · ‎10-16-2017

@Neha G, Glad that it worked for you. Can you please accept the answer and start a new thread for this so that the main thread doesn't get deviated. Please share more logs related to the alerts in the new thread. Thanks, Aditya

Shelton · ‎10-27-2017

@Neha You never responded.

rajat_dua09 · ‎10-13-2018

This was fixed for me by updating fqdn name to point to domain name by updating /etc/hosts and resolv.conf.

lmccay · ‎10-09-2017

The JOBTRACKER and NAMENODE services are a bit odd at first glance. Don't confuse their use with proxying of RPC services. These are used in order to realize the rewrite requirements for Oozie. Oozie responses include some host and port information that we need to be able to identify internally through the ServiceRegistryService gateway service.

nhgodwal · ‎10-09-2017

Actually I was looking a way to integrate authorization in Knox as per the user list and their access levels in hdfs in my hadoop cluster. I still need to look into the option you suggested.

Online	Offline
Last Visited	‎10-18-2018 10:31 PM

Date Registered	‎10-03-2017 09:03 PM
Last Visited	‎10-18-2018 10:31 PM
Total Messages Posted	82
Total Kudos Received	2

Re: How can I install Ranger policymanger on one n...

Re: Is it possible to have Ranger admin ui on one ...

Is there any support in Apache Nifi to write to we...

Re: NIFI: PutHDFS processor writing 0 bytes

Re: Can we use Ranger ui via Knox?

Re: What are these java patches getting applied at...

Re: Ranger admin error when creating service repo ...

Re: How to create service in hdfs-plugin with Rest...

Re: Ranger command-line installation -> How to set...

Re: Help me understand LDAP/AD group level authori...

Re: Ranger Knox Plugin is failing in test connecti...

Re: Can Ranger sync the users and groups from sql ...

Re: How can I integrate multiple LDAP with Ranger ...

Re: Ranger usersync service not able to sync LDAP ...

How SSO works with Knox Shell classes?

Can someone help me understand how block reads hap...

Ranger is failing at "Install,Start and Test" at R...

Re: Please help me understand how Knoxsso setup wo...

Re: Can someone help me understand Knox impersonat...

Re: Can we use webhdfs:// as a KNOX scheme instead...

Re: In Kerberos setting, in a HDP, how to confirm ...

Re: I have a kerberized hadoop cluster (single nod...

Re: How can I access my single node hadoop cluster...

Re: I recently deployed single node Hadoop cluster...

Re: NameNode not starting after enabling Kerberos ...

Re: Failed to create principal - hadoop@domain - c...

Re: Failed to connect to KDC - Failed to communica...

Re: How Apache Knox gets integrated with data encr...

Re: Can I export HDFS access control list as authe...