Member since
10-03-2017
82
Posts
2
Kudos Received
0
Solutions
05-28-2018
02:17 PM
@GN_Exp If the above answer helped addressed your question, please take a moment to login and click the "accept" link on the answer.
... View more
05-15-2018
06:55 PM
Yes, and having the DB on its own host is recommended if possible.
... View more
05-14-2018
03:54 PM
I am looking to write few files in HDFS via knox. There is one solution I came across i.e. nifi to create dataflow to write files to hdfs. But as per my requirement, I have to write into HDFS via knox. Is there any suggestion how would i do this? And if it is not possible to write into HDFS in Nifi via Knox then is there any alternative solution?
... View more
Labels:
05-17-2018
07:42 PM
@Mohsin Aqee I am also facing similar issue. PutHDFS processor is writing empty files. My HDFS is running in Kubernetes cluster and namenode and datanodes running on different pods in cluster. I am able to connect to namenode with external hostname address for namnenode with this -> hdfs://<Kubernetes-ip>:9000 in core-site.xml. PutHDFS processor not giving me any error if I have this property dfs.client.use.datanode.hostname=true but if it is false then I would get IOException as below: Caused by: org.apache.hadoop.ipc.RemoteException: File /.test.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and 2 node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1547)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:724)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) I think this means its not able to connect to internal hostname in cluster,. Hence I gave external address to datanode port in hdfs-site.xml but still didn't work. I have knox gateway in my cluster too. Do you know if I can write files with webhdfs via knox using Nifi?
... View more
04-16-2018
11:14 PM
@GN_Exp Can you please try this: 1. Create a file as following: "/etc/knox/conf/topologies/ui.xml". and add the following contents to it. Make sure to change the "rangeradmin.example.com:6080" with your Ranger Hostname <topology>
<gateway>
<provider>
<role>authentication</role>
<name>Anonymous</name>
<enabled>true</enabled>
</provider>
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>false</enabled>
</provider>
</gateway>
<service>
<role>AMBARI</role>
<url>http://ambariserver.example.com:8080</url>
</service>
<service>
<role>AMBARIUI</role>
<url>http://ambariserver.example.com:8080</url>
</service>
<service>
<role>RANGER</role>
<url>http://rangeradmin.example.com:6080/</url>
</service>
<service>
<role>RANGERUI</role>
<url>http://rangeradmin.example.com:6080/</url>
</service>
</topology> . 2. Then restart "KNOX" service and then try accessing the following URL to access Ranger Admin UI. Please change the "knoxserver.example.com" with your own knox host URL. https://knoxserver.example.com:8443/gateway/ui/ranger/index.html . Additionally if you want to To make Knox use Ranger authorization then please refer to the following article: https://community.hortonworks.com/articles/114601/how-to-configure-and-troubleshoot-a-knox-topology.html
... View more
04-27-2018
04:15 AM
Yes, the java patches are required to accommodate the changes in new versions.
... View more
05-16-2019
12:01 AM
Did this issue get resolved? I'm using HDP 3.1 with Ranger 1.2.0, and I have the correct Unlimited JCE, but still get this error when using the test connection button.
... View more
03-12-2018
03:25 PM
1 Kudo
I am able to create with the below rest call: curl -ivu admin:admin -H "Content-Type: application/json" -d '{"name":"hdfs-test-service","description":"testing","repositoryType":"hdfs","config":"{\"username\":\"admin\",\"password\":\"admin\",\"fs.default.name\":\"hdfs://<namenode-host>:9000\",\"hadoop.security.authorization\":true,\"hadoop.security.authentication\":\"simple\"}","isActive":true}' -X POST http://<ranger-host>:6080/service/public/api/repository
... View more
03-06-2018
12:59 AM
Hi @GN_Exp, you can update the user configuration from the command line utilities using the scripts under ranger-admin for example to update the password of any user ( including the admin user ) Password Utility :
python changepasswordutil.py <loginID> <currentPassword> <newPassword>
for admin Account :
python changepasswordutil.py admin <currentPassword> <new_password>
to Update the User Name :
python changeusernameutil.py <loginID> <currentPassword> <newUserName>
Hope this help !!
... View more
03-02-2018
12:59 AM
@GN_Exp Ranger is the component that enforces access policies to Hadoop resources in the cluster. Ranger must be configured to sync against local UNIX groups or against and AD/LDAP to have a list users and groups for which you will be creating policies. Ranger has no way of knowing which users and/or groups exist without that configuration. When a user attempts to access a resource like Hive, Ranger will use the current user running the command (say hive from the command line) to see if policies allow or deny the requesting user access. A properly secure cluster would use Kerberos for non-repudiation.
... View more
03-15-2018
09:45 PM
That's correct @GN_Exp. If you want to do SLA in Knox via Ranger plugin then you'd need kerberos too.
... View more
02-22-2018
10:14 PM
Ranger doens't support sync from sql database. But one more option that Ranger usersync supports is to sync from file (which can be of csv, json, or text file format). One option is to dump the users and groups from sql to a file and sync from file. https://community.hortonworks.com/questions/1372/how-to-configure-ranger-usersync-to-sync-users-fro.html
... View more
02-22-2018
10:08 PM
Currently ranger usersync doesn't support multiple LDAP. But once the users and groups are sync'd to Ranger, they are mainly used for configuring policies in Ranger. Ranger doesn't contact AD/LDAP during authorization or policy enforcement. Mostly it relies on hadoop group mapping for during authorization. I am not sure what exactly is the use case you are trying to solve. Can you share some details?
... View more
02-27-2018
06:35 PM
Thanks @spolavarapu. This worked for me.
... View more
12-12-2017
11:19 PM
I am trying to write java program using Knox shell classes. I am interested to knox how SSO can be possible in hadoop sessions or tokens can be passed instead of credentials in Hadoop session ?
... View more
Labels:
12-07-2017
10:17 PM
I am trying to understand complete HDFS file read workflow over http in case of webhdfs. When HTTP client request to read a file, the request goes to Namenode. Namenode responds back to client with the datanode address(block location) with block access token for client's authentication. This reponse of Namenode to client is in the form of redirect and with this redirect client send request to datanodes to read required blocks. Now my question is, how data streams are happening from datanode to client?
... View more
Labels:
12-01-2017
06:46 PM
I am installing Ranger through Ambari but its failing while "Install,Start and Test" as shown in attached picture. Below is the log for the same: stdOUT: 2017-12-01 12:46:53,991 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://mastervm/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='J10007' and active='N' and updated_by='mastervm';"2017-12-01 12:46:55,100 [I] java patch PatchForHiveServiceDefUpdate_J10007 is applied.. 2017-12-01 12:46:55,100 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://mastervm/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'J10008' and active = 'Y';"2017-12-01 12:46:56,163 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://mastervm/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'J10008' and active = 'N';"2017-12-01 12:46:57,237 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://mastervm/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('J10008', now(), 'Ranger 0.7.0.2.6.2.0-205', now(), 'mastervm','N') ;"2017-12-01 12:46:58,431 [I] java patch PatchForTagServiceDefUpdate_J10008 is being applied..2017-12-01 12:47:33,831 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://mastervm/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='J10008' and active='N' and updated_by='mastervm';"2017-12-01 12:47:34,820 [I] java patch PatchForTagServiceDefUpdate_J10008 is applied.. 2017-12-01 12:47:34,821 [JISQL] /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://mastervm/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'J10009' and active = 'Y';" Command failed after 1 tries STDError : Python script has been killed due to timeout after waiting 600 secs
... View more
Labels:
01-12-2018
10:19 AM
It's similar to session maintenance in web - post login, browser receives a cookie which it sends with every request so that it's authenticity session is maintained. So, with knox it should be possible same way (if you can manage to send cookie with each request post the first one). Kerberos is another authentication mechanism so don't think adding the 2 together will help nullify one of it.
... View more
10-25-2017
06:28 PM
This is because 'guest' user in the Demo Ldap does not belong to group "users". For this to work you should create a group "users" in the demo ldap (users.ldif) and add guest user to it. You could try to impersonate 'guest' to a user who belongs to 'users' group on the machine.
... View more
10-25-2017
06:31 PM
Hello @Neha G, you cannot access "webhdfs://" protocol using Knox, Knox only supports Http, Https and WS protocols.
... View more
10-17-2017
04:54 PM
@Neha G In a kerberized cluster there are 2 types of keytabs or principals headless and service principals. Headless principals are not bound to a specific host or node and are presented like @ SRV.COM Service principals are bound to a specific service and host or node, and are presented like with syntax: /@ SRV.COM So when you initialize the hdfs.headless.keytab is as DoAs so the user will take hdfs permissions
... View more
10-17-2017
06:24 PM
Hello @Neha G Sure, this guide should help you http://knox.apache.org/books/knox-0-13-0/user-guide.html#Secure+Clusters
... View more
10-18-2017
04:29 AM
@Neha G The answer to both your questions is Yes (for Active Directory integration and installing Kerberos client). You need to first understand how Kerberos works and how it integrates with Hadoop before attempting to create users and connecting to cluster with users authenticated by Keberos. The following link does a really good job explaining how to setup Kerberos and integrate with Active Directory. https://hortonworks.com/blog/enabling-kerberos-hdp-active-directory-integration/
... View more
10-16-2017
05:04 AM
@Neha G You can use the basic hdfs commands to achieve this. hdfs dfs -copyFromLocal <local file system path> <hdfs path> - to copy files from local to hdfs hdfs dfs -copyToLocal <hdfs path> <local fs path> - copy from hdfs to local hdfs dfs -put <local file system path> <hdfs path> - to copy files from local to hdfs hdfs dfs -get <hdfs path> <local fs path> - copy from hdfs to local hdfs dfs -ls <path> - list the files. Check this link for more commands. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html Make sure to run kinit with hdfs keytab before running the above commands as it is a secure enviroment ( kinit -kt /etc/security/keytabs/hdfs.headless.keytab <principal>) If you don't know the value of principal , you can run the below command and get it [root@xxxxx ~]# klist -kte /etc/security/keytabs/hdfs.headless.keytab
Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 09/10/17 14:06:14 hdfs@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
1 09/10/17 14:06:14 hdfs@EXAMPLE.COM (arcfour-hmac)
1 09/10/17 14:06:14 hdfs@EXAMPLE.COM (des-cbc-md5)
1 09/10/17 14:06:14 hdfs@EXAMPLE.COM (des3-cbc-sha1)
1 09/10/17 14:06:14 hdfs@EXAMPLE.COM (aes256-cts-hmac-sha1-96) In the above output hdfs@EXAMPLE.COM is the principal, so your kinit command would be kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs@EXAMPLE.COM Note : You can also use ambari files view to do these operations. GUI would be easy. Thanks, Aditya
... View more
10-16-2017
04:13 AM
@Neha G, Glad that it worked for you. Can you please accept the answer and start a new thread for this so that the main thread doesn't get deviated. Please share more logs related to the alerts in the new thread. Thanks, Aditya
... View more
10-27-2017
11:28 AM
@Neha You never responded.
... View more
10-13-2018
11:26 AM
This was fixed for me by updating fqdn name to point to domain name by updating /etc/hosts and resolv.conf.
... View more
10-09-2017
03:32 PM
The JOBTRACKER and NAMENODE services are a bit odd at first glance. Don't confuse their use with proxying of RPC services. These are used in order to realize the rewrite requirements for Oozie. Oozie responses include some host and port information that we need to be able to identify internally through the ServiceRegistryService gateway service.
... View more
10-09-2017
03:32 PM
Actually I was looking a way to integrate authorization in Knox as per the user list and their access levels in hdfs in my hadoop cluster. I still need to look into the option you suggested.
... View more