About krajguru

krajguru · ‎07-03-2017

@Rishi Currently if your cluster in not kerberised, any user can just export the HADOOP_USER_NAME variable and can perform any activities., there is no way to restrict that. For example : [kunal@s261 ~]$ hdfs dfs -ls /mapred Found 1 items drwxr-xr-x - hdfs hdfs 0 2017-04-24 11:33 /mapred/system [kunal@s261 ~]$ hdfs dfs -ls /mapred/system [kunal@s261 ~]$ [kunal@s261 ~]$ [kunal@s261 ~]$ [kunal@s261 ~]$ hdfs dfs -rmr /mapred/system rmr: DEPRECATED: Please use 'rm -r' instead. 17/04/26 14:30:56 WARN fs.TrashPolicyDefault: Can't create trash directory: hdfs://s261.openstacklocal:8020/user/kunal/.Trash/Current/mapred org.apache.hadoop.security.AccessControlException: Permission denied: user=kunal, access=WRITE, inode="/user/kunal/.Trash/Current/mapred":hdfs:hdfs:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) Then if you export the above variable, we can delete the file : [kunal@s261 ~]$ export HADOOP_USER_NAME=hdfs [kunal@s261 ~]$ [kunal@s261 ~]$ [kunal@s261 ~]$ hdfs dfs -rmr /mapred/system rmr: DEPRECATED: Please use 'rm -r' instead. 17/04/26 14:31:15 INFO fs.TrashPolicyDefault: Moved: 'hdfs://s261.openstacklocal:8020/mapred/system' to trash at: hdfs://s261.openstacklocal:8020/user/hdfs/.Trash/Current/mapred/system The only way is to setup kerberos which can fix this issue, even if you export the variable the user is derived from the kerberos principal : [root@krajguru-e1 ~]# kinit kunal Password for kunal@LAB.HORTONWORKS.NET: [root@krajguru-e1 ~]# [root@krajguru-e1 ~]# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: kunal@LAB.HORTONWORKS.NET Valid starting Expires Service principal 07/03/2017 12:24:39 07/03/2017 22:24:39 krbtgt/LAB.HORTONWORKS.NET@LAB.HORTONWORKS.NET renew until 07/10/2017 12:24:34 [root@krajguru-e1 ~]# [root@krajguru-e1 ~]# hdfs dfs -ls /mapred/ Found 1 items drwxr-xr-x - hdfs hdfs 0 2017-04-21 11:47 /mapred/system [root@krajguru-e1 ~]# [root@krajguru-e1 ~]# export HADOOP_USER_NAME=hdfs [root@krajguru-e1 ~]# [root@krajguru-e1 ~]# hdfs dfs -rmr /mapred/system rmr: DEPRECATED: Please use 'rm -r' instead. 17/07/03 12:25:11 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 360 minutes, Emptier interval = 0 minutes. rmr: Failed to move to trash: hdfs://e1.openstacklocal:8020/mapred/system: Permission denied: user=kunal, access=WRITE, inode="/mapred/system":mapred:hdfs:drwxr-xr-x

krajguru · ‎06-30-2017

@amankumbare Ambari does not set the samaccountname while creating service principals, its AD which randomly populates the value, and if I'm not wrong ambari does not need samaccountname for service principals.

rpathak · ‎03-20-2017

@krajguru You can't run MR or Tez jobs using the REST API. The YARN REST API is meant for developers of applications such as Distributed shell, MR and Tez, not for users who submit applications. However for spark jobs on yarn REST api can be used as given here. https://community.hortonworks.com/articles/28070/starting-spark-jobs-directly-via-yarn-rest-api.html

krajguru · ‎12-31-2016

PROBLEM : Currently in Ranger Audit UI (HDP 2.4) , we do not have a feature or a search filter so that one can pull out a report which answers the question "who made a change to a particular policy" without scrolling through all the pages of audit. A search by Policy Id / Policy Name should solve that. RESOLUTION : We have a Internal Feature request raised to track the same

krajguru · ‎12-22-2016

PROBLEM Running sqoop import command in direct mode for accessing a Netezza data warehouse appliance hangs at 100% map sqoop import --options-file sqoop_opts_file.opt . . . INFO mapreduce.Job: Running job: job_1465914632244_0005 INFO mapreduce.Job: Job job_1465914632244_0005 running in uber mode : false INFO mapreduce.Job: map 0% reduce 0% INFO mapreduce.Job: map 25% reduce 0% INFO mapreduce.Job: map 50% reduce 0% INFO mapreduce.Job: map 100% reduce 0% The sqoop_opts_file.opt had the following options : -connect jdbc:netezza://xxxxxxxxxxxxxxxxxxxxxx:5480/ --username XXXX --password *************** --direct --direct-split-size 1000000 --compress --table table_name --target-dir /user/root/table_name --verbose Yarn logs show the below errors ERROR [Thread-14] org.apache.sqoop.mapreduce.db.netezza.NetezzaJDBCStatementRunner: Unable to execute external table export org.netezza.error.NzSQLException: ERROR: found delim ',' in a data field, specify escapeChar '\' option in the external table definition RESOLUTION : Add --input-escaped-by '\' parameter to sqoop command and then run the command

arullaldivakar1 · ‎12-20-2018

Hello, Is there a property to add anonymous bind in the custom configuration or so ? Regards Arullaldivakar

krajguru · ‎12-22-2016

PROBLEM : The example workflow submitted by users was failing with the below ClassNotFound exceptions java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.example.SampleMapper not found ROOT CAUSE : These classes are found in oozie-examples-<version>.jar file, which is expected to be present in the lib folder of the job. RESOLUTION : Add the oozie-examples-<version>.jar file to the lib folder of the job and then submit the job.

krajguru · ‎12-22-2016

PROBLEM When we query hbase tables through hive, it always creates a fetch task instead of running a MR task. The parameterhive.fetch.task.conversion.threshold controls whether a fetch task would run or a Map Reduce. If hive.fetch.task.conversion.thresholdis less than the table size, it will use MapReduce Job. The default value of the above parameter is 1GB. Create a 'hbase_hive' external table in hive, make sure the hbase table is more than 1GB. [root@node1 ~]# hadoop fs -du -s -h /apps/hbase/data/data/default/hbase-hive 3.4 G /apps/hbase/data/data/default/hbase-hive From beeline analyze the explain plan, which launches a fetch task instead of Map Reduce job, even when the size of the table is more than 1GB 0: jdbc:hive2://node1.hwxblr.com:10000/> explain select * from hbase_hive where key = '111111A111111' ; +----------------------------------------------------------------------------------------------------------+--+ | Explain | +----------------------------------------------------------------------------------------------------------+--+ | STAGE DEPENDENCIES: | | Stage-0 is a root stage | | | | STAGE PLANS: | | Stage: Stage-0 | | Fetch Operator | | limit: -1 | ROOT CAUSE The reason for this behavior is that the fetch task conversion means initiate a local task (inside the client itself) instead of submitting a job to the cluster. For Hive on Hbase table, it does not have any stats and hence the return value would always be less than the fetch task conversion size and would launch the local task at client side. RESOLUTION Query the table by setting the hive.fetch.task.conversion to 'minimal' before executing the query for Hive hbase tables. Do not set this property permanently in hive-site.xml to 'minimal'.

majmaj0707 · ‎05-18-2016

If you can, it would be easier to start over the installation from clean/fresh nodes. dien gia dung noi com dien

Online	Offline
Last Visited	‎08-14-2019 06:45 PM

Member Since	‎05-17-2016 11:12 AM
Last Visited	‎08-14-2019 06:45 PM
Posts	46
Kudos received	22

Cloudera Community

Cloudera Community

Re: How name node tackles data node failure in Had...

Re: Knox is converting a space in the folder name ...

Re: How can I get the ranger audits from solr (Amb...

Re: How does the Ambari Kerberization Wizard gener...

Re: cannot see the audits for hdfs user being audi...

Re: How to prevent users from modifying HADOOP_USE...

Re: How does the Ambari Kerberization Wizard gener...

Re: Can we submit a Yarn job through REST ?

Ranger Audit UI lacks the feature to search the au...

Sqoop import for Netezza connector in direct mode ...

Re: Issues when configuring Ranger usersync to use...

Example Oozie workflow submitted fails with ClassN...

Querying Hbase tables through Hive always creates ...

Re: HOW TO REMOVE USERS while installing hadoop