you can set hive.mapred.mode = strict; Quoting from doc: https://blog.cloudera.com/improving-query-performance-using-partitioning-in-apache-hive/ If your partitioned table is very large, you could block any full table scan queries by putting Hive into strict mode using the set hive.mapred.mode=strict command. In this mode, when users submit a query that would result in a full table scan (i.e. queries without any partitioned columns) an error is issued.   ... View more

ERROR 2020Feb19 02:01:21,086 main com.client.engineering.group.JOB.main.JOBMain: org.apache.hadoop.hbase.client.RetriesExhaustedException thrown: Can't get the location On this application which particular table are you trying to access? Did you validate if the user mcaf has permission to access the concerned table (https://docs.cloudera.com/documentation/enterprise/5-14-x/topics/cdh_sg_hbase_authorization.html#topic_8_3_2 has the commands) If there is no permission for the concerned user, grant them required privileges.   If you notice privileges required for mcaf are already provided. Then checking hbase master logs during the issue timeframe would give further clues.   Qn: And do we need to execute 'kinit mcaf' every time before submitting the job ? And how can we configure scheduled jobs ? Ans: Yes and how are you scheduling the jobs? If its a shell script then you can include kinit command with mcaf's keytab which would avoid prompting for password       ... View more

yes, you are in right direction. You can set min.user.id to a value lower value like 500 and then re-submit the job ... View more

Actually we use mcaf as a user to execute the jobs but why http user coming to the picture ? --> By this do you mean, you switch to mcaf unix user[su - mcaf] and then run job? If yes, then its wrong. Post enabling kerberos hdfs and yarn recognises the user by the tgt and not by unix user id. So even if you su to mcaf and then have tgt as different user[say HTTP]. then yarn/hdfs recognises you by that tgt user.   Can you kinit mcaf, then run klist[to ensure you have mcaf tgt] and submit the job?  ... View more

The klist result shows you are submitting job as  HTTP user   hostname.org:~:HADOOP QA]$ klist Ticket cache: FILE:/tmp/krb5cc_251473 Default principal: HTTP/hostname.org@FQDN.COM   WARN security.UserGroupInformation: PriviledgedActionException as:HTTP/hostname.org@FQDN.COM (auth:KERBEROS) cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=HTTP, access=WRITE, inode="/user":mcaf:supergroup:drwxr-xr-x   The above error just implies you don't have write permission for HTTP user on /user directory. So you can either provide write permission for "others" for /user in hdfs so that HTTP user can write or run the job after you kinit as user mcaf which has write permission ... View more

What version of CDH/HDP are you trying this on?   Per the query you shared you are running a insert query and for insert you wont see any results on the console? Are you trying to run another select query after this insert query which is not showing results?    What results do you get for this query   SELECT n.Id, t.job_id, t.job_title, n.Name, n.Email, n.Mobile_Number, n.Education, n.Total_Experiance, n.project_id, ((count(n.new_skills)*100)/s.skill_count) Average FROM rmt_demo.resume_convert n JOIN rmt_demo.job_description_convert t ON n.new_skills = t.skills and n.job_position = t.job_title JOIN rmt_demo.skill_count s ON n.job_position = s.job_title JOIN rmt_demo.education e ON n.education = e.education GROUP BY n.Id,t.job_id,t.job_title, n.Name, n.Email, n.Mobile_Number, n.Education, n.Total_Experiance,n.project_id,s.skill_count;   ... View more

You can pass that as command line argument   Example: hbase org.apache.hadoop.hbase.mapreduce.RowCounter -Dmapreduce.job.cache.files=/test 'hbase_table_t10' hbase org.apache.hadoop.hbase.mapreduce.RowCounter -Dmapreduce.job.cache.files=/test '<table_name>'   ... View more

Sorry, I've not come across any scripts yet. For observability the cluster utilisation report is something that you can review to understand how weightage influenced the load. More details are in this link https://docs.cloudera.com/documentation/enterprise/5-14-x/topics/admin_cluster_util_report.html#concept_edr_ntt_2v ... View more