map reduce 2.0 throwing error after enabling kerberos security in cloudera 5.4.3.

[#1]

I am getting the below error after enbaling kerberos security in CDH 5.4.3.

hive> select count(*) from hive_test;
Query ID = hdfs_20150810165757_e7420efe-67e7-4a75-bf78-0d1383f7cc09
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1439195152382_0012, Tracking URL = http://hdp-poc2.tcshydnextgen.com:8088/proxy/application_1439195152382_0012/
Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1439195152382_0012
Hadoop job information for Stage-1: number of mappers: 0; number of reducers: 0
2015-08-10 16:57:19,271 Stage-1 map = 0%,  reduce = 0%
Ended Job = job_1439195152382_0012 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1:  HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

[#2]

There is something really strange with this job.

The job should at least do something but the submitted job shows: "number of mappers: 0; number of reducers: 0"

A count(*) should at least have 1 mapper (go over the input) and 1 reducer (to sum up).

Does this happen for every hive job?

Can you run a simple MapReduce example like pi on the cluster to make sure that the cluster works at that level?

Wilfred

[#3]

Hi Wilfred,

Many thanks for the reply.

-bash-4.1$ hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 10 100
Number of Maps  = 10
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
15/08/18 15:57:14 INFO client.RMProxy: Connecting to ResourceManager at hdp-poc2.tcshydnextgen.com/10.138.90.72:8032
15/08/18 15:57:14 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 40 for hdfs on ha-hdfs:nameservice1
15/08/18 15:57:14 INFO security.TokenCache: Got dt for hdfs://nameservice1; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice1, Ident: (HDFS_DELEGATION_TOKEN token 40 for hdfs)
15/08/18 15:57:14 INFO input.FileInputFormat: Total input paths to process : 10
15/08/18 15:57:14 INFO mapreduce.JobSubmitter: number of splits:10
15/08/18 15:57:15 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1439544552504_0019
15/08/18 15:57:15 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice1, Ident: (HDFS_DELEGATION_TOKEN token 40 for hdfs)
15/08/18 15:57:15 INFO impl.YarnClientImpl: Submitted application application_1439544552504_0019
15/08/18 15:57:15 INFO mapreduce.Job: The url to track the job: http://hdp-poc2.tcshydnextgen.com:8088/proxy/application_1439544552504_0019/
15/08/18 15:57:15 INFO mapreduce.Job: Running job: job_1439544552504_0019
15/08/18 15:57:17 INFO mapreduce.Job: Job job_1439544552504_0019 running in uber mode : false
15/08/18 15:57:17 INFO mapreduce.Job:  map 0% reduce 0%
15/08/18 15:57:17 INFO mapreduce.Job: Job job_1439544552504_0019 failed with state FAILED due to: Application application_1439544552504_0019 failed 2 times due to AM Container for appattempt_1439544552504_0019_000002 exited with  exitCode: -1000
For more detailed output, check application tracking page:http://hdp-poc2.tcshydnextgen.com:8088/proxy/application_1439544552504_0019/Then, click on links to logs of each attempt.
Diagnostics: Application application_1439544552504_0019 initialization failed (exitCode=255) with output: Requested user hdfs is not whitelisted and has id 493,which is below the minimum allowed 1000

Failing this attempt. Failing the application.
15/08/18 15:57:17 INFO mapreduce.Job: Counters: 0
Job Finished in 3.528 seconds
java.io.FileNotFoundException: File does not exist: hdfs://nameservice1/user/hdfs/QuasiMonteCarlo_1439893630990_2073090894/out/reduce-out
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1132)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
    at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1750)
    at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1774)
    at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
    at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
-bash-4.1$

Note: I am learning Hadoop and I am a newbie to Hadoop. and as a part of learning I am executing various jobs.

Here are the actions performed:

1) MR job has been executed with hdfs user and su  command is used to login as hdfs user and kinit hdfs command has been executed before MR job command

has been executed.

Above error is observed.

From the above it is clear that hdfs is not a whitelisted user as its uidNumber is < 1000.

Is this whitelist error responsible for the hive query failure? and when hive query is executed with hdfs user then error2 is reported.

If my observation is correct may I know why hive query result is not giving exact root cause of the failure i.e. white listed error message which is

observed during MR job execution.

This is not related to above query, but could you please provide answer to our query..

2) With a local user a hive query is executed from hue and also from hive directly.

without issuing kinit command hive query is successful from hue but not from hive.

So, my question is:

Are operations performed from hue are independent of kerberos security? If yes, may I know the reason for the same?

[#4 · Accepted solution]

Depending on how you have set up yarn hive should be part of the "allowed.system.users" list for the NM's that list will white list all system users below the "min.user.id".

There is also a list of "banned.users" that lists all users that are not allowed to run containers.

All these three need to be in sync to allow running a container.

The hdfs user should not be allowed since it is the superuser and could circumvent the HDFS access permissions.

When you execute a job from hue authentication is taken  care of by hue. It will make sure that some kind of kerberos initialisation is performed.

Wilfred