Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Create users in hadoop/HDP 2.5

SOLVED Go to solution
Highlighted

Create users in hadoop/HDP 2.5

New Contributor

I just created users in /home in edge node for users to work on hadoop. But users not able to run hive/Yarn jobs. My cluster is already kerberized. Is there any way to assign users to submit their jobs in edge node.

Do we need to these users in hadoop/hdfs ?

If cluster has kerberos and ranger is there any different way to do it ?

Please suggest me.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Create users in hadoop/HDP 2.5

Mentor

@Sam Red

When your a cluster integrated with Kerberos security then authenticated user must exist in the every node where the task runs. So create the berlin user on all the hosts and add user berlin to the hadoop group that should resolve the problem.

Please revert

8 REPLIES 8

Re: Create users in hadoop/HDP 2.5

Mentor

@Sam Red

For a user to submit successfully or even connect to hive in a kerberized environment the user MUST first get a valid Kerberos ticket.

Can you users list the principals attached to their keytabs eg for user opera1

# su - opera1$ 
$ klist 
klist: No credentials cache found (filename: /tmp/krb5cc_1001)

Above output show no valid kerberos ticket ,then follow the below steps as user opera1 assuming your keytabs are in that directory and the REALM is DEV.COM

$ klist -kt /etc/security/keytabs/opera1.keytab
Keytab name: FILE:/etc/security/keytabs/opera1.keytab
KVNO              Timestamp                    Principal
---- ------------------- ------------------------------------------------------
 1              08/24/2017 18:28:26            opera1@DEV.COM
$ kinit -kt /etc/security/keytabs/opera1.keytab opera1@UGANDA.COM
$ hive

The user should be able now to run a job on the cluster !

Please let me know

Re: Create users in hadoop/HDP 2.5

New Contributor

@Geoffrey Shelton Okot

Thank you again. these is the issue i am getting from beeline.

beeline> !connect jdbc:hive2://hostname.host.com:2181,hostname.host.com:2181,hostname.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 username password
Connected to: Apache Hive (version 1.2.1000.2.5.6.0-40)
Driver: Hive JDBC (version 1.2.1000.2.5.6.0-40)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://host> select max(_TIMESTAMP(ts)) ;
INFO  : Tez session hasn't been created yet. Opening session
ERROR : Failed to execute tez graph.
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_ failed 2 times due to AM Container for appattempt_ exited with  exitCode: -1000

Diagnostics: Application application_ID initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is berlin
main : requested yarn user is berlin
User berlin not found


Failing this attempt. Failing the application.
        at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:779)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:217)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:287)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:166)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1151)
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
        at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
        at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:253)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)
        at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:264)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1)



Re: Create users in hadoop/HDP 2.5

Rising Star

Hi,

Could you please execute your query with beeline --verbose=true.

What happens if the query is run from Hive Cli?

Re: Create users in hadoop/HDP 2.5

Mentor

@Sam Red

When your a cluster integrated with Kerberos security then authenticated user must exist in the every node where the task runs. So create the berlin user on all the hosts and add user berlin to the hadoop group that should resolve the problem.

Please revert

Re: Create users in hadoop/HDP 2.5

New Contributor

@Geoffrey Shelton Okot

Is there any way to add user in all hosts ? Please let me know

Re: Create users in hadoop/HDP 2.5

Contributor

@Geoffrey Shelton Okot

What about if the cluster is not using kerberos (eg. hadoop.security.authentication=local)?

Re: Create users in hadoop/HDP 2.5

Mentor

@Sam Red

Unfortunately, you will have to use the classic way :-) depending on your OS adapt appropriate commands as root, below example is on centos6

# useradd user15
# passwd user15

And repeat that on all the hosts in the cluster, from the Ambari server if you created a passwordless ssh then it's easier!

# ssh root@host5
[root@host5 ~]# useradd user15 
[root@host5 ~]# passwd user15

Tedious work ..... if you have a cluster with 100 nodes!

Re: Create users in hadoop/HDP 2.5

New Contributor

You need to create user directories on HDFS for each user.

$ su - hdfs

Because your cluster is kerberized you need to get a token for the hdfs user. Something like this

$ kinit -k -t /etc/security/keytabs/hdfs.service.keytab hdfs/node1.hadoop.com@HADOOP.COM

Now you can create an HDFS directory for your user.

$ hadoop fs -mkdir /user/berlin
$ hadoop fs -chown berlin /user/berlin

If you're querying Hive, you could use a desktop tool like Aginity Workbench, which supports Kerberos connections.