Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Create users in hadoop/HDP 2.5

avatar
Contributor

I just created users in /home in edge node for users to work on hadoop. But users not able to run hive/Yarn jobs. My cluster is already kerberized. Is there any way to assign users to submit their jobs in edge node.

Do we need to these users in hadoop/hdfs ?

If cluster has kerberos and ranger is there any different way to do it ?

Please suggest me.

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Sam Red

When your a cluster integrated with Kerberos security then authenticated user must exist in the every node where the task runs. So create the berlin user on all the hosts and add user berlin to the hadoop group that should resolve the problem.

Please revert

View solution in original post

10 REPLIES 10

avatar
Master Mentor

@Sam Red

For a user to submit successfully or even connect to hive in a kerberized environment the user MUST first get a valid Kerberos ticket.

Can you users list the principals attached to their keytabs eg for user opera1

# su - opera1$ 
$ klist 
klist: No credentials cache found (filename: /tmp/krb5cc_1001)

Above output show no valid kerberos ticket ,then follow the below steps as user opera1 assuming your keytabs are in that directory and the REALM is DEV.COM

$ klist -kt /etc/security/keytabs/opera1.keytab
Keytab name: FILE:/etc/security/keytabs/opera1.keytab
KVNO              Timestamp                    Principal
---- ------------------- ------------------------------------------------------
 1              08/24/2017 18:28:26            opera1@DEV.COM
$ kinit -kt /etc/security/keytabs/opera1.keytab opera1@UGANDA.COM
$ hive

The user should be able now to run a job on the cluster !

Please let me know

avatar
Contributor

@Geoffrey Shelton Okot

Thank you again. these is the issue i am getting from beeline.

beeline> !connect jdbc:hive2://hostname.host.com:2181,hostname.host.com:2181,hostname.host.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 username password
Connected to: Apache Hive (version 1.2.1000.2.5.6.0-40)
Driver: Hive JDBC (version 1.2.1000.2.5.6.0-40)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://host> select max(_TIMESTAMP(ts)) ;
INFO  : Tez session hasn't been created yet. Opening session
ERROR : Failed to execute tez graph.
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_ failed 2 times due to AM Container for appattempt_ exited with  exitCode: -1000

Diagnostics: Application application_ID initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is berlin
main : requested yarn user is berlin
User berlin not found


Failing this attempt. Failing the application.
        at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:779)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:217)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:287)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:166)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1151)
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
        at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
        at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:253)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)
        at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:264)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1)



avatar
Rising Star

Hi,

Could you please execute your query with beeline --verbose=true.

What happens if the query is run from Hive Cli?

avatar
Master Mentor

@Sam Red

When your a cluster integrated with Kerberos security then authenticated user must exist in the every node where the task runs. So create the berlin user on all the hosts and add user berlin to the hadoop group that should resolve the problem.

Please revert

avatar
Contributor

@Geoffrey Shelton Okot

Is there any way to add user in all hosts ? Please let me know

avatar
Expert Contributor

@Geoffrey Shelton Okot

What about if the cluster is not using kerberos (eg. hadoop.security.authentication=local)?

avatar
Contributor

Hello 🙂 I have the same issue!

I have integrated the edge node with Active directory users could connect and submit theirs jobs to yarn before enabling Kerberos on the cluster.

Actually I have used samba on edge node to create users folders and get information about users 

 

Now I configured the Kerberos and so I am getting the same error user1 not found , user1 is in AD 

should I now add this user with normal command add user on alll nodes ? How could it be as AD user and not local one ? I did not configured samba on others nodes may I do it ? 

thanks a lot in advance 

avatar

@asmarz,

As this is an older thread which was previously marked 'Solved', you would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity to provide details specific to your environment (for example, what happened once you added the affected user accounts with "normal" command add user on all nodes)  that could aid others in providing a more relevant, accurate answer to your question.

 

 

Bill Brooks, Community Moderator
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Master Mentor

@Sam Red

Unfortunately, you will have to use the classic way 🙂 depending on your OS adapt appropriate commands as root, below example is on centos6

# useradd user15
# passwd user15

And repeat that on all the hosts in the cluster, from the Ambari server if you created a passwordless ssh then it's easier!

# ssh root@host5
[root@host5 ~]# useradd user15 
[root@host5 ~]# passwd user15

Tedious work ..... if you have a cluster with 100 nodes!