Member since
07-27-2015
7
Posts
0
Kudos Received
0
Solutions
08-11-2015
06:19 AM
We have kerberos enabled in the our dev cluster.. Hadoop has no issues reading/writing to Hbase but Spark on YARN throws all kinds of expection when it tries to read/write from hbase. 15/08/11 13:07:47 WARN security.UserGroupInformation: PriviledgedActionException as:usera (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 15/08/11 13:07:47 WARN ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 15/08/11 13:07:47 ERROR ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:605) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:154) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:731) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:728) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) This is our Hbase code: // Instantiating Configuration class Configuration config = HBaseConfiguration.create(); System.setProperty("java.security.krb5. conf ", "/etc/krb5. conf "); System.setProperty("java.security.auth.login.config", "/etc/ zookeeper / conf /jaas.conf"); config.set("hbase.zookeeper.quorum", "host-node2"); config.set("hbase.zookeeper.property.clientPort", "2181"); UserGroupInformation.setConfiguration( config ); UserGroupInformation.loginUserFromKeytab(" usera @DOMAIN.COM", "/home/ ubuntu /usera.keytab"); // Instantiating HTable class HTable hTable = new HTable( config , "test"); Put p = new Put(Bytes.toBytes("row100")); // adding values using add() method // accepts column family name, qualifier/row name ,value p.add(Bytes.toBytes("cf1"), Bytes.toBytes(columnQualifier),Bytes.toBytes(s)); System.out.println("Added row to Hbase table"); // Saving the put Instance to the HTable. hTable.put(p); hTable.close(); I've given permissions for usera to read/write from test table What seem to be causing this? Any help much appreciated.
... View more
Labels:
08-03-2015
03:14 AM
Thanks for your reply. My intention here is to not allow any users to run spark jobs for example. Since spark runs on Yarn, my assumption was giving only few users access will help secure my cluster and prevent everyone from submitting spark jobs. But even though I set the ACL's , any user is able to submit spark jobs. Any help on how to solve? Thanks, your help is much appreciated.
... View more
08-03-2015
02:34 AM
Hi Wilfred, Can you let me know how I can setup ACL's on the queues for YARN? Thanks.
... View more
07-31-2015
05:32 AM
I tried to blacklist users by putting them in banned users( YARN Configuration->Banned Users List) list but it didn't work. How can I do this in a kerberos enabled cluster? By this, I mean the banning the users from accessing HDFS directories, running spark jobs etc.
... View more
07-30-2015
04:19 AM
I tried to whitelist users by putting them in banned users( YARN Configuration->Banned Users List) list but it didn't work. How can I do this in a kerberos enabled cluster? By this, I mean the banning the users from accessing HDFS directories, running spark jobs etc.
... View more
07-28-2015
05:38 AM
Thanks, that fixed the issue. I opened up both TCP and UDP ports and it's working now.
... View more
- Tags:
- up
07-27-2015
05:59 AM
Hi, We're trying to setup Kerberos on our cloudera cluster. We have managed to setup a working KDC and we even reached till the pen-ultimate step of cluster Kerberos setup wizard when Cloudera complained it couldn't start our services. From what we can tell, the machines are discoverable to each other and we're able to do kinit successfully. Stack Trace: Exception in secureMain java.io.IOException: Login failure for hdfs/domain@realm from keytab hdfs.keytab: javax.security.auth.login.LoginException: Connection refused at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:976) at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:243) at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:207) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2288) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2337) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2514) at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.start(SecureDataNodeStarter.java:79) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243) Caused by: javax.security.auth.login.LoginException: Connection refused at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:767) at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687) at javax.security.auth.login.LoginContext.login(LoginContext.java:595) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:967) ... 11 more Any help would be appreciated.
... View more