Member since
03-24-2015
11
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
12825 | 01-04-2016 03:21 AM |
03-03-2020
07:05 PM
I face this error when build with "mvn clean package -P mpack": [WARNING] The requested profile "mpack" could not be activated because it does not exist.
[ERROR] Failed to execute goal on project das-commons: Could not resolve dependencies for project com.hortonworks.das:das-commons:jar:1.2.0-SNAPSHOT: Failure to find org.apache.hadoop:hadoop-common:jar:3.1.0.3.0.0.0-1634 in <a href="https://repo.maven.apache.org/maven2" target="_blank">https://repo.maven.apache.org/maven2</a> was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
... View more
10-27-2017
12:58 AM
Hi @andrzej_jedrzej, can you explain how can you solve this problem? thank you.
... View more
01-04-2017
01:41 AM
1 Kudo
Still no luck with your method. After more dig, combine with this method works for me: https://community.cloudera.com/t5/Cloudera-Manager-Installation/Kerberos-authentication-from-windows-machine/td-p/29634 Hope help others
... View more
12-08-2016
02:36 AM
Hi KarthikR, Try to recreate oozie credential on your Kerberos server. Then rerun oozie "Install ShareLib" wizard.
... View more
01-06-2016
08:56 PM
Any update on this issue?
... View more
01-04-2016
03:39 AM
I fix this by adding following line: export CDH_MR2_HOME=$HADOOP_HOME in .bash_profile
... View more
01-04-2016
03:21 AM
Hi Harsh, @Harsh J wrote: Could you re-run the command also with the below env set? $ export HADOOP_ROOT_LOGGER=TRACE,console $ export HADOOP_OPTS="-Dsun.security.krb5.debug=true -Djavax.net.debug=ssl" $ hadoop fs -ls / Here is the result: 16/01/04 17:42:07 DEBUG util.Shell: setsid exited with exit code 0
16/01/04 17:42:07 DEBUG conf.Configuration: parsing URL jar:file:/app/hadoop-2.6.0-cdh5.4.5/share/hadoop/common/hadoop-common-2.6.0-cdh5.4.5.jar!/core-default.xml
16/01/04 17:42:07 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@4ae69619
16/01/04 17:42:07 DEBUG conf.Configuration: parsing URL file:/home/user01/yarn-conf/core-site.xml
16/01/04 17:42:07 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@30317bdd
16/01/04 17:42:08 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)], about=, type=DEFAULT, always=false, sampleName=Ops)
16/01/04 17:42:08 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)], about=, type=DEFAULT, always=false, sampleName=Ops)
16/01/04 17:42:08 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[GetGroups], about=, type=DEFAULT, always=false, sampleName=Ops)
16/01/04 17:42:08 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
16/01/04 17:42:08 DEBUG security.Groups: Creating new Groups object
16/01/04 17:42:08 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000; warningDeltaMs=5000
>>>KinitOptions cache name is /tmp/krb5cc_501
>>>DEBUG <CCacheInputStream> client principal is user01@DEVELOPMENT.COM
>>>DEBUG <CCacheInputStream> server principal is krbtgt/DEVELOPMENT.COM@DEVELOPMENT.COM
>>>DEBUG <CCacheInputStream> key type: 23
>>>DEBUG <CCacheInputStream> auth time: Mon Jan 04 17:41:23 WIB 2016
>>>DEBUG <CCacheInputStream> start time: Mon Jan 04 17:41:06 WIB 2016
>>>DEBUG <CCacheInputStream> end time: Tue Jan 05 03:41:23 WIB 2016
>>>DEBUG <CCacheInputStream> renew_till time: Mon Jan 11 17:41:06 WIB 2016
>>> CCacheInputStream: readFlags() FORWARDABLE; RENEWABLE; INITIAL; PRE_AUTH;
16/01/04 17:42:08 DEBUG security.UserGroupInformation: hadoop login
16/01/04 17:42:08 DEBUG security.UserGroupInformation: hadoop login commit
16/01/04 17:42:08 DEBUG security.UserGroupInformation: using kerberos user:user01@DEVELOPMENT.COM
16/01/04 17:42:08 DEBUG security.UserGroupInformation: Using user: "user01@DEVELOPMENT.COM" with name user01@DEVELOPMENT.COM
16/01/04 17:42:08 DEBUG security.UserGroupInformation: failure to login
javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name user01@DEVELOPMENT.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to user01@DEVELOPMENT.COM
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:596)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:812)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:774)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2753)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2745)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:354)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:100)
at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
Caused by: java.lang.IllegalArgumentException: Illegal principal name user01@DEVELOPMENT.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to user01@DEVELOPMENT.COM
at org.apache.hadoop.security.User.<init>(User.java:50)
at org.apache.hadoop.security.User.<init>(User.java:43)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:197)
... 30 more
Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to user01@DEVELOPMENT.COM
at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389)
at org.apache.hadoop.security.User.<init>(User.java:48)
... 32 more
ls: failure to login From logs above shows that kerberos client config is still pointed to default /etc/krb5.conf. I use different path by exporting env variable KRB5_CONFIG. After edit /etc/krb5.conf to the proper value, its now works properly. I can browse HDFS and submit job to YARN. @Harsh J wrote: Is this remote host also carrying the Unlimited JCE policy jars under its JDK, so it may use AES-256 if that is in use? I use JDK from cloudera: jdk1.7.0_67-cloudera Thank you very much Harsh.
... View more
12-22-2015
02:18 AM
Hi, It is possible to access secure cluster from host that not part of the cluster as service (hdfs/yarn/etc) gateway? I've download client configuration from cluster and configure krb5.conf. kinit is succeeded but still unable to connect to hdfs. $ klist
Ticket cache: FILE:/tmp/krb5cc_501
Default principal: user01@DEVELOPMENT.COM
Valid starting Expires Service principal
12/22/15 14:57:07 12/23/15 00:57:11 krbtgt/DEVELOPMENT.COM@DEVELOPMENT.COM
renew until 12/29/15 14:57:07
$ export HADOOP_OPTS="-Dsun.security.krb5.debug=true -Djavax.net.debug=ssl"
$ hadoop fs -ls /
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
>>>KinitOptions cache name is /tmp/krb5cc_501
>>>DEBUG <CCacheInputStream> client principal is user01@DEVELOPMENT.COM
>>>DEBUG <CCacheInputStream> server principal is krbtgt/DEVELOPMENT.COM@DEVELOPMENT.COM
>>>DEBUG <CCacheInputStream> key type: 23
>>>DEBUG <CCacheInputStream> auth time: Tue Dec 22 14:57:11 WIB 2015
>>>DEBUG <CCacheInputStream> start time: Tue Dec 22 14:57:07 WIB 2015
>>>DEBUG <CCacheInputStream> end time: Wed Dec 23 00:57:11 WIB 2015
>>>DEBUG <CCacheInputStream> renew_till time: Tue Dec 29 14:57:07 WIB 2015
>>> CCacheInputStream: readFlags() FORWARDABLE; RENEWABLE; INITIAL; PRE_AUTH;
ls: failure to login
... View more
06-25-2015
12:59 AM
Hi, I have a job that successfully finished in one cluster and failed in other one. How can I compare YARN settings/configurations between those two cluster? Thank you.
... View more
06-16-2015
05:03 AM
Hi, I applied point 1 with following syntax, and still not work: $ hadoop jar JarName.jar ClassName -D oozie.launcher.mapreduce.task.classpath.user.prece dence=true param1 param2 In my case, I have fat-jar with modified parquet lib in it. But cluster always picking default parquet lib in CDH. In standalone mode, my program is running fine.
... View more