question Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2 in Archives of Support Questions (Read Only)

Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

balusu — Fri, 16 Sep 2022 13:23:39 GMT

Users are synced to hosts as user@example.com, I can do hadoop fs -ls as hdfs user with out a problem, But when I tried as a user from AD I am getting the error "INFO util.KerberosName: No auth_to_local rules applied to user@userexample.com

Here is the complete log:

[exampleuser@example.com@explehost1 ~]$ hadoop fs -ls
18/06/28 02:20:56 DEBUG util.Shell: setsid exited with exit code 0
18/06/28 02:20:56 DEBUG conf.Configuration: parsing URL jar:file:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hadoop-common-2.6.0-cdh5.14.2.jar!/core-default.xml
18/06/28 02:20:56 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@271053e1
18/06/28 02:20:56 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/conf.cloudera.YARN/core-site.xml
18/06/28 02:20:56 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@5bc79255
18/06/28 02:20:56 DEBUG core.Tracer: sampler.classes = ; loaded no samplers
18/06/28 02:20:56 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FsShell/hostiP"
18/06/28 02:20:56 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FsShell/hostiP)
18/06/28 02:20:56 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers
18/06/28 02:20:56 TRACE core.Tracer: Created Tracer(FsShell/hostiP) for FsShell
18/06/28 02:20:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, always=false, sampleName=Ops, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
18/06/28 02:20:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, always=false, sampleName=Ops, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
18/06/28 02:20:56 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, always=false, sampleName=Ops, type=DEFAULT, valueName=Time, value=[GetGroups])
18/06/28 02:20:56 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, always=false, sampleName=Ops, type=DEFAULT, valueName=Time, value=[Renewal failures since startup])
18/06/28 02:20:56 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, always=false, sampleName=Ops, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login])
18/06/28 02:20:56 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
18/06/28 02:20:56 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
18/06/28 02:20:56 DEBUG security.Groups: Creating new Groups object
18/06/28 02:20:56 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=1000; warningDeltaMs=5000
18/06/28 02:20:56 DEBUG security.UserGroupInformation: hadoop login
18/06/28 02:20:56 DEBUG security.UserGroupInformation: hadoop login commit
18/06/28 02:20:56 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: exampleuser@example.com
18/06/28 02:20:56 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: exampleuser@example.com" with name exampleuser@example.com
18/06/28 02:20:56 INFO util.KerberosName: No auth_to_local rules applied to exampleuser@example.com
18/06/28 02:20:56 DEBUG security.UserGroupInformation: User entry: "exampleuser@example.com"
18/06/28 02:20:56 DEBUG security.UserGroupInformation: UGI loginUser:exampleuser@example.com (auth:SIMPLE)
18/06/28 02:20:56 DEBUG core.Tracer: sampler.classes = ; loaded no samplers
18/06/28 02:20:56 TRACE core.TracerId: ProcessID(fmt=%{tname}/%{ip}): computed process ID of "FSClient/hostiP"
18/06/28 02:20:56 TRACE core.TracerPool: TracerPool(Global): adding tracer Tracer(FSClient/hostiP)
18/06/28 02:20:56 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers
18/06/28 02:20:56 TRACE core.Tracer: Created Tracer(FSClient/hostiP) for FSClient
18/06/28 02:20:56 DEBUG hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
18/06/28 02:20:56 DEBUG hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = false
18/06/28 02:20:56 DEBUG hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
18/06/28 02:20:56 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = /var/run/hdfs-sockets/dn
18/06/28 02:20:56 DEBUG hdfs.DFSClient: Sets dfs.client.block.write.replace-datanode-on-failure.min-replication to 0
18/06/28 02:20:56 TRACE security.SecurityUtil: Name lookup for namenode XX.XX.XX.XX18/06/28 02:20:56 TRACE security.SecurityUtil: Name lookup for spmbaexampleuser.example.com took 0 ms.
18/06/28 02:20:56 DEBUG hdfs.HAUtil: No HA service delegation token found for logical URI hdfs://nameservice1
18/06/28 02:20:56 DEBUG hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
18/06/28 02:20:56 DEBUG hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = false
18/06/28 02:20:56 DEBUG hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
18/06/28 02:20:56 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = /var/run/hdfs-sockets/dn
18/06/28 02:20:56 DEBUG retry.RetryUtils: multipleLinearRandomRetry = null
18/06/28 02:20:56 DEBUG ipc.Server: rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWrapper, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@534a5a98
18/06/28 02:20:56 DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@1e6a3214
18/06/28 02:20:56 DEBUG azure.NativeAzureFileSystem: finalize() called.
18/06/28 02:20:56 DEBUG azure.NativeAzureFileSystem: finalize() called.
18/06/28 02:20:57 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
18/06/28 02:20:57 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
18/06/28 02:20:57 DEBUG unix.DomainSocketWatcher: org.apache.hadoop.net.unix.DomainSocketWatcher$2@6a8b3053: starting with interruptCheckPeriodMs = 60000
18/06/28 02:20:57 TRACE unix.DomainSocketWatcher: DomainSocketWatcher(337574644): adding notificationSocket 168, connected to 167
18/06/28 02:20:57 DEBUG util.PerformanceAdvisory: Both short-circuit local reads and UNIX domain socket are disabled.
18/06/28 02:20:57 DEBUG sasl.DataTransferSaslUtil: DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
18/06/28 02:20:57 TRACE ipc.ProtobufRpcEngine: 1: Call -> namenode/XX.XX.XX.XX:8020: getFileInfo {src: "/user/exampleuser@example.com"}
18/06/28 02:20:57 DEBUG ipc.Client: The ping interval is 60000 ms.
18/06/28 02:20:57 DEBUG ipc.Client: Connecting to namenode/XX.XX.XX.XX:8020
18/06/28 02:20:57 DEBUG ipc.Client: IPC Client (1029472813) connection to namenode/XX.XX.XX.XX:8020 from exampleuser@example.com: starting, having connections 1
18/06/28 02:20:57 DEBUG ipc.Client: IPC Client (1029472813) connection to namenode/XX.XX.XX.XX:8020 from exampleuser@example.com sending #0 org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo
18/06/28 02:20:59 DEBUG ipc.Client: IPC Client (1029472813) connection to namenode/XX.XX.XX.XX:8020 from exampleuser@example.com got value #0
18/06/28 02:20:59 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 2343ms
18/06/28 02:20:59 TRACE ipc.ProtobufRpcEngine: 1: Response <- namenode/XX.XX.XX.XX:8020: getFileInfo {}
ls: `.': No such file or directory
18/06/28 02:20:59 TRACE core.TracerPool: TracerPool(Global): removing tracer Tracer(FsShell/hostiP)
18/06/28 02:20:59 DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@1e6a3214
18/06/28 02:20:59 DEBUG ipc.Client: removing client from cache: org.apache.hadoop.ipc.Client@1e6a3214
18/06/28 02:20:59 DEBUG ipc.Client: stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@1e6a3214
18/06/28 02:20:59 DEBUG ipc.Client: Stopping client
18/06/28 02:20:59 DEBUG ipc.Client: IPC Client (1029472813) connection to namenode/XX.XX.XX.XX:8020 from exampleuser@example.com: closed
18/06/28 02:20:59 DEBUG ipc.Client: IPC Client (1029472813) connection to namenode/XX.XX.XX.XX:8020 from exampleuser@example.com: stopped, remaining connections 0

In need of a serious help, Thanks in advance!

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

bgooley — Thu, 28 Jun 2018 03:48:32 GMT

Hi @balusu,

Actually, the error in your log snippet is:

18/06/28 02:20:56 INFO util.KerberosName: No auth_to_local rules applied to exampleuser@example.com.

This error occurs when no rules in your "hadoop.security.auth_to_local" property in the server's core-site.xml matched the principal, "exampleuser@example.com"

This is not a kerberos error; rather, this is a message being returned by hadoop code when hadoop tries to map your principal to a unix user name.

Generally, if you are attempting to act on a hadoop service with a user who is not in the hadoop cluster's Kerberos realm, you need to make sure that the hadoop.security.auth_to_local property includes rules that will match the principal and convert the string to just a username. Cloudera Manager will create such rules for you if you add the other realm to the "Trusted Realms" or "Trusted Kerberos Realms" configuration.

see:

https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cm_sg_kerbprin_to_sn.html

Note that you will need to deploy client configuration and restart the cluster after making this change.

-Ben

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

balusu — Thu, 28 Jun 2018 03:56:55 GMT

Hi Ben,

Thanks for the quick reply. I have already tried that but the error remains same:

Trusted Kerberos Realms: Example.COM
Additional Rules to Map Kerberos Principals to Short Names: RULE:[1:$1](sbalusu\..*)s/sbalusu\.(.*)/$1/g

Thanks & Regards,
Siva

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

balusu — Thu, 28 Jun 2018 04:57:00 GMT

Hi Ben,

I had the pattern wrong in the rule, Here is the updated and working one:

RULE:[1:$1@$0](.*@\EXAMPLE.COM)s/@\EXAMPLE.COM//
RULE:[2:$1@$0](.*@\EXAMPLE.COM)s/@\EXAMPLE.COM//

Thanks & Regards,

Siva

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

bgooley — Thu, 28 Jun 2018 05:02:13 GMT

That's great news.

To avoid any confusion, the automatically generated auth_to_local rules (based on a realm listed in "Trusted Kerbreros Realms" would look like this:

RULE:[1:$1@$0](.*@\QEXAMPLE.COM\E$)s/@\QEXAMPLE.COM\E$//
RULE:[2:$1@$0](.*@\QEXAMPLE.COM\E$)s/@\QEXAMPLE.COM\E$//

It appears that perhaps some of your characters were interpretted as special when you printed the generaged rules.

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

balusu — Thu, 28 Jun 2018 05:09:56 GMT

I tried exactly same but it threw the below error

Failed to start namenode.
java.util.regex.PatternSyntaxException: Illegal/unsupported escape sequence near index 22
.*@\EXAMPLE.COM\E$
^

also i had to add example.com to make it work, Can you please suggest if there is a way to ignore case in the rule.

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

bgooley — Thu, 28 Jun 2018 16:06:12 GMT

@balusu,

As mentioned, you would want to add the realm to the HDFS configuration "Trusted Kerberos Realms". This will allow Cloudera Manager to generate the necessary auth_to_local rule for that realm.

The regex you used is, indeed, not correct as you have two "\E" but no "\Q" to match.

I am not sure, exactly, what trouble you had with the case of realms, but the realm should always be in uppercase format.

For more information on regex, etc., this is a great resource:

https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cdh_sg_kerbprin_to_sn.html#topic_19_1

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

balusu — Thu, 28 Jun 2018 18:36:34 GMT

@bgooley

Ya, The pattern is wrong and I am glad that the documentation link you provided is very clear.

I observed an interesting thing in our environment:

When only example.com as trusted domain:
[sbalusu@example.com@host ~]$ hadoop org.apache.hadoop.security.HadoopKerberosName sbalusu@example.com
Name: sbalusu@example.com to sbalusu
[sbalusu@example.com@host ~]$ hadoop org.apache.hadoop.security.HadoopKerberosName sbalusu@EXAMPLE.COM
Name: sbalusu@EXAMPLE.COM to sbalusu

When only EXAMPLE.COM as trusted domain:
[sbalusu@example.com@host ~]$ hadoop org.apache.hadoop.security.HadoopKerberosName sbalusu@EXAMPLE.COM
Name: sbalusu@EXAMPLE.COM to sbalusu
[sbalusu@example.com@host ~]$ hadoop org.apache.hadoop.security.HadoopKerberosName sbalusu@example.com
18/06/28 17:41:41 INFO util.KerberosName: No auth_to_local rules applied to sbalusu@example.com
Name: sbalusu@example.com to sbalusu@example.com

and kinit shows EXAMPLE.COM
[sbalusu@example.com@host ~]$ kinit
Password for sbalusu@EXAMPLE.COM:

Not sure from where hadoop is picking lower case realm.

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

bgooley — Thu, 05 Jul 2018 15:20:02 GMT

@balusu,

Can you clarify what you are trying to test with lower case realms? The realm in the kerberos principal should be uppercase, so the lower case is not expected.

If you "kinit" make certain you specify the realm in uppercase.

The auth_to_local rules are not intended to match a lowercase realm, so the response you get is expected.

-Ben

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

balusu — Thu, 05 Jul 2018 15:39:18 GMT

@bgooley

I completely agree with the Uppercase realm concept and it worked fine for many clusters we deployed for different clients but somehow the current cluster only works when I have the lower case in trusted realms.

Thanks,
Siva

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

bgooley — Thu, 05 Jul 2018 17:24:34 GMT

No problem. I as long as you have a reasonable solution to address the issue, that's all good. 🙂

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

balusu — Fri, 06 Jul 2018 18:59:58 GMT

@bgooley hdfs is not picking up the users from supergroup@domain.com, does auth to local rule works for groups?

hadoop.security.group.mapping org.apache.hadoop.security.ShellBasedUnixGroupsMapping

[sbalusu@domain.com@hostname ~]$ hadoop fs -chown hdfs:supergroup /user/test
chown: changing ownership of '/user/test': Non-super user cannot change owner

[sbalusu@domain.com@hostname ~]$ getent group supergroup@domain.com
supergroup@domain.com:*:514734591:sbalusu@supergroup.com

I tried both group short name as well as group fqdn:
dfs.permissions.supergroup, dfs.permissions.superusergroup supergroup@domain.com

dfs.permissions.supergroup, dfs.permissions.superusergroup supergroup

any suggestions?

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

bgooley — Fri, 06 Jul 2018 19:07:10 GMT

@balusu,

auth_to_local is used to map a user's principal to a unix name only. It is not used for anything group-oriented.

By default, only the "hdfs" user is a superuser so it is the only user who can perform "chown" operations.

If you want to make other users superusers, you can do so by defining which group will be the "supergroup" and which users belong to it.

The group must be accessible via the OS (getent group supergroup). The default name for the supergroup is "supergroup"

In cloudera Manager you can see this configuration in HDFS --> Configuration --> Superuser Group

is there a reason you are trying to attach the "@domain" onto the group name?

I would recommend adding a group named "supergroup" if you don't need to change the default. Then add sbalusu as a member.

Note this has nothing to do with Kerberos at all at this point... this is all group mapping for hadoop.

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

balusu — Fri, 06 Jul 2018 19:34:44 GMT

@bgooley

I appolgise for the confusion, The supergroup I mentioned is hadoopadmingroup@example.com

In cloudera Manager i changed this configuration in HDFS --> Configuration --> Superuser Group

and tried setting it to

hadoopadmingroup@example.com and then hadoopadmingroup, both of them did not worked.

sssd is set up to have a domain name at the end of Unix group and Unix user, Somehow hdfs is not able to map user to group with the domain name at the end.

True, I agree this is not a Kerberos issue. My intention is to find if Hadoop can work having a domain name at the end of the group so that I can have a conversation with Unix team to trim domain name at the end of the group.

Thanks,

Siva

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

bgooley — Fri, 06 Jul 2018 19:38:21 GMT

@balusu,

Yeah, I'm not sure if supergroup mapping will work if the group has the domain on it. I can't confirm it won't, but if you changed the group name, restarted HFDS, and still didn't have group access, that does indicate the config may not work.

You may try running "hdfs groups <user>" to see if that command "sees" your groups....

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

balusu — Fri, 06 Jul 2018 19:52:19 GMT

@bgooley

Ya, it does not seem to be working.

HDFS --> Configuration --> Superuser Group = hadoopadmingroup@example.com and then hadoopadmingroup, both of them yielded zero groups.

[sbalusu@example.com@hostname ~]$ hdfs groups sbalusu@example.com
sbalusu_c@example.com :
[sbalusu@example.com@hostname ~]$ hdfs groups sbalusu_c
sbalusu_c :

Thanks & Regards,
Siva

Re: Kerberos ticket error:No rules applied to hdfs@CDH5.14.2

balusu — Mon, 09 Jul 2018 20:18:26 GMT

I have the SSSD configured to short name and everything looks good now!!! Thanks @bgooley