java.lang.IllegalArgumentException: Illegal principal name

[#1]

Hi , I am running into a problem .

i have  CDH cluster(kerberized cluster )  on unix and ETL client on windows .

so to authenticate i have installed kerberos client on on window and generate the ticket for my user .

but while running a job i am getting 

Starting job testjob at 18:40 16/02/2017.

[statistics] connecting to socket on port 3487
[statistics] connected
[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in component tHDFSOutput_1
java.io.IOException: failure to login
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:823)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2859)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2851)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2714)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:382)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181)
at local_project.testjob_0_1.testjob.tFixedFlowInput_1Process(testjob.java:709)
at local_project.testjob_0_1.testjob.tHDFSConnection_1Process(testjob.java:419)
at local_project.testjob_0_1.testjob.runJobInTOS(testjob.java:1156)
at local_project.testjob_0_1.testjob.main(testjob.java:990)
Caused by: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name 25761081@PROD.GLOBAL.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to 25761081@PROD.GLOBAL.COM
[FATAL]: local_project.testjob_0_1.testjob - tHDFSOutput_1 failure to login
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at javax.security.auth.login.LoginContext.invoke(Unknown Source)
at javax.security.auth.login.LoginContext.access$000(Unknown Source)
at javax.security.auth.login.LoginContext$4.run(Unknown Source)
at javax.security.auth.login.LoginContext$4.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(Unknown Source)
at javax.security.auth.login.LoginContext.login(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:798)
... 11 more
Caused by: java.lang.IllegalArgumentException: Illegal principal name 25761081@PROD.GLOBAL.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to 25761081@PROD.GLOBAL.COM
at org.apache.hadoop.security.User.<init>(User.java:50)
at org.apache.hadoop.security.User.<init>(User.java:43)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:197)
... 23 more
Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to 25761081@PROD.GLOBAL.COM
at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389)
at org.apache.hadoop.security.User.<init>(User.java:48)
... 25 more
[statistics] disconnected
Job testjob ended at 18:40 16/02/2017. [exit code=1]

************************

this is windows klist 

C:\Program Files\MIT\Kerberos\bin>klist
Ticket cache: FILE:C:\temp\krb5cache
Default principal: 25761081@PROD.GLOBAL.COM

Valid starting Expires Service principal
02/16/17 18:53:18 02/17/17 04:53:18 krbtgt/PROD.GLOBAL.COM@PROD.GLOBAL.COM
renew until 02/23/17 18:53:13

**********************

this is the krb5.ini

[realms]
CDH.PROD.GLOBAL.COM = {
kdc = XXXXXXXXXXXXXX
admin_server = XXXXXXXXXXXX
}
PROD.GLOBAL.COM = {
kdc = XXXXXXXXXXXXXXX:88
admin_server = XXXXXXXXXXXX:749
default_domain = PROD.GLOBAL.COM
}

Please suggest

[#2]

I believe you are missing the realm rule in the core-site.xml 

please check the tag in the core-site.xml

hadoop.registry.kerberos.realm 

Took this reference from hadoop.apaches core-site.xml api 

hadoop.registry.kerberos.realm		

The kerberos realm: used to set the realm of system principals which do not declare their realm, and any other accounts that need the value. 
If empty, the default realm of the running process is used. 
If neither are known and the realm is needed, then the registry service/client will 
fail

[#3]

No rules applied to 25761081@PROD.GLOBAL.COM

You do not have any auth-to-local rules set up for this format.

You should also have the realm in the Trusted Realm list as mentioned by @csguna.

https://www.cloudera.com/documentation/enterprise/5-4-x/topics/sg_auth_to_local_isolate.html

[#4]

<name>hadoop.security.auth_to_local</name>
<value>RULE:[1:$1@$0](.*@\QCDH.PROD.GLOBAL.COM\E$)s/@\QCDH.PROD.GLOBAL.COM\E$//
RULE:[2:$1@$0](.*@\QCDH.PROD.GLOBAL.COM\E$)s/@\QCDH.PROD.GLOBAL.COM\E$//
RULE:[1:$1@$0](.*@\QPROD.GLOBAL.COM\E$)s/@\QPROD.GLOBAL.COM\E$//
RULE:[2:$1@$0](.*@\QPROD.GLOBAL.COM\E$)s/@\QPROD.GLOBAL.COM\E$//
DEFAULT</value>

trusted realms
CDH.PROD.GLOBAL.COM
PROD.GLOBAL.COM

Also cluster is in PROD.GLOBAL.COM domain while ETL clint is in EXAMPLE.COM domain.

[#5]

See the below link. It has a good chart to help understand the rules. The ones you have in place cover a Kerberos principal with 3 parts. These are the host and service principals, hdfs/hostname.realm.com@realm.com, for both realms.

You need something more along the lines of [1:$1]. This will strip of the domain name. The remain username will be left and what is used to access resources across the cluster.

https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_kerbprin_to_sn.html

[#6]

Miss the last line on your last message. If the client, and the user, are in a third domain then you need to add that rule and add the new realm to the trusted realms list.

[#7]

Hello everybody
I have the same problem is what you have solved your problem
Any sugestion plz 

can you share your solution plz @MSharma