Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Write to Kerberos HDFS from IDE on local machine

Write to Kerberos HDFS from IDE on local machine

New Contributor

I am using IntelliJ to write some files to HDFS in a HDP 2.4 kerborized enviroment.

Getting this error: Exception in thread "main" java.io.IOException: Can't get Master Kerberos principal for use as renewer

This is my relevant code:

  val config=new Configuration()
config.set("hadoop.security.authentication", "kerberos");
UserGroupInformation.setConfiguration(config);
UserGroupInformation.loginUserFromKeytab("testuser@HDP.DEV","C:\\GobyCode\\IDEargs\\testuser.keytab")
...........
activityByProduct.write.mode(SaveMode.Append).parquet("hdfs://sandbox.hortonworks.com:8020/user/maria_dev")
This is my IDE runtime VM parameters
-Djava.security.auth.login.config=C:/GobyCode/IDEargs/testuser_jaas.conf -Djava.security.krb5.conf=C:/GobyCode/IDEargs/krb5.conf

Any ideas?
4 REPLIES 4
Highlighted

Re: Write to Kerberos HDFS from IDE on local machine

Explorer

Did you set "yarn.resourcemanager.principal" property?

If no please do since RM is the default token revewer so Yarn client will try to find out the renewer based on this property.

Highlighted

Re: Write to Kerberos HDFS from IDE on local machine

New Contributor

No I have not.

Where/how do I set the that property?

Thanks

Highlighted

Re: Write to Kerberos HDFS from IDE on local machine

Explorer

I beleive:

config.set("yarn.resourcemanager.principal","rm/_HOST@HDP.DEV");
Highlighted

Re: Write to Kerberos HDFS from IDE on local machine

New Contributor

System.setProperty(HDFS_USER, "HDFSUSER");

System.setProperty("java.security.krb5.conf", "C:\\Kerberos\\krb5.conf");

System.setProperty("sun.security.krb5.debug", "true");

conf = new Configuration();

conf.set(FileSystem.FS_DEFAULT_NAME_KEY, db.getHdfsPrefix());

conf.set("hadoop.security.authentication", "kerberos");

UserGroupInformation.loginUserFromKeytab("username@DOMAIN>COM", "c:\\kerberos.keytab");

FileSystem fs = FileSystem.get(conf);

FileStatus[] fsStatus = fs.listStatus(new Path("/"));

for (int i = 0; i < fsStatus.length; i++)

{

System.out.println(fsStatus[i].getPath().toString());

}

Don't have an account?
Coming from Hortonworks? Activate your account here