Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Solr Indexing using Java with Kerberos enabled

Solr Indexing using Java with Kerberos enabled

Explorer

Hello all !

 

I am trying to read parquette file from hdfs and index into solr using Java.

I am following the code here: https://github.com/cloudera/cdk/blob/master/cdk-morphlines/cdk-morphlines-core/src/test/java/com/clo...

 

Without Kerberos it works fine and I am able to index. However, when I am trying to implement while kerberos is enabled, I am getting this below error. It seems I need to somehow add some configuration for Morphline to use Kerberos auth. Can someone help please ?

 

Exception in thread "main" org.kitesdk.morphline.api.MorphlineRuntimeException: org.kitesdk.morphline.api.MorphlineRuntimeException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
	at org.kitesdk.morphline.api.MorphlineContext$DefaultExceptionHandler.handleException(MorphlineContext.java:289)
	at com.solrjindex.MorphlineParquetIndexer.main(MorphlineParquetIndexer.java:48)
Caused by: org.kitesdk.morphline.api.MorphlineRuntimeException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
	at org.kitesdk.morphline.hadoop.parquet.avro.ReadAvroParquetFileBuilder$ReadAvroParquetFile.doProcess(ReadAvroParquetFileBuilder.java:185)
	at org.kitesdk.morphline.base.AbstractCommand.process(AbstractCommand.java:161)
	at org.kitesdk.morphline.base.AbstractCommand.doProcess(AbstractCommand.java:186)
	at org.kitesdk.morphline.base.AbstractCommand.process(AbstractCommand.java:161)
	at com.solrjindex.MorphlineParquetIndexer.main(MorphlineParquetIndexer.java:39)
Caused by: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
	at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2103)
	at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2084)
	at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:731)
	at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:110)
	at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:796)
	at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:792)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:792)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1565)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1608)
	at parquet.hadoop.ParquetReader.<init>(ParquetReader.java:114)
	at parquet.hadoop.ParquetReader.<init>(ParquetReader.java:77)
	at parquet.avro.AvroParquetReader.<init>(AvroParquetReader.java:62)
	at org.kitesdk.morphline.hadoop.parquet.avro.ReadAvroParquetFileBuilder$ReadAvroParquetFile.doProcess(ReadAvroParquetFileBuilder.java:168)
	... 4 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
	at org.apache.hadoop.ipc.Client.call(Client.java:1472)
	at org.apache.hadoop.ipc.Client.call(Client.java:1409)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
	at com.sun.proxy.$Proxy19.getListing(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:573)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy20.getListing(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2101)
	... 17 more

My run command:

 

java -Djava.security.auth.login.config="/home/cloudera/workspace/jaas.conf" -jar /home/cloudera/workspace/Java_workspace/SolrJParquetIndex/out/artifacts/SolrJParquetIndex_jar/SolrJParquetIndex.jar /home/cloudera/workspace/solr_home/morphlines/solrj_morphline.conf hdfs://quickstart.cloudera:8020/user/hive/warehouse/solrtest/000000_0
1 REPLY 1
Highlighted

Re: Solr Indexing using Java with Kerberos enabled

Cloudera Employee

Hi,

 

Can you please show us the contents of your jaas.conf file? I assume you are using a cached credentials but it would be good to confirm.  If you're leveraging cached credentials you need to "kinit" as the user ahead of time.  I'm sure you probably doing all of this but we should make sure.

 

It may help to add additional Kerberos logging via the following JVM property:

 

  -Dsun.security.krb5.debug=true

 

Nick

Don't have an account?
Coming from Hortonworks? Activate your account here