Reply
Explorer
Posts: 6
Registered: ‎05-24-2017

Solr Indexing using Java with Kerberos enabled

Hello all !

 

I am trying to read parquette file from hdfs and index into solr using Java.

I am following the code here: https://github.com/cloudera/cdk/blob/master/cdk-morphlines/cdk-morphlines-core/src/test/java/com/clo...

 

Without Kerberos it works fine and I am able to index. However, when I am trying to implement while kerberos is enabled, I am getting this below error. It seems I need to somehow add some configuration for Morphline to use Kerberos auth. Can someone help please ?

 

Exception in thread "main" org.kitesdk.morphline.api.MorphlineRuntimeException: org.kitesdk.morphline.api.MorphlineRuntimeException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
	at org.kitesdk.morphline.api.MorphlineContext$DefaultExceptionHandler.handleException(MorphlineContext.java:289)
	at com.solrjindex.MorphlineParquetIndexer.main(MorphlineParquetIndexer.java:48)
Caused by: org.kitesdk.morphline.api.MorphlineRuntimeException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
	at org.kitesdk.morphline.hadoop.parquet.avro.ReadAvroParquetFileBuilder$ReadAvroParquetFile.doProcess(ReadAvroParquetFileBuilder.java:185)
	at org.kitesdk.morphline.base.AbstractCommand.process(AbstractCommand.java:161)
	at org.kitesdk.morphline.base.AbstractCommand.doProcess(AbstractCommand.java:186)
	at org.kitesdk.morphline.base.AbstractCommand.process(AbstractCommand.java:161)
	at com.solrjindex.MorphlineParquetIndexer.main(MorphlineParquetIndexer.java:39)
Caused by: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
	at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2103)
	at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2084)
	at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:731)
	at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:110)
	at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:796)
	at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:792)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:792)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1565)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1608)
	at parquet.hadoop.ParquetReader.<init>(ParquetReader.java:114)
	at parquet.hadoop.ParquetReader.<init>(ParquetReader.java:77)
	at parquet.avro.AvroParquetReader.<init>(AvroParquetReader.java:62)
	at org.kitesdk.morphline.hadoop.parquet.avro.ReadAvroParquetFileBuilder$ReadAvroParquetFile.doProcess(ReadAvroParquetFileBuilder.java:168)
	... 4 more
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
	at org.apache.hadoop.ipc.Client.call(Client.java:1472)
	at org.apache.hadoop.ipc.Client.call(Client.java:1409)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
	at com.sun.proxy.$Proxy19.getListing(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:573)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy20.getListing(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2101)
	... 17 more

My run command:

 

java -Djava.security.auth.login.config="/home/cloudera/workspace/jaas.conf" -jar /home/cloudera/workspace/Java_workspace/SolrJParquetIndex/out/artifacts/SolrJParquetIndex_jar/SolrJParquetIndex.jar /home/cloudera/workspace/solr_home/morphlines/solrj_morphline.conf hdfs://quickstart.cloudera:8020/user/hive/warehouse/solrtest/000000_0
Highlighted
Cloudera Employee
Posts: 7
Registered: ‎06-10-2015

Re: Solr Indexing using Java with Kerberos enabled

Hi,

 

Can you please show us the contents of your jaas.conf file? I assume you are using a cached credentials but it would be good to confirm.  If you're leveraging cached credentials you need to "kinit" as the user ahead of time.  I'm sure you probably doing all of this but we should make sure.

 

It may help to add additional Kerberos logging via the following JVM property:

 

  -Dsun.security.krb5.debug=true

 

Nick

Announcements