Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Anyone with experience on HBase-Indexer for solr (Lucidworks / NGData)

Highlighted

Anyone with experience on HBase-Indexer for solr (Lucidworks / NGData)

Expert Contributor

I wanted to reach out and see if anyone has experience with the Lucidworks / NGData Hbase-Indexer for solr. If so, I would sure like to talk to you for half an hour!

Out at client site and trying to get this working, but the docs are a little slim on implementing on a secure cluster. Have worked through a number of issues, but down to an authentication problem.

When starting the hbase-indexer server, it always picks up the user starting the server rather than the principals specified in the configs. We have tried this with multiple users and kinit’ing as various principals (self, solr, hbase).

If anyone has any similar experiences or additional references, I haven’t found, it would be very much appreciated.

Thanks in advance,

Jim Barnett

Environment:

HDP 2.2.8

Solr 4.10.2

Hbase-indexer 1.6

Error Output

/opt/hbase-indexer/bin/hbase-indexer
serverorg.apache.hadoop.hbase.security.AccessDeniedException: Kerberos
principal name does NOT have the expected hostname part: root  at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.saslReadAndProcess(RpcServer.java:1301)  at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1513)  at
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:802)  at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:593)  at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:568)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  at
java.lang.Thread.run(Thread.java:745)16/05/25 00:59:06 DEBUG
ipc.RpcServer: RpcServer.listener,port=44043: DISCONNECTING client
10.26.73.227:36705 because read count=-1. Number of active connections: 1

hbase-indexer-site.xml

<configuration><property><name>hbaseindexer.zookeeper.connectstring</name> 
<value>hornn-h01msdc01.ds.dtveng.net:2181,hornn-h01msdc02.ds.dtveng.net:2181,hordn-h01msdc01.ds.dtveng.net:2181</value></property><property><name>hbase.zookeeper.quorum</name> 
<value>hornn-h01msdc01.ds.dtveng.net,hornn-h01msdc02.ds.dtveng.net,hordn-h01msdc01.ds.dtveng.net</value></property><property> 
<name>hbaseindexer.authentication.type</name>  <value>kerberos</value>  </property>  <property> 
<name>hbaseindexer.authentication.kerberos.keytab</name> 
<value>/etc/security/keytabs/hbase.headless.keytab</value>  </property>  <property> 
<name>hbaseindexer.authentication.kerberos.principal</name> 
<value>HTTP/hordn-h01msdc05.ds.dtveng.net@DS.DTVENG.NET</value>  </property>  <property> 
<name>hbaseindexer.authentication.kerberos.name.rules</name> 
<value>DEFAULT</value>  </property>  </configuration>

jaas.conf (for hbase-indexer)

jaas.confClient {  com.sun.security.auth.module.Krb5LoginModule required  useKeyTab=true  keyTab="/etc/security/keytabs/hbase.service.keytab"  storeKey=true  useTicketCache=true  debug=true  principal="hbase/hordn-h01msdc05.ds.dtveng.net@DS.DTVENG.NET";};

hbase-site.xml

    <-- Manually added for hbase-indexer -->
    <property>
      <name>hbase.replication</name>
      <value>true</value>
    </property>
    <property>
      <name>replication.source.ratio</name>
      <value>1.0</value>
    </property>
    <property>
     
<name>replication.source.nb.capacity</name>
      <value>1000</value>
    </property>
    <property>
     
<name>replication.replicationsource.implementation</name>
     
<value>com.ngdata.sep.impl.SepReplicationSource</value>
    </property>
    <!-- Auto Added for Replication -->
    <property>
      <name>zookeeper.znode.parent</name>
      <value>/hbase-secure</value>
    </property>
    <property>
      <name>zookeeper.znode.replication</name>
      <value>replication</value>
    </property>
    <property>
     
<name>zookeeper.znode.replication.peers</name>
      <value>peers</value>
  </property>

hbase-indexer (shell script)

# Additions only

HBASE_INDEXER_OPTS="$HBASE_INDEXER_OPTS -Dlww.jaas.file=/opt/hbase-indexer/bin/jaas.conf"

HBASE_INDEXER_OPTS="$HBASE_INDEXER_OPTS -Dlww.jaas.appname=Client"

References:

Setting up search (very useful, but no Kerberos)

https://community.hortonworks.com/articles/1181/hbase-indexing-to-solr-with-hdp-search-in-hdp-23.htm...

Installing and configuring Indexer

https://github.com/NGDATA/hbase-indexer/wiki/Installation

Installing and configuring Indexer with Kerberos

https://doc.lucidworks.com/lucidworks-hdpsearch/2.3/Guide-Jobs.html

11 REPLIES 11
Highlighted

Re: Anyone with experience on HBase-Indexer for solr (Lucidworks / NGData)

It appears that your JAAS config file is incorrect. It should probably look something like:

jaas.confClient {
  com.sun.security.auth.module.Krb5LoginModule required
  doNotPrompt=true
  useKeyTab=true
  keyTab="/etc/security/keytabs/hbase.service.keytab"
  principal="hbase/hordn-h01msdc05.ds.dtveng.net@DS.DTVENG.NET"
  storeKey=true
  useTicketCache=false
  debug=true;
};

The real difference here is the "useTicketCache" value. By setting it to true, you declare that the executing user's ticket cache is to be used when obtaining Kerberos tickets. This is why the identity of the user that is starting the service is being used. By setting it to false, a new ticket cache will be created using the princiapl and keytab details specified in the config.

See https://docs.oracle.com/javase/7/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5Lo....

Highlighted

Re: Anyone with experience on HBase-Indexer for solr (Lucidworks / NGData)

Expert Contributor

Have changed the jaas file to the following, but still no joy!

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  doNotPrompt=true
  useKeyTab=true
  keyTab="/etc/security/keytabs/hbase.service.keytab"
  storeKey=true
  useTicketCache=false
  debug=true
  principal="hbase/hordn-h01msdc05.ds.dtveng.net@DS.DTVENG.NET";
};

I am wondering if we are using the correct java parameters to point to the file. The hbase-indexer docs (https://doc.lucidworks.com/lucidworks-hdpsearch/2.3/Guide-Jobs.html#solr) use the lww.jaas.fil; whereas, the java docs (http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html) show the java.security.auth.login.config setting. We have added both.

-XX:+UseConcMarkSweepGC  -Dlww.jaas.file=/opt/hbase-indexer-1.6-SNAPSHOT/bin/jaas.conf -Dlww.jaas.appname=Client -Dhbaseindexer.log.dir=/opt/hbase-indexer-1.6-SNAPSHOT/bin/../logs -Dhbaseindexer.log.file=hbase-indexer.log -Dhbaseindexer.home.dir=/opt/hbase-indexer-1.6-SNAPSHOT/bin/.. -Dhbaseindexer.id.str= -Dhbaseindexer.root.logger=DEBUG,console -Djava.security.auth.login.config=/opt/hbase-indexer-1.6-SNAPSHOT/bin/jaas.conf

Note: Also tried to put the app name "Client" after the java.security.auth.login.config, but that just blew up with java thinking it was a class name!

Highlighted

Re: Anyone with experience on HBase-Indexer for solr (Lucidworks / NGData)

@jbarnett

Does the user executing the command have read access to /etc/security/keytabs/hbase.service.keytab?

Does /etc/security/keytabs/hbase.service.keytab contain keytab entries for hbase/hordn-h01msdc05.ds.dtveng.net@DS.DTVENG.NET?

klist -kte /etc/security/keytabs/hbase.service.keytab
Highlighted

Re: Anyone with experience on HBase-Indexer for solr (Lucidworks / NGData)

Expert Contributor

The permissions were not available to root. Changed to 755, but no change. Grrrrr!

Also, corrected the native lib not found, because I thought that was falling back and using local, but that did not help either.

Re: Anyone with experience on HBase-Indexer for solr (Lucidworks / NGData)

The hbase-indexer-site.xml looks wrong

hbaseindexer.authentication.kerberos.keytab=/etc/security/keytabs/hbase.headless.keytab

hbaseindexer.authentication.kerberos.principal=HTTP/hordnh01msdc05.ds.dtveng.net@DS.DTVENG.NET

I will expect:

hbaseindexer.authentication.kerberos.principal=hbase@DS.DTVENG.NET

Should it not contain

zookeeper.znode.parent=/hbase-secure

Highlighted

Re: Anyone with experience on HBase-Indexer for solr (Lucidworks / NGData)

Expert Contributor

We had changed that so that the keytab and the principal match yesterday (both should be headless for principal short name HTTP). However, this did not solve the problem.

Highlighted

Re: Anyone with experience on HBase-Indexer for solr (Lucidworks / NGData)

Dear @jbarnett,

Sorry to revive this post, but were you able to find a solution? I'm having the exactly same challenge.

Many thanks in advance.

Best regards,

-Pedro Drummond.

Highlighted

Re: Anyone with experience on HBase-Indexer for solr (Lucidworks / NGData)

Super Collaborator

What version of Solr are you using?

Highlighted

Re: Anyone with experience on HBase-Indexer for solr (Lucidworks / NGData)

Expert Contributor

We never did get this to work with HDP 2.2.8, HBase .98, and solr 4.10. We engaged Lucidworks, and there answers are below. If you are on HDP 2.3.x, HBase 1.0, and solr 5.2.x this should be supported, and we can get updates from Lucidworks if needed. Customer decided to put on hold since they had a workaround and wait until they upgrade to 2.3.4+ We also did a POC with NiFi showing parallel ingest to HBase and Solr. If anyone is interested, I can get you info.

Reference internal case# 00082768

Answers to your questions from LucidWorks:

1. Were there other additional fixes made to the code around this ? (I.e. Just this probably won???t fix) We're not entirely sure of your question here, but what you're describing sounds like configuration errors; we made a lot of changes to support newer versions (kerberized support for Solr).

2. Is the 2.2.2 version of hbase-indexer tested with Hbase 0.98? (I was able to compile with it, but don???t know if it???s tested)No, it was not tested with 0.98. It was only tested with 1.1.2.

3. Comment: The hbase-indexer 2.2.2 version won???t even compile with solr 4.10.2 so that???s a no go.Yes, we made a few changes to support Solr 5.2.1 (kerberized support for Solr) and later more changes to support 5.5.1

It appears that you're trying to use hbase-indexer with HDP 2.2. It was not developed, nor tested, for HDP 2.2. Furthermore, it was only released for HDP 2.3.x.

The package of Solr Lucid made for HDP 2.2 included Solr 4.10.x; the release for HDP 2.3 included Solr 5.2.1. As we note above, changes were made to the hbase-indexer to support Solr 5.x, and these changes are not backward-compatible with pre-Solr 5.x releases.

@PedroDrummond

@ThiagoSantiago

Don't have an account?
Coming from Hortonworks? Activate your account here