Member since
09-15-2015
457
Posts
507
Kudos Received
90
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
16847 | 11-01-2016 08:16 AM | |
12462 | 11-01-2016 07:45 AM | |
11382 | 10-25-2016 09:50 AM | |
2437 | 10-21-2016 03:50 AM | |
5097 | 10-14-2016 03:12 PM |
01-12-2016
09:06 AM
3 Kudos
Renewing Kerberos tickets does not work in the Solr Standalone (single node) solution at the moment. There is a bug in Solr that prevents the renewal of Kerberos tickets (I have an open ticket regarding this issue). However the SolrCloud solution should renew its tickets. Below is a sample configuration that will enable Kerberos for SolrCloud and ensure that Solr Kerberos tickets are renewed. (Note: this will also enable Kerberos authentication for the SolrAdmin UI). /opt/lucidworks-hdpsearch/solr/bin/jaas.conf Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/solr.headless.keytab"
storeKey=true
debug=true
principal="solr-mycluster@EXAMPLE.COM";
};
/opt/lucidworks-hdpsearch/solr/bin/solr.in.sh SOLR_HOST=`hostname -f`
ZK_HOST="zookeeperA.example.com:2181,zookeeperB.example.com:2181,zookeeperC.example.com:2181/solr"
SOLR_KERB_PRINCIPAL=HTTP/${SOLR_HOST}@EXAMPLE.COM
SOLR_KERB_KEYTAB=/etc/security/keytabs/solr-spnego.service.keytab
SOLR_JAAS_FILE=/opt/lucidworks-hdpsearch/solr/bin/jaas.conf
SOLR_AUTHENTICATION_CLIENT_CONFIGURER=org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer
SOLR_AUTHENTICATION_OPTS=" -DauthenticationPlugin=org.apache.solr.security.KerberosPlugin -Djava.security.auth.login.config=${SOLR_JAAS_FILE} -Dsolr.kerberos.principal=${SOLR_KERB_PRINCIPAL} -Dsolr.kerberos.keytab=${SOLR_KERB_KEYTAB} -Dsolr.kerberos.cookie.domain=${SOLR_HOST} -Dhost=${SOLR_HOST} -Dsolr.kerberos.name.rules=RULE:[1:$1@$0](solr-mycluster@EXAMPLE.COM)s/.*/solr/DEFAULT"
/etc/security/keytabs/solr.headless.keytab Keytab file for the solr-mycluster@EXAMPLE.COM principal Add the following file and znode to Zookeeper: /opt/lucidworks-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost zookeeperA.example.com:2181,zookeeperB.example.com:2181,zookeeperC.example.com:2181 -cmd makepath /solr
/opt/lucidworks-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost zookeeperA.example.com:2181,zookeeperB.example.com:2181,zookeeperC.example.com:2181 -cmd put /solr/security.json '{"authentication":{"class": "org.apache.solr.security.KerberosPlugin"}}'
Solrconfig.xml Your solrconfig.xml looks fine. <directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory">
<str name="solr.hdfs.home">hdfs://mycluster/solr</str>
<str name="solr.hdfs.confdir">/etc/hadoop/conf</str>
<bool name="solr.hdfs.security.kerberos.enabled">true</bool>
<str name="solr.hdfs.security.kerberos.keytabfile">/etc/security/keytabs/solr.headless.keytab</str>
<str name="solr.hdfs.security.kerberos.principal">solr-mycluster@EXAMPLE.COM</str>
<bool name="solr.hdfs.blockcache.enabled">true</bool>
<int name="solr.hdfs.blockcache.slab.count">1</int>
<bool name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
<int name="solr.hdfs.blockcache.blocksperbank">16384</int>
<bool name="solr.hdfs.blockcache.read.enabled">true</bool>
<bool name="solr.hdfs.blockcache.write.enabled">true</bool>
<bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
<int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">64</int>
<int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">512</int>
</directoryFactory>
You can find some more information on this page => https://cwiki.apache.org/confluence/display/RANGER/How+to+configure+Solr+Cloud+with+Kerberos+for+Ranger+0.5
... View more
01-12-2016
08:14 AM
1 Kudo
You might have to change the current java version by using /usr/sbin/alternatives --config java This will change the java version for the whole system and not just Ambari and the HDP components.
... View more
12-31-2015
08:22 AM
did you change any config? how did you add the user pt0034? Could you start the query again and monitor the hiveserver2.log file (/var/log/hive/...) and check whether some errors show up.
... View more
12-28-2015
02:51 PM
whats the name of the user that is getting this error?
... View more
12-28-2015
05:58 AM
1 Kudo
Ubuntu 14 is suppurted and the repositories should be available. Did you use the following repositories: public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/ubuntu14 public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.3.4.0/
... View more
12-23-2015
08:03 AM
1 Kudo
Thanks for the addt. info. When you start your Job through your bash script, do you have a valid Kerberos ticket on the machine or does your MR Job use a keytab file to retrieve a valid Kerberos ticket? Without a valid ticket, Solr will always deny access. You might want to enable Kerberos security for Zookeeper as well, see this https://cwiki.apache.org/confluence/display/solr/K... "When setting up a kerberized Solr cluster, it is recommended to enable Kerberos security for Zookeeper as well. In such a setup, the client principal used to authenticate requests with Zookeeper can be shared for internode communication as well." Also see this article https://cwiki.apache.org/confluence/display/RANGER... Alternatively, you could try the Hadoop Job Jar to ingest your data (I have successfully used it both in kerberized and non-kerberized Solr enivornments): hadoop jar /opt/lucidworks-hdpsearch/job/lucidworks-hadoop-job-2.0.3.jar com.lucidworks.hadoop.ingest.IngestJob -Dlww.commit.on.close=true -Dlww.jaas.file=jaas.conf -cls com.lucidworks.hadoop.ingest.DirectoryIngestMapper --collection test -i file:///data/* -of com.lucidworks.hadoop.io.LWMapRedOutputFormat --zkConnect horton01.example.com:2181,horton02.example.com:2181,horton03.example.com:2181/solr Could you share more details about your bash script and MR job?
... View more
12-23-2015
07:40 AM
2 Kudos
@Davide Isoardi were you able to test the above solution?
... View more
12-22-2015
05:15 PM
1 Kudo
Thanks 🙂
... View more
12-22-2015
12:55 PM
7 Kudos
. @Suresh Bonam yes you can calculate the nth element, see the following example:
Create sample table (emp=employees)
create table emp(id Int, salary Double)
row format delimited
fields terminated by ','
stored as textfile
I have added the following data to that table:
1,423
2,23
3,67878
4,84
5,9999999
6,1
7,578
8,8877
9,4445
10,756
11,84
Understanding the data
Lets look at the sorted salaries first:
select * from emp order by salary;
Result:
emp.id emp.salary
6 1
2 23
4 84
11 84
1 423
7 578
10 756
9 4445
8 8877
3 67878
5 9999999
Note: We have two identical salaries in the table (=84)
Create query to select unique salaries
In order to select the nth-element we need to remove the duplicate values (e.g. distinct) and create some kind of row counter or index. I decided to use Group-By to remove the duplicate salaries and row_number-UDF to generate the index.
select salary, ROW_NUMBER() over (ORDER BY salary) as row_no from emp group by salary
Result:
salary row_no
1 1
23 2
84 3
423 4
578 5
756 6
4445 7
8877 8
67878 9
9999999 10
Putting everything together
select * from (
select salary, ROW_NUMBER() over (ORDER BY salary) as row_no from emp group by salary) res
where res.row_no = 4
Result:
res.salary res.row_no
423 4
Note: We received the correct result and not 84 since we've removed the duplicate salaries in the subquery 🙂
... View more
12-22-2015
11:07 AM
2 Kudos
Yes, the name of the user that started the job or workflow will be available through this variable. See this code partial: https://github.com/apache/oozie/blob/master/core/s... /**
* Return the job owner user name.
*
* @return the job owner user name.
*/
public static String wf_user() {
return getWorkflow().getUser();
}
... View more