Member since
08-05-2016
12
Posts
9
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
831 | 12-11-2017 03:17 AM |
02-22-2018
08:02 PM
I’m finishing this component in nexts 3 weeks... is a processor fto NiFi.
... View more
12-11-2017
03:17 AM
can you see this link: Partioning recomendations
... View more
12-11-2017
01:39 AM
1 Kudo
Secondary Namenode is one of the poorly named component in Hadoop. By its name, it gives a sense that its a backup for the Namenode.But in reality its not. Lot of beginners in Hadoop get confused about what exactly SecondaryNamenode does and why its present in HDFS. This article explains exactly how SecondaryNamenode works http://blog.madhukaraphatak.com/secondary-namenode---what-it-really-do/ In a very simple way as if Standby were Streaming Replication with integrated failover; While SecondaryNamenode was Timed LogShipping
... View more
08-16-2016
10:14 PM
2 Kudos
Authors -
Gerdan dos Santos - https://community.hortonworks.com/users/12342/gerdan.html Pedro Drummond - https://community.hortonworks.com/users/10987/pedrodrummond.html
After deleting users using REST API, one might get error "404 not found". This articles demonstrates how to automatically delete the "ghost" users from MySQL and Posgres. Reproducing the Error Creating testUser 1. Create testUser at Ranger UI - http://{ranger_host}:6080
2. Username: testUser; Group: public
.
3. Check if the default Resource Based Policies is assigned to testUser
.
4. If testUser is not assigned to the policy, add it and click the save button
. Deleting testUser using REST API
Reference:
https://community.hortonworks.com/questions/1021/how-to-remove-all-external-users-from-the-ranger-r.html
Through the terminal, delete the testUser using REST API.
For instance:
curl -X DELETE -v -u admin:password http://{RangerHost}:6080/service/xusers/users/userName/testUser
You should get something like this:
.
Verifying the Error
1. Edit the Resource Based Policies
.
2. Just click the save button
.
3. There you go! Error 404 Not Found
This happens because once you delete the testUser using REST API, you are only deleting the user from the UI and not from the permissioning that it is assigned to. Solving this issue
All you have to do is connecting in the Ranger's Metadata DB and run the following scripts thought the database client logged with a db user that has full permissions to the database:
For MySQL:
-------------------------
----------MYSQL----------
-------------------------
CREATE TEMPORARY TABLE IF NOT EXISTS ranger.tmp_users_clean as
select id from ranger.x_portal_user
where
login_id not in (
select user_name from ranger.x_user
);
delete from ranger.x_auth_sess where user_id in(
select id from ranger.tmp_users_clean
);
delete from ranger.x_portal_user_role where user_id in(
select id from ranger.tmp_users_clean
);
delete from ranger.x_user_module_perm where user_id in(
select id from ranger.tmp_users_clean
);
delete from ranger.x_portal_user where id in(
select id from ranger.tmp_users_clean
);
For Postgres:
-------------------------
--------POSTGRESQL-------
-------------------------
delete from x_auth_sess where user_id in(
select id from x_portal_user
where
login_id not in (
select user_name from x_user
)
);
delete from x_portal_user_role where user_id in(
select id from x_portal_user
where
login_id not in (
select user_name from x_user
)
);
delete from x_user_module_perm where user_id in(
select id from x_portal_user
where
login_id not in (
select user_name from x_user
)
);
delete from x_portal_user where id in(
select id from x_portal_user
where
login_id not in (
select user_name from x_user
)
);
Just a little tip: if you don't know where your Ranger's metadata DB is located, you may find it at Ambari UI -> Ranger -> Configs -> Ranger DB host.
For more info: gerdan@gmail.com and pedro.dru@hotmail.com.
Thank you!
... View more
Labels:
08-10-2016
01:41 AM
5 Kudos
Author: Gerdan Rezende dos Santos - https://community.hortonworks.com/users/12342/gerdan.html
Co-author: Pedro Drummond - https://community.hortonworks.com/users/10987/pedrodrummond.html
1 - PREREQUISITES
Before you begin, you need to have:
1.
HDP 2.4 kerberized cluster;
2.
Configured repos for HDPSearch;
3.
Java + JCE installed with defined JAVA_HOME;
4.
Knowledge of you kerberos’ REALM.
2 - KERBEROS CONFIGURATION
2.1 - “Principal” Definition
It’s a Kerberos term which defines an entity by means of an
address, user or service.
This address is commonly divided into 3 parts:
The first part contains the
username or the service name;
The second part contains the
hostname if one is defining a service, or left in blank if one is defining a
user;
The third part contains the REALM.
◦Ex:
user1/solr.community.hortonworks.com@LOCAL.COM
2.2 - Installing Kerberos Clients for HDPSearch Host
For RedHat or Centos:
yum -y install krb5-libs krb5-auth-dialog krb5-workstation
Then copy to the host client the file /etc/krb5.conf from the
kerberos server.
2.3 - Creating the Principal WITH Keytab (Authentication through keytab) as Root
Authenticate at
kadmin console with a user that
has administrative permission, in this example we’re using “
root/admin@LOCAL.COM”.
[root@host ~] kadim –p root/admin@LOCAL.COM
Authenticating as principal root/admin@LOCAL.COM with password.Password for
root/admin@LOCAL.COM:[enter password]
At this point you’ll be at
kadmin console. To
add a user, use the command add_principal or the short version addprinc. Also
specify the option -randkey while creating the user since specifying a password
is useless once you generate the keytab for that user.
Now the HBase and HTTP users for the host where your HDPSearch
is located must be created and then
merged to the same file. In
this example our host is solr.community.hortonworks.com.
kadmin: addprinc -randkey \ hbase/solr.community.hortonworks.com@LOCAL.COM
WARNING: no policy specified for hbase/solr.community.hortonworks.com@LOCAL.COM; defaulting to no policy
Principal "hbase/solr.community.hortonworks.com@LOCAL.COM" created.
kadmin: addprinc -randkey \ HTTP/solr.community.hortonworks.com@LOCAL.COM
WARNING: no policy specified for HTTP/solr.community.hortonworks.com@LOCAL.COM; defaulting to no policy
Principal "HTTP/solr.community.hortonworks.com@LOCAL.COM" created.
#STILL AT THE KADMIN CONSOLE, EXPORT BOTH KEYTABS TO THE SAME FILE.
kadmin: xst -k /etc/security/keytabs/http.hbase.keytab \ HTTP/solr.community.hortonworks.com \ hbase/solr.community.hortonworks.com
Entry for principal HTTP/solr.community.hortonworks.com with kvno 8, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/http.hbase.keytab.
Entry for principal HTTP/solr.community.hortonworks.com with kvno 8, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/http.hbase.keytab.
Entry for principal HTTP/solr.community.hortonworks.com with kvno 8, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/security/keytabs/http.hbase.keytab.
Entry for principal HTTP/solr.community.hortonworks.com with kvno 8, encryption type arcfour-hmac added to keytab WRFILE:/etc/security/keytabs/http.hbase.keytab.
Entry for principal HTTP/solr.community.hortonworks.com with kvno 8, encryption type des-hmac-sha1 added to keytab WRFILE:/etc/security/keytabs/http.hbase.keytab.
Entry for principal HTTP/solr.community.hortonworks.com with kvno 8, encryption type des-cbc-md5 added to keytab WRFILE:/etc/security/keytabs/http.hbase.keytab.
Entry for principal hbase/solr.community.hortonworks.com with kvno 5, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/http.hbase.keytab.
Entry for principal hbase/solr.community.hortonworks.com with kvno 5, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/http.hbase.keytab.
Entry for principal hbase/solr.community.hortonworks.com with kvno 5, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/security/keytabs/http.hbase.keytab.
Entry for principal hbase/solr.community.hortonworks.com with kvno 5, encryption type arcfour-hmac added to keytab WRFILE:/etc/security/keytabs/http.hbase.keytab.
Entry for principal hbase/solr.community.hortonworks.com with kvno 5, encryption type des-hmac-sha1 added to keytab WRFILE:/etc/security/keytabs/http.hbase.keytab.
Entry for principal hbase/solr.community.hortonworks.com with kvno 5, encryption type des-cbc-md5 added to keytab WRFILE:/etc/security/keytabs/http.hbase.keytab.
kadmin:
For generating the keytab use the command
xst as you
can see above. The option
-k defines the location where the file that
contains the keytab(s) will be recorded and following parameter defines the
user(s) related to this keytab.
3 - HDPSEARCH INSTALLATION
yum -y install lucidworks-hdpsearch
chown -R solr:solr /opt/lucidworks-hdpsearch
The next steps must be executed with the solr user.
su solr
4 - CONFIGURING SOLR TO STORE INDEX FILES AT HDFS (Optional)
For the lab we’ll use the schemaless setup that comes with
solr;
·
Schemaless setup is a set of SOLR attributes that allows indexing
documents.
·
An example of this setup can be found at /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs.
Create the necessary directories at HDFS:
sudo -u hdfs hadoop fs -mkdir /user/solr
sudo -u hdfs hadoop fs -chown solr /user/solr
Edit the file located at the directory:
vi /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs_hdfs/conf/solrconfig.xml
Replace the section:
<directoryFactory name="DirectoryFactory"
class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}">
</directoryFactory>
by:
<directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory">
<str name="solr.hdfs.home">hdfs://sandbox.hortonworks.com/user/solr</str>
<bool name="solr.hdfs.blockcache.enabled">true</bool>
<int name="solr.hdfs.blockcache.slab.count">1</int>
<bool name="solr.hdfs.blockcache.direct.memory.allocation">false</bool>
<int name="solr.hdfs.blockcache.blocksperbank">16384</int>
<bool name="solr.hdfs.blockcache.read.enabled">true</bool>
<bool name="solr.hdfs.blockcache.write.enabled">false</bool>
<bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
<int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int>
<int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int>
</directoryFactory>
Change the locktype to:
<lockType>hdfs</lockType>
Save the file and exit.
5 - INITIALIZE SOLR IN CLOUD MODE
mkdir -p ~/solr-cores/core1
mkdir -p ~/solr-cores/core2
cp /opt/lucidworks-hdpsearch/solr/server/solr/solr.xml ~/solr-cores/core1
cp /opt/lucidworks-hdpsearch/solr/server/solr/solr.xml ~/solr-cores/core2
/opt/lucidworks-hdpsearch/solr/bin/solr start -cloud -p 8983 -z zookeeper.hortonworks.com:2181 -s ~/solr-cores/core1
/opt/lucidworks-hdpsearch/solr/bin/solr restart -cloud -p 8984 -z zookeeper.hortonworks.com:2181 -s ~/solr-cores/core2
6 - CREATE AND VALIDATE A COLLECTION NAMED “LABS” WITH 2 SHARDERS AND REPLICATION FACTOR OF 2
/opt/lucidworks-hdpsearch/solr/bin/solr create -c labs
-d /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs_hdfs/conf -n labs -s 2 -rf 2
Using the browser, visit http://solr.hortonworks.com:8983/solr/#/~cloud.
You should see the collection “labs” with 2 shards, each one of them with a
replication factor of 2.
7 - HBASE & HBASE-INDEXER CONFIGURATION
For pointing Solr to Zookeeper we need to configure the
hbase-indexer-site.xml file.
At the HBase-Indexer host, open the file located at:
vi /opt/lucidworks-hdpsearch/hbase-indexer/conf/hbase-indexer-site.xml
This is an example of how it should look like:
<?xml version="1.0"?>
<configuration>
<property>
<name>hbaseindexer.zookeeper.connectstring</name>
<value>zookeeperHOST1:2181,zookeeperHOST2:2181</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value> zookeeperHOST1,zookeeperHOST2</value>
</property>
</configuration>
Copy to the HBase server the necessary libraries to activate
the replication:
scp user@HOST_INDEXER:/opt/lucidworks-hdpsearch/hbase-indexer/lib/hbase-sep* user@HOST_HBASE:/usr/hdp/current/hbase-master/lib/
At AMBARI user interface (version 2.2.2.0), go to HBase ->
Configs -> Custom hbase-site and click “Add Property ...” and then add the
following properties:
hbase.replication=true
replication.source.ratio=1.0
replication.source.nb.capacity=1000
replication.replicationsource
Click the “Save” button. It should look like the screen bellow
having entered the above properties:
Restart the services requested by AMBARI.
Copy the hbase-site.xml file to the HBase-Indexer
configuration directory:
cp /etc/hbase/conf/hbase-site.xml /opt/lucidworks-hdpsearch/hbase-indexer/conf/
8 - Start the HBase-Indexer with the kerberized Cluster
Before anything else you’ll need to replace the zkcli.sh file
that comes with hbase-indexer. It prevents connecting to the kerberized HBase
so you can listen to the replication stream and thus generate json documents to
Solr.
cp /usr/hdp/2.4.2.0-258/zookeeper/bin/zkCli.sh /opt/lucidworks-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh
Create a JAAS file to notify the kerberos user which will be used
to authenticate:
vim /opt/lucidworks-hdpsearch/hbase-indexer/conf/hbase_indexer.jaas
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/etc/security/keytabs/hbase.service.keytab" #keytab do usuário
principal="hbase/All_FQDN@LOCAL.COM"; #usuário
};
Alter the new zkcli.sh to use this JAAS file:
vim /opt/lucidworks-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh
#Add the following information to the JAAS file to be able to authenticate
-Djava.security.auth.login.config=/opt/lucidworks-hdpsearch/hbase-indexer/conf/hbase_indexer.jaas
The result is:
All set! Now initialize the HBase-Indexer service:
/opt/lucidworks-hdpsearch/hbase-indexer/bin/hbase-indexer server
9 - Create an Index and Test the Solution
Access HBase and create a table:
hbase(main):001:0> create 'indexdemo-user', { NAME => 'info', REPLICATION_SCOPE => '1' }
Create a XML file to build the bridge between the contents of
the HBase table and what we’re indexing in Solr. Further examples and a read-me
file you may find at /opt/lucidworks-hdpsearch/hbase-indexer/demo.
vi /opt/lucidworks-hdpsearch/hbase-indexer/indexdemo-indexer.xml
<?xml version="1.0"?>
<indexer table="indexdemo-user">
<field name="firstname_s" value="info:firstname"/>
<field name="lastname_s" value="info:lastname"/>
<field name="age_i" value="info:age" type="int"/>
</indexer>
Add an index:
/opt/lucidworks-hdpsearch/hbase-indexer/bin/hbase-indexer add-indexer -n indexLab -c /opt/lucidworks-hdpsearch/hbase-indexer/indexdemo-indexer.xml -cp solr.zk=zk01.hortonworks.com,zk02.hortonworks.com,zk03.hortonworks.com -cp solr.collection=labs -z zk01.hortonworks.com:2181
Check if it was created and if it runs:
/opt/lucidworks-hdpsearch/hbase-indexer/bin/hbase-indexer list-indexers -z zk01.hortonworks.com:2181
indexLab
+ Lifecycle state: ACTIVE
+ Incremental indexing state: SUBSCRIBE_AND_CONSUME
+ Batch indexing state: INACTIVE
+ SEP subscription ID: Indexer_indexLab
+ SEP subscription timestamp: 2016-08-05T16:14:12.305-03:00
+ Connection type: solr
+ Connection params:
+ solr.zk = zk01.hortonworks.com,zk02.hortonworks.com,zk03.hortonworks.com
+ solr.collection = labs
+ Indexer config:
218 bytes, use -dump to see content
+ Indexer component factory: com.ngdata.hbaseindexer.conf.DefaultIndexerComponentFactory
+ Additional batch index CLI arguments:
(none)
+ Default additional batch index CLI arguments:
(none)
+ Processes
+ 1 running processes
+ 0 failed processes
Insert some records into HBase to verify it into Solr:
put 'indexdemo-user', 'row1', 'info:firstname', 'Gerdan'
put 'indexdemo-user', 'row1', 'info:lastname', 'Santos'
put 'indexdemo-user', 'row2', 'info:firstname', 'Pedro'
put 'indexdemo-user', 'row2', 'info:lastname', 'Drummond'
put 'indexdemo-user', 'row3', 'info:firstname', 'Cleire'
put 'indexdemo-user', 'row3', 'info:lastname', 'Ritfield'
put 'indexdemo-user', 'row4', 'info:firstname', 'Horton'
put 'indexdemo-user', 'row4', 'info:lastname', 'Works'
Perform a commit in Solr via terminal (
very important):
#Via terminal
curl http://solar.hortonworks.com:8983/solr/labs/update?commit=true
Or via browser:
Verify in Solr if the indexed data appears:
There you go! The HBase indexer with a kerberized cluster is
now working:
10 - References
https://community.hortonworks.com/articles/1181/hbase-indexing-to-solr-with-hdp-search-in-hdp-23.html
https://web.mit.edu/kerberos/krb5-1.12/doc/admin/admin_commands/kadmin_local.html
... View more
Labels: