Apache Solr 5.2.1 support for encryption of the index data stored on disk? ... View more

I'm seeing this message repeated over and over in the log. It doesn't seem to be causing an issue so can this log entry be disabled? 2016-08-12 14:54:37,199 ERROR server.TThreadPoolServer (TThreadPoolServer.java:run(296)) - Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:328) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ... View more

Documentation referred: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/configure-ranger_policy_admin_ha.html Environment Information: [ hivenv ] >> Cluster name hivenv-ambari-server.hwxblr.com >> 10.0.1.26 >> Ambari Server hn1.hwxblr.com >> 10.0.1.21 >> Existing Ranger Admin hn3.hwxblr.com >> 10.0.1.23 >> Load Balancer hn2.hwxblr.com >> 10.0.1.25 >> Additional Ranger Admin # hadoop version Hadoop 2.7.1.2.3.2.0-2950 Subversion git@github.com:hortonworks/hadoop.git -r5cc60e0003e33aa98205f18bccaeaf36cb193cc Compiled by jenkins on 2015-09-30T18:08Z Compiled with protoc 2.5.0 From source with checksum 69a3bf8c667267c2c252a54fbbf23d This command was run using /usr/hdp/2.3.2.0-2950/hadoop/lib/hadoop-common-2.7.1.2.3.2.0-2950.jar # uname -a Linux hn1.hwxblr.com 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/redhat-release CentOS release 6.7 (Final) Steps to be followed: Install the Ranger Admin component on the hosts you wish to use – hn1.hwxblr.com. For information about installing Ranger over Ambari, see Installing Ranger Over Ambari 2.0 . Configure a load balancer to balance the loads among the various Ranger Admin instances and take note of the load balancer URL. Step 1: Before Installing HAProxy on the server we need to install epel repository on our system depending on our operating system version using following command. CentOS/RHEL 5 , 32 bit: # rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm CentOS/RHEL 5 , 64 bit: # rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm CentOS/RHEL 6 , 32 bit: # rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm CentOS/RHEL 6 , 64 bit: # rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm Step 2: Install HAProxy using Yum. [root@hn3 ~]# yum install haproxy Step 3: Now we will configure HAProxy. [root@hn3 ~]# vi /etc/haproxy/haproxy.cfg Please refer for brief documentation before editing this file : https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Load_Balancer_Administration/ch-haproxy-setup-VSA.html [root@hn3 ~]# cat haproxy.cfg #--------------------------------------------------------------------- # Example configuration for a possible web application. See the # full configuration options online. # # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt # #--------------------------------------------------------------------- #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 30000 #--------------------------------------------------------------------- # round robin balancing between Ranger HA #--------------------------------------------------------------------- frontend haproxy bind 10.0.1.28:6080 reqadd X-Forwarded-Proto:\ http default_backend ranger_ha backend ranger_ha 10.0.1.28:6080 balance roundrobin mode http stats enable stats hide-version stats uri /stats stats realm Haproxy\ Statistics stats auth haproxy:redhat option httpchk option httpclose option forwardfor cookie LB insert server hn1.hwxblr.com 10.0.1.27:6080 cookie A check server hn2.hwxblr.com 10.0.1.29:6080 check [root@hn3 ~]# Step 5: Start the HAProxy service [root@hn3 ~]# service haproxy start Step 6: To make the HAProxy service persist through reboots [root@hn3 ~]# chkconfig haproxy on Update the Policy Manager external URL in all Ranger Admin clients (Ranger UserSync and Ranger plug-ins) to point to the load balancer URL. Ambari >> Ranger >> Configs >>Ranger Settings >> External URL (policymgr_external_url ) : http://hn3.hwxblr.com:6080 Enable Ranger Admin HA Wizard >> On Ambari >> Ranger >> Service Actions >> Enable Ranger HA URL to load balancer: http://10.0.1.23:6080 Select additional Ranger Admin as hn2.hwxblr.com Install Additional Ranger Admin Access Load Balancer URL: http://10.0.1.23:6080 , you should be able to access Ranger policies. ... View more

I am following the below documentation to install Zeppelin on SLES11 SP4 : http://hortonworks.com/hadoop-tutorial/apache-zeppelin-hdp-2-4/ I am aware of this bug: https://issues.apache.org/jira/browse/AMBARI-15659 and in /var/lib/ambari-server/resources/stacks/HDP/2.4/services/ZEPPELIN/metainfo.xml <osSpecific> <osFamily>suse11</osFamily> <packages> <package> <name>gcc-fortran</name> </package> <package> <name>libblas3</name> </package> <package> <name>liblapack3</name> </package> <package> <name>python-devel</name> </package> <!-- <package> <name>python-pip</name> </package> --> <package> <name>zeppelin</name> </package> </packages> </osSpecific> The error message is Package 'gcc-gfortran' not found . I want to understand from which file is it fetching the os-family and trying to install centos/redhat related packages instead of suse11 specific packages Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/2.4/services/ZEPPELIN/package/scripts/master.py", line 235, in <module> Master().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/2.4/services/ZEPPELIN/package/scripts/master.py", line 54, in install self.install_packages(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 404, in install_packages Package(name) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 49, in action_install self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/zypper.py", line 57, in install_package self.checked_call_until_not_locked(cmd, sudo=True, logoutput=self.get_logoutput()) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 72, in checked_call_until_not_locked return self.wait_until_not_locked(cmd, is_checked=True, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 80, in wait_until_not_locked code, out = func(cmd, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of '/usr/bin/zypper --quiet install --auto-agree-with-licenses --no-confirm gcc-gfortran' returned 104. Package 'gcc-gfortran' not found. ... View more

Use Case : 1 . ADDING a member to another group and being able to manage them internally without having to deal with outside or additional products. 2 . To be able to easily determine what members reside in what groups instead of having to scroll down page after page to see what members are in what groups especially when you have hundreds of users to keep track of. 3 . To easily administer various groups without having the hassle of creating more and more Active Directory/LDAP associations and having to submit change control requests to other departments for something we should be able to administer on our own. Ranger User Sync Process supports reading users and group information from one of the following sources: Unix Text file - CSV or JSON format LADP/AD CSV Format : If the filename does not end with .json, each line in the file will be treated as a delimiter separated fields of the following format. Default delimiter is a comma; this can be changed using configuration shown above. user-1,group-1,group-2,group-3 user-2,group-x,group-y,group-z CSV File Format e.g. UserGroupSyncFile.txt "user21","group20","group218","group26","group27","group262","group242","group219","group23" "user22","group20","group218","group26" "user23","user24","group20","group218" To run it as Command Line tool: java -Dlogdir=/var/log/ranger/usersync -cp /usr/hdp/current/ranger-usersync/dist/*:/usr/hdp/current/ranger-usersync/lib/*:/usr/hdp/current/ranger-usersync/conf org.apache.ranger.unixusersync.process.FileSourceUserGroupBuilder /tmp/UserGroupSyncFile.txt Steps : Create a group called solr_group and add certain users (imported from LDAP) into that group that we know will use SOLR. All the users are associated with the groups defined through LDAP and nothing else but we want to create additional groups and link users to those groups on Ranger. 1. Cluster with Ranger and configure with LDAP users. Here it is "packer". 2. Create a internal group on Ranger UI. Here it is "solr_group". 3. Edit an external LDAP user to add it to the group that we created. 4. Unable to edit the group field(greyed out) on Ranger UI for that LDAP user. [root@sandbox ~]# vi /tmp/ugsync.txt [root@sandbox ~]# cat /tmp/ugsync.txt "packer","packer","mygrp","test","solr_group" [root@sandbox ~]# java -Dlogdir=/var/log/ranger/usersync -cp /usr/hdp/current/ranger-usersync/dist/*:/usr/hdp/current/ranger-usersync/lib/*:/usr/hdp/current/ranger-usersync/conf org.apache.ranger.unixusersync.process.FileSourceUserGroupBuilder /tmp/ugsync.txt log4j: reset attribute= "false".log4j: Threshold ="null".log4j: Level value for root is [info].log4j: root level set to INFOlog4j: Class name: [org.apache.log4j.DailyRollingFileAppender]log4j: Setting property [file] to [/var/log/ranger/usersync/usersync.log].log4j: Setting property [datePattern] to ['.'yyyy-MM-dd].log4j: Parsing layout of class: "org.apache.log4j.PatternLayout"log4j: Setting property [conversionPattern] to [%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %m%n].log4j: setFile called: /var/log/ranger/usersync/usersync.log, truelog4j: setFile endedlog4j: Appender [logFile] to be rolled at midnight.log4j: Adding appender named [logFile] to category [root].log4j: /var/log/ranger/usersync/usersync.log -> /var/log/ranger/usersync/usersync.log.2016-04-04log4j: setFile called: /var/log/ranger/usersync/usersync.log, truelog4j: setFile ended [root@sandbox ~]# cd /var/log/ranger/usersync ... View more

HBase tables are not highly available by default. To enable high availability, designate a table as HA during table creation using the HBase shell using the REGION_REPLICATION keyword. As the following example creates a table named t1 that is replicated to one secondary replica: CREATE 't1', 'f1', {REGION_REPLICATION => 2} To replicate t1 to two secondary regions, set REGION_REPLICATION to 3: CREATE 't1', 'f1', {REGION_REPLICATION => 3} How we can prevent users from enabling their tables HA ? ... View more

We have configured Ranger User Sync process to read users and group information from a text file with CSV File Format e.g. UserGroupSyncFile.txt "user21","group20","group218","group26","group27","group262","group242","group219","group23" "user22","group20","group218","group26" "user23", "user24","group20","group218" To modify a user and the groups they belong to, is there also a command at the group level, where you can specify a file and have the group name and all of the members the belong to that group? (ie: "group-name", "user-1", "user-2", "user-3" ) ... View more

Is there a dependency in Ambari/HDP so we have to install Oracle jdbc jars in all slave node . Hadoop uses jdbc jars coming with instant client to connect Oracle backend databases, and as Hadoop administrators use sqlplus to verify the database and do trouble-shooting. ... View more