Ambari Server version: 2.5.2.0-298 HDP version: 2.6.4.0-91 Ansible version: 2.4 OS version: centos7 There are options of deployment tools to do automatic HDP installation, one of the reasons for why ansible is popular is ansible is lightweight and it totally relies on ssh connection without any physical agent. But ansible script has its organizational structure that we will not talk here in detail, and for people who are not familiar with its grammar and its running logic, can reference to this quick tutorial https://serversforhackers.com/c/an-ansible2-tutorial but when you delve into full deck of scripts for hdp installation, you'd better to have the document along with https://docs.ansible.com/ansible/2.4/porting_guide_2.4.html And beware of composing the full ansible scripts from scratch is a little bit boring, to save time you can build your own on some other existing scripts, for hdp installation we can start with the following link, but to fit for your case, still, need to do tailoring https://github.com/hortonworks/ansible-hortonworks Script editor can help improve your work efficiency, here I recommend composing script in IntelliJ IDEA which can give you a neat and clear layout to put things in the place. So, we actually now have a project built in IntelliJ IDEA. Next, I'm gonna give you a full picture of this ansible project structure. Basically, we first need to define the cluster layout, so we have an inventory file 'static' as shown in the above, and then to divide the installation into a sequence of macro steps or phases, we then defined a bunch of roles, which organize related tasks and data into one coherent structure. Roles have a directory structure here we only use some of them including default, tasks, templates, and vars. Default and tasks directory respectively contain a main.yml which put together a sequence of micro-steps into a series of tasks. Template files can contain template variables, based on Python's Jinja2 template engine. Files in here should end in .j2, but can otherwise have any name. In our project, we use the template to generate blueprint or local repository files. The vars directory contains yml files which simply list variables we'll use in the roles. Now, let me walk you through the installation process. The script "deploy.cluster.yml" gives out a workflow for the installation as following, # Will Prepare the Nodes - include: "prepare_nodes.yml" # Install Repository - include: "deploy.httpd.yml" # Install Ambari Agent - include: "install_ambari.yml" tags: install_ambari # Register local repository to Ambari Server - include: "configure_ambari.yml" # Generate blueprint - include: "apply_blueprint.yml" tags: blueprint - include: "post_install.yml" let's say we will build from clean os only with password-less ssh configured, so the very first step is doing some basic package installation and os settings. packages: - python-httplib2 # required for calling Ambari API from Ansible - openssh-clients # scp required by Ambari - curl # curl required by Ambari - unzip # unzip required by Ambari - tar # tar required by Ambari - wget # wget required by Ambari - openssl # openssl required by Ambari - chrony # ntp required by Hadoop ntp_service_name: chronyd firewall_service_name: firewalld update_grub_command: 'grub2-mkconfig -o "$(readlink -n /etc/grub2.cfg)"' And this preparation is fulfilled by "common" role, next we step to configure repositories for ambari and hdp. Let's say in most cases the cluster has limit to access the internet, so better download and configure local repositories for cluster nodes to use. To this end, we need do two things: build up HTTP server and put the repository in the root directory, second is generate repo files and scatter across the cluster. The following is the snippet from tasks of role "offline-repo", this is for downloading and decompose the tarball into http server root directory. # download the ambari tarball- name: download the ambari tarball get_url: url: "http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/{{ ambari_version.split('-')[0] }}/ambari-{{ ambari_version.split('-')[0] }}-centos7.tar.gz"dest: "/var/www/html/ambari.tar.gz"# unzip the ambari tarball- name: unzip the ambari tarball command: tar -zxvf /var/www/html/ambari.tar.gz -C /var/www/html/ # download the hdp tarball- name: download the hdp tarball get_url: url: "http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/{{ hdp_version.split('-')[0] }}/HDP-{{ hdp_version.split('-')[0] }}-centos7-rpm.tar.gz"dest: "/var/www/html/hdp/hdp.tar.gz"# unzip the hdp tarball- name: unzip the hdp tarball command: tar -zxvf /var/www/html/hdp/hdp.tar.gz -C /var/www/html/hdp/ # download the hdp tarball- name: download the hdp tarball get_url: url: "http://public-repo-1.hortonworks.com/HDP-UTILS-{{ utils_version }}/repos/centos7/HDP-UTILS-{{ utils_version }}-centos7.tar.gz"dest: "/var/www/html/hdp/hdp-utilits.tar.gz"# unzip the hdp-utilits tarball- name: unzip the hdp-utilits tarball command: tar -zxvf /var/www/html/hdp/hdp-utilits.tar.gz -C /var/www/html/hdp/ # download the hdp-gpl tarball- name: download the hdp tarball get_url: url: "http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/{{ hdp_version.split('-')[0] }}/HDP-GPL-{{ hdp_version.split('-')[0] }}-centos7-rpm.tar.gz"dest: "/var/www/html/hdp.gpl.tar.gz"# unzip the hdp-utilits tarball- name: unzip the hdp-gpl tarball command: tar -zxvf /var/www/html/hdp.gpl.tar.gz -C /var/www/html/ And then generate repo files according to the repo template. This is the repo for ambari, need substitute placeholder ambari_version with the exact version number that we gives in vars file "all" in group vars directory [AMBARI-{{ ambari_version }}] name=AMBARI Version - AMBARI-{{ ambari_version }} baseurl=http://{{ groups['repo'][0] }}/ambari/centos7 gpgcheck=1 gpgkey=http://{{ groups['repo'][0] }}/ambari/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 And this is repo for hdp, need substitute the placeholder hdp_version with the exact version number that we gives in vars file "all" in group vars directory #VERSION_NUMBER={{ hdp_version }} [HDP-{{ hdp_version }}] name=HDP Version - HDP-{{ hdp_version }} baseurl=http://{{ groups['repo'][0] }}/hdp/HDP/centos7/{{ hdp_version }} gpgcheck=1 gpgkey=http://{{ groups['repo'][0] }}/hdp/HDP/centos7/{{ hdp_version }}/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [HDP-UTILS-{{ utils_version }}] name=HDP-UTILS Version - HDP-UTILS-{{ utils_version }} baseurl=http://{{ groups['repo'][0] }}/hdp gpgcheck=1 gpgkey=http://{{ groups['repo'][0] }}/hdp/HDP/centos7/{{ hdp_version }}/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 After deploy repo files across cluster nodes, then we start to install ambari server and agents via role "ambari-server" and "ambari-agent" respectively. Because Ambari-server relies on a database to store metadata to maintain health and manage the cluster. Let's say we use mysql to do the database job, then here has role "database" to install mysql and configure database and users as follows, - block: - name: Create the ambari database (mysql) mysql_db: name: "{{ database_options.ambari_db_name }}" state: present - name: Create the ambari database user and host based access (mysql) mysql_user: name: "{{ database_options.ambari_db_username }}" host: "{{ hostvars[inventory_hostname]['ansible_fqdn'] }}" priv: "{{ database_options.ambari_db_name }}.*:ALL" password: "{{ database_options.ambari_db_password }}" state: present - name: Create the ambari database user and IP based access (mysql) mysql_user: name: "{{ database_options.ambari_db_username }}" host: "{{ hostvars[inventory_hostname]['ansible_'~hostvars[inventory_hostname].ansible_default_ipv4.alias]['ipv4']['address'] }}" priv: "{{ database_options.ambari_db_name }}.*:ALL" password: "{{ database_options.ambari_db_password }}" state: present - name: Create the hive database (mysql) mysql_db: name: "{{ database_options.hive_db_name }}" state: present when: hiveserver_hosts is defined and hiveserver_hosts|length > 0 - name: Create the hive database user and host based access (mysql) mysql_user: name: "{{ database_options.hive_db_username }}" host: "{{ hostvars[item]['ansible_fqdn'] }}" priv: "{{ database_options.hive_db_name }}.*:ALL" password: "{{ database_options.hive_db_password }}" state: present with_items: "{{ hiveserver_hosts }}" when: hiveserver_hosts is defined and hiveserver_hosts|length > 0 - name: Create the hive database user and IP based access (mysql) mysql_user: name: "{{ database_options.hive_db_username }}" host: "{{ hostvars[item]['ansible_'~hostvars[item].ansible_default_ipv4.alias]['ipv4']['address'] }}" priv: "{{ database_options.hive_db_name }}.*:ALL" password: "{{ database_options.hive_db_password }}" state: present with_items: "{{ hiveserver_hosts }}" when: hiveserver_hosts is defined and hiveserver_hosts|length > 0 - name: Create the oozie database (mysql) mysql_db: name: "{{ database_options.oozie_db_name }}" state: present when: oozie_hosts is defined and oozie_hosts|length > 0 - name: Create the oozie database user and host based access (mysql) mysql_user: name: "{{ database_options.oozie_db_username }}" host: "{{ hostvars[item]['ansible_fqdn'] }}" priv: "{{ database_options.oozie_db_name }}.*:ALL" password: "{{ database_options.oozie_db_password }}" state: present with_items: "{{ oozie_hosts }}" when: oozie_hosts is defined and oozie_hosts|length > 0 - name: Create the oozie database user and IP based access (mysql) mysql_user: name: "{{ database_options.oozie_db_username }}" host: "{{ hostvars[item]['ansible_'~hostvars[item].ansible_default_ipv4.alias]['ipv4']['address'] }}" priv: "{{ database_options.oozie_db_name }}.*:ALL" password: "{{ database_options.oozie_db_password }}" state: present with_items: "{{ oozie_hosts }}" when: oozie_hosts is defined and oozie_hosts|length > 0 - name: Create the ranger admin database (mysql) mysql_db: name: "{{ database_options.rangeradmin_db_name }}" state: present when: rangeradmin_hosts is defined and rangeradmin_hosts|length > 0 - name: Create the ranger admin database user and host based access (mysql) mysql_user: name: "{{ database_options.rangeradmin_db_username }}" host: "{{ hostvars[item]['ansible_fqdn'] }}" priv: "{{ database_options.rangeradmin_db_name }}.*:ALL" password: "{{ database_options.rangeradmin_db_password }}" state: present with_items: "{{ rangeradmin_hosts }}" when: rangeradmin_hosts is defined and rangeradmin_hosts|length > 0 - name: Create the ranger admin database user and IP based access (mysql) mysql_user: name: "{{ database_options.rangeradmin_db_username }}" host: "{{ hostvars[item]['ansible_'~hostvars[item].ansible_default_ipv4.alias]['ipv4']['address'] }}" priv: "{{ database_options.rangeradmin_db_name }}.*:ALL" password: "{{ database_options.rangeradmin_db_password }}" state: present with_items: "{{ rangeradmin_hosts }}" when: rangeradmin_hosts is defined and rangeradmin_hosts|length > 0 - name: Create the registry database (mysql) mysql_db: name: "{{ database_options.registry_db_name }}" state: present when: registry_hosts is defined and registry_hosts|length > 0 - name: Create the registry database user and host based access (mysql) mysql_user: name: "{{ database_options.registry_db_username }}" host: "{{ hostvars[item]['ansible_fqdn'] }}" priv: "{{ database_options.registry_db_name }}.*:ALL" password: "{{ database_options.registry_db_password }}" state: present with_items: "{{ registry_hosts }}" when: registry_hosts is defined and registry_hosts|length > 0 - name: Create the registry database user and IP based access (mysql) mysql_user: name: "{{ database_options.registry_db_username }}" host: "{{ hostvars[item]['ansible_'~hostvars[item].ansible_default_ipv4.alias]['ipv4']['address'] }}" priv: "{{ database_options.registry_db_name }}.*:ALL" password: "{{ database_options.registry_db_password }}" state: present with_items: "{{ registry_hosts }}" when: registry_hosts is defined and registry_hosts|length > 0 - name: Create the streamline database (mysql) mysql_db: name: "{{ database_options.streamline_db_name }}" state: present when: streamline_hosts is defined and streamline_hosts|length > 0 - name: Create the streamline database user and host based access (mysql) mysql_user: name: "{{ database_options.streamline_db_username }}" host: "{{ hostvars[item]['ansible_fqdn'] }}" priv: "{{ database_options.streamline_db_name }}.*:ALL" password: "{{ database_options.streamline_db_password }}" state: present with_items: "{{ streamline_hosts }}" when: streamline_hosts is defined and streamline_hosts|length > 0 - name: Create the streamline database user and IP based access (mysql) mysql_user: name: "{{ database_options.streamline_db_username }}" host: "{{ hostvars[item]['ansible_'~hostvars[item].ansible_default_ipv4.alias]['ipv4']['address'] }}" priv: "{{ database_options.streamline_db_name }}.*:ALL" password: "{{ database_options.streamline_db_password }}" state: present with_items: "{{ streamline_hosts }}" when: streamline_hosts is defined and streamline_hosts|length > 0 when: database == "mysql" or database == "mariadb"<br> After ambari configured, we then start hdp installation by uploading blueprint. For generating the blueprint we need beforehand to give which compoents should install on which cluster node, and that we already define in vars file as shown in the following picture. And then role "ambari-blueprint" parse the var "blueprint_dynamic" definition and apply template to generate corresponding blueprint file, { "configurations" : [ { "hadoop-env" : { "dtnode_heapsize" : "1024m", "namenode_heapsize" : "2048m", "namenode_opt_maxnewsize" : "384m", "namenode_opt_newsize" : "384m" } }, { "hdfs-site" : { "dfs.datanode.data.dir" : "/hadoop/hdfs/data", "dfs.datanode.failed.volumes.tolerated" : "0", "dfs.replication" : "3" } }, { "yarn-site" : { "yarn.client.nodemanager-connect.retry-interval-ms" : "10000" } }, { "hive-site" : { "javax.jdo.option.ConnectionDriverName": "com.mysql.jdbc.Driver", "javax.jdo.option.ConnectionURL": "jdbc:mysql://cheny0.field.hortonworks.com:3306/hive", "javax.jdo.option.ConnectionUserName": "hive", "javax.jdo.option.ConnectionPassword": "hive", "hive.metastore.failure.retries" : "24" } }, { "hiveserver2-site" : { "hive.metastore.metrics.enabled" : "true" } }, { "hive-env" : { "hive_database": "Existing MySQL / MariaDB Database", "hive_database_type": "mysql", "hive_database_name": "hive", "hive_user" : "hive" } }, { "oozie-site" : { "oozie.service.JPAService.jdbc.driver": "com.mysql.jdbc.Driver", "oozie.service.JPAService.jdbc.url": "jdbc:mysql://cheny0.field.hortonworks.com:3306/oozie", "oozie.db.schema.name": "oozie", "oozie.service.JPAService.jdbc.username": "oozie", "oozie.service.JPAService.jdbc.password": "oozie", "oozie.action.retry.interval" : "30" } }, { "oozie-env" : { "oozie_database": "Existing MySQL / MariaDB Database", "oozie_user" : "oozie" } }, { "hbase-site" : { "hbase.client.retries.number" : "35" } }, { "core-site": { "fs.trash.interval" : "360" } }, { "storm-site": { "storm.zookeeper.retry.intervalceiling.millis" : "30000" } }, { "kafka-broker": { "zookeeper.session.timeout.ms" : "30000" } }, { "zoo.cfg": { "clientPort" : "2181" } } ], "host_groups" : [ { "name" : "master-nodes", "configurations" : [ ], "components" : [ { "name" : "MAPREDUCE2_CLIENT" }, { "name" : "YARN_CLIENT" }, { "name" : "HDFS_CLIENT" }, { "name" : "INFRA_SOLR_CLIENT" }, { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "ZOOKEEPER_SERVER" }, { "name" : "METRICS_MONITOR" }, { "name" : "INFRA_SOLR" }, { "name" : "NAMENODE" }, { "name" : "METRICS_GRAFANA" }, { "name" : "NODEMANAGER" } ] }, { "name" : "second-master-nodes", "configurations" : [ ], "components" : [ { "name" : "MAPREDUCE2_CLIENT" }, { "name" : "YARN_CLIENT" }, { "name" : "HDFS_CLIENT" }, { "name" : "INFRA_SOLR_CLIENT" }, { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "ZOOKEEPER_SERVER" }, { "name" : "SECONDARY_NAMENODE" }, { "name" : "RESOURCEMANAGER" }, { "name" : "APP_TIMELINE_SERVER" }, { "name" : "HISTORYSERVER" }, { "name" : "METRICS_MONITOR" }, { "name" : "DATANODE" }, { "name" : "NODEMANAGER" } ] }, { "name" : "slave-nodes", "configurations" : [ ], "components" : [ { "name" : "MAPREDUCE2_CLIENT" }, { "name" : "YARN_CLIENT" }, { "name" : "HDFS_CLIENT" }, { "name" : "INFRA_SOLR_CLIENT" }, { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "ZOOKEEPER_SERVER" }, { "name" : "DATANODE" }, { "name" : "NODEMANAGER" }, { "name" : "METRICS_MONITOR" }, { "name" : "METRICS_COLLECTOR" } ] } ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.6" } } As known, the blueprint can trigger Ambari server to do the automatic installation of hdp without manual operations. The underlying procedure implemented in ansible script is same as the standard blueprint installation as in the following link https://community.hortonworks.com/content/kbentry/47171/automate-hdp-installation-using-ambari-blueprints-1.html And now when you turn to web browser and open the first page of ambari server, you'll see the following installation progress bar And finally, we will get an installed hdp launched on your cluster Now, need to explain why I don't employ the latest version 2.6.1.5-3 of Ambari server, because I tried it coupled with HDP 2.6.4.0-91, often reapt the failure on master node. That error says cannot match package for hadoop_${stack_version}-mapreduce when it's trying to install mapreduce2 client. I ever tried to first manually successfully install the hpd, and then export blueprint and use this same blueprint to trigger the installation and also block at the same failure. I've checked that this package is actually installed on the master node during ambari server installation, it should not prompt up again when installing hdp. It might be a bug, but here need more effort to dive into the concret ambari Python script, so a way around is downgrade ambari server, that is why here we use ambari server 2.5*. More important thing that I almost forget is my full deck of ansible script that you leave your mailbox address below, that I can share, sorry for the time being cannot share in the clould. ... View more

The HDP document present details steps of configuring SSL for each service component. However, more than often the services and corresponding applications work in the composite security environment, such as authentication with wire encryption, saying Kerberos + SSL. The blog in below link provided a solution for Kafka after being ssl secured. The solution is to join the certificate of Kafka brokers in JDK truststore. https://community.hortonworks.com/articles/55154/kafka-ssl-kerberos-cheat-sheet-settingsconsole-com.html Here I want to intensify a bit further, we will talk about the solution that places Kafka brokers' certificates outside of JDK truststore, and methods to testify. And for Kafka client Java application, how it works with ssl configuration, as well as that for Spark streaming successfully connecting ssl-secured Kafka brokers. Before those topics, we need to specify how to conduct ssl configuration for Kafka broker. The environment is HDP 2.6.3.0 with FreeIPA kerberized, Spark 2.2.0 and Kafka 0.10.1 with ssl securied. 1. configure ssl for Kafka 1.1 On each broker, create keystore files, certificates, and truststore files. a. Create a keystore file: keytool -genkey -alias <host> -keyalg RSA -keysize 1024 –dname CN=<host>,OU=hw,O=hw,L=paloalto,ST=ca,C=us –keypass <KeyPassword> -keystore <keystore_file> -storepass <storePassword> b. Create a certificate: keytool -export -alias <host> -keystore <keystore_file> -rfc –file <cert_file> -storepass <StorePassword> c. Create a truststore file: keytool -import -noprompt -alias <host> -file <cert_file> -keystore <truststore_file> -storepass <truststorePassword> 1.2 Create one truststore file that contains the public keys from all certificates. a. Log on to one host and import the truststore file for that host: keytool -import -noprompt -alias <hostname> -file <cert_file> -keystore <all_jks> -storepass <allTruststorePassword> b. Copy the <all_jks> file to the other nodes in your Kafka cluster, and repeat the keytool command on each broker server. 2. Configure SSL for Kafka Borker Suppose we have those keystore and truststore in the flowing path on each broker: [root@cheny0 ssl]# ls -l total 12 -rw-r--r-- 1 root root731 Nov5 04:24 cheny0.field.hortonworks.com.crt -rw-r--r-- 1 root root 1283 Nov5 04:22 cheny0.field.hortonworks.com.jks -rw-r--r-- 1 root root577 Nov5 04:24 cheny0.field.hortonworks.com_truststore.jks [root@cheny0 ssl]# pwd /temp/ssl<br> Then we configure in Ambair->Kafka->Configuration as following, we wanna plaintext communication among brokers, and configure ssl port as 6668, plaintext port as 6667. and for another ssl configuration, we need to customize save and restart to make the configuration take effect, and then using 'netstat -pan |grep 6668' to check if it really start. Or we should find a similar line in /var/log/kafka/server.log as follows [2017-11-05 04:34:15,789] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: PLAINTEXTSASL -> EndPoint(cheny0.field.hortonworks.com,6667,PLAINTEXTSASL),SASL_SSL -> EndPoint(cheny0.field.hortonworks.com,6668,SASL_SSL) (kafka.utils.ZkUtils) To testify ssl port, we can use command openssl s_client -debug -connect localhost:9093 -tls1 And it should respond the certificate of connected broker, as shown in following Server certificate -----BEGIN CERTIFICATE----- MIIB6DCCAVGgAwIBAgIEaSkk1DANBgkqhkiG9w0BAQsFADAnMSUwIwYDVQQDExxj aGVueTAuZmllbGQuaG9ydG9ud29ya3MuY29tMB4XDTE3MTEwNDIwMjIyMFoXDTE4 MDIwMjIwMjIyMFowJzElMCMGA1UEAxMcY2hlbnkwLmZpZWxkLmhvcnRvbndvcmtz LmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAp9KsltYleQQ7KE8ZG4LN lb5Jlj6kJuuYDDgOJftqMlelbGqnAEbn6cpvD13GP710jwOC1kINMANu6Y4cRNFs 3s4EuWC/hFA4SDzUnAaqvhBbAHPqDx1WonSg4P6333/O1v1Or5vHNaQiaEY7EWcx shU5PrtDwudo9s38L2HYsMcCAwEAAaMhMB8wHQYDVR0OBBYEFI5OYMK1mbL14ypD gcxRtxFI3eZ8MA0GCSqGSIb3DQEBCwUAA4GBAAcP33K8Ld3ezJxN2ss0BeW83ouw DUvAuZ6VU+TtSdo6x5uwkVlIIvITcoKcVd4O75idhIAxPajIBspzojsKg1rgGjlc 2G4pYjUbkH/P+Afs/YMWRoH91BKdAzhA4/OhiCq2xV+s+AKTgXUVmHnRkBtaTvX8 6Oe7jmXIouzJkiU/ -----END CERTIFICATE----- So far, that only proves 6668 is really listening and it's ssl secured. But not be sure the ssl we configured works or not. That needs us using Kafka tools to testify further. 3. Enable Kafka Tools after SSL Configured 3.1 first we check if the plaintext port work The following command steps lead you to test if Kafka works on plaintext port 6667, and there use Kerberos configuration, which links to kafka_client_jass.conf and kafka_jaas.conf in path /usr/hdp/current/kafka-broker/conf/. If there prompt out errors of Kerberos/GSSAPI, should check the jaas configuration. //log in user 'kafka' [root@cheny0 bin]# kinit -kt /etc/security/keytabs/kafka.service.keytab kafka/cheny0.field.hortonworks.com@FIELD.HORTONWORKS.COM //move to path where place Kafka tools script [root@cheny0 ~]# cd /usr/hdp/current/kafka-broker/bin //launch producer [root@cheny0 bin]# ./kafka-console-producer.sh --broker-list cheny0.field.hortonworks.com:6667 --topic test --security-protocol PLAINTEXTSASL //launch consumer [root@cheny0 bin]# ./kafka-console-consumer.sh --bootstrap-server cheny0.field.hortonworks.com:6667 --topic test --from-beginning --security-protocol PLAINTEXTSASL 3.2 check if the ssl port 6668 work The key point lies in configure ssl properties in consumer.properties and producer.properties files under path /usr/hdp/current/kafka-broker/config/. If here encounter problems we should first proofread the properties we configured, and then doubt on if the key or truststore is correct. //log in user 'kafka' [root@cheny0 bin]# kinit -kt /etc/security/keytabs/kafka.service.keytab kafka/cheny0.field.hortonworks.com@FIELD.HORTONWORKS.COM //move to path where place Kafka tools script [root@cheny0 ~]# cd /usr/hdp/current/kafka-broker/bin //add the following lines in /usr/hdp/current/kafka-broker/config/producer.properties and consumer.properties ssl.keystore.location = /temp/ssl/cheny0.field.hortonworks.com.jks ssl.keystore.password = password ssl.key.password = password ssl.truststore.location = /temp/ssl/cheny0.field.hortonworks.com_truststore.jks ssl.truststore.password = password //launch producer [root@cheny0 bin]# ./kafka-console-producer.sh --broker-list cheny0.field.hortonworks.com:6668 --topic test --security-protocol SASL_SSL --producer.config /usr/hdp/current/kafka-broker/config/producer.properties //launch consumer [root@cheny0 bin]# ./kafka-console-consumer.sh --new-consumer --bootstrap-server cheny0.field.hortonworks.com:6668 --topic test --from-beginning --security-protocol SASL_SSL --consumer.config /usr/hdp/current/kafka-broker/config/consumer.properties If the bove passed through, that say your Kafka ssl configuration work fine. Then we get to next talking how for Kafka Java applications to work. 4. Enable Kafka Java Applications after SSL Configured For Kafka performance tool, if we change to use 'java' command to launch the ProducerPerformance, the piont is we can use option --producer.config to link the ssl properties we configured the same in previous section. //launch java application to procuding test message java -Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/conf/kafka_jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=true -cp /usr/hdp/current/kafka-broker/libs/scala-library-2.11.8.jar:/usr/hdp/current/kafka-broker/libs/*:/temp/kafka-tools-1.0.0-SNAPSHOT.jar org.apache.kafka.tools.ProducerPerformance --num-records 500000 --record-size 1000 --topic chen --throughput 100000 --num-threads 2 --value-bound 5000 --print-metrics --producer-props bootstrap.servers=cheny0:6668 compression.type=gzip max.in.flight.requests.per.connection=1 linger.ms=10 security.protocol=SASL_SSL --producer.config=/usr/hdp/current/kafka-broker/config/producer.properties //lauch consumer to receive the test message [root@cheny0 bin]# ./kafka-console-consumer.sh --new-consumer --bootstrap-server cheny0.field.hortonworks.com:6668 --topic chen --security-protocol SASL_SSL --consumer.config /usr/hdp/current/kafka-broker/config/consumer.properties For generalized Java application where it uses Kafka client, then we should finalize the properties configurations in Java code. That is we will introduce in next section where Spark application connects Kafka with SSL+Kerberos. 5. Spark Streaming Connects SSL-secured Kafka in Kerberos Environment Becuase here we launch Spark or Spark streaming on yarn cluster, and the spark (streaming) job is first uploaded onto hdfs path /user/kakfa.., and distributed into each executors. So, if it connects to Kafka and needs to include credentials of Kafka, i.e., jaas and keytab, so that can successfully reach Kafka broker. I had previousaly written an artical on this topic, for those who is interested can be referred to the following link https://community.hortonworks.com/content/kbentry/139597/other-notable-points-for-sparkstreaming-connecting.html And the following use the same Spark streaming Java program that we introduce in the above weblink. The part that we should pay more attention in the following snippets, is the Java code which includes Kafak ssl properties configuration. If it fails to run, you should first proofread those configuration code. //first time executing, creat kafka path in hdfs [root@cheny0 temp]# kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-cheny@FIELD.HORTONWORKS.COM [root@cheny0 temp]# hdfs dfs -mkdir /user/kafka [root@cheny0 temp]# hdfs dfs -chown kafka:hdfs /user/kafka //change the benchmark java code HashMap<String, Object> kafkaParams = new HashMap<String, Object>(); kafkaParams.put( "bootstrap.servers", "cheny0.field.hortonworks.com:6668" ); kafkaParams.put( "group.id", "test_sparkstreaming_kafka" ); kafkaParams.put( "auto.offset.reset", "latest" ); kafkaParams.put( "security.protocol", "SASL_SSL" ); kafkaParams.put("key.deserializer", StringDeserializer.class); kafkaParams.put("value.deserializer", StringDeserializer.class); kafkaParams.put("ssl.keystore.location","/temp/ssl/cheny0.field.hortonworks.com.jks" ); kafkaParams.put("ssl.keystore.password","password" ); kafkaParams.put("ssl.key.password","password" ); kafkaParams.put("ssl.truststore.location","/temp/ssl/cheny0.field.hortonworks.com_truststore.jks" ); kafkaParams.put("ssl.truststore.password", "password" ); //compile out the application jar and copy to broker's path cheny0.field.hortonworks.com:/temp sh-3.2# cd /Users/chen.yang/workspace/kafkaproducer sh-3.2# mvn package sh-3.2# cd target sh-3.2# scp kafka-producer-1.0-SNAPSHOT.jar cheny0:/tmp //copy keytab and jass to /temp [root@cheny0 temp]# scp /usr/hdp/current/kafka-broker/conf/kafka_jaas.conf /temp/ [root@cheny0 temp]# scp /etc/security/keytabs/kafka.service.keytab ./ //revise the jaas.config KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true doNotPrompt=true principal="kafka/cheny0.field.hortonworks.com@FIELD.HORTONWORKS.COM" keyTab="kafka.service.keytab" useTicketCache=true renewTicket=true serviceName="kafka"; }; Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true doNotPrompt=true principal="kafka/cheny0.field.hortonworks.com@FIELD.HORTONWORKS.COM" keyTab="kafka.service.keytab" useTicketCache=true renewTicket=true serviceName="kafka"; }; //copy /temp to other yarn cluster node [root@cheny0 temp]# scp -r /temp cheny1:/ [root@cheny0 temp]# scp -r /temp cheny2:/ [root@cheny0 bin]# kinit -kt /etc/security/keytabs/kafka.service.keytab kafka/cheny0.field.hortonworks.com@FIELD.HORTONWORKS.COM [root@cheny0 temp]# spark-submit --master=yarn --deploy-mode=cluster --files /temp/kafka_jaas.conf,/temp/kafka.service.keytab --conf "spark.driver.extraJavaOptions=-Djava.security.auth.login.config=kafka_jaas.conf" --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=kafka_jaas.conf" --jars /usr/hdp/current/spark2-client/jars/spark-streaming_2.11-2.2.0.2.6.3.0-235.jar,/usr/hdp/current/spark2-client/jars/spark-streaming-kafka-0-10_2.11-2.2.0.2.6.3.0-235.jar,/usr/hdp/current/kafka-broker/libs/kafka-clients-0.10.1.2.6.3.0-235.jar,/usr/hdp/current/spark2-client/examples/jars/spark-examples_2.11-2.2.0.2.6.3.0-235.jar --class spark2.example.Spark2Benchmark /temp/kafka-producer-1.0-SNAPSHOT.jar //kill the job to save yarn resource [root@cheny0 temp]# yarn application -kill application_1509825587944_0012 17/11/05 15:41:00 INFO client.RMProxy: Connecting to ResourceManager at cheny1.field.hortonworks.com/xxx.xx.xxx.243:8050 17/11/05 15:41:01 INFO client.AHSProxy: Connecting to Application History server at cheny1.field.hortonworks.com/xxx.xx.xxx.xxx:10200 Killing application application_1509825587944_0012 17/11/05 15:41:01 INFO impl.YarnClientImpl: Killed application application_1509825587944_0012 6. Summary To check and enable those Kafka command, scripts and applications to work after ssl secured is not easier than that of Kerberos configuration. So, we should also check at each step to make sure we need not set back. That is the way I learn. Hope you guys get success as well. And for any suggestion or question, please fee free to leave your commend below. Have a good day. ... View more

Most of online article already specified the code and submission command for Spark connecting Kafka in kerberos environment. Here we talk other issues you probably encountered when doing debugging, which are of the kind of easily ignored. I experienced and be tortured, but no pain no gain. Here records them and hope do help to you. The environment is HDP 2.6.1.0 with FreeIPA kerberized, Spark 1.6.3 and Kafka 0.10.1 The test code I borrowed from below link, but added lines enabling it running in kerberized. https://github.com/eBay/Spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKafkaWordCount.java package spark.example; import java.util.HashMap; import java.util.HashSet; import java.util.Arrays; import java.util.regex.Pattern; import scala.Tuple2; import com.google.common.collect.Lists; import kafka.serializer.StringDecoder; import org.apache.spark.SparkConf; import org.apache.spark.api.java.function.*; import org.apache.spark.streaming.api.java.*; import org.apache.spark.streaming.kafka.KafkaUtils; import org.apache.spark.streaming.Durations; public final class JavaDirectKafkaWordCount { private static final Pattern SPACE = Pattern.compile( " " ); public static void main(String[] args) { // Create context with a 2 seconds batch interval SparkConf sparkConf = new SparkConf().setAppName("JavaDirectKafkaWordCount"); JavaStreamingContext jssc = new JavaStreamingContext( sparkConf, Durations.seconds( 200 ) ); HashSet<String> topicsSet = new HashSet<String>( Arrays.asList( "chen" ) ); HashMap<String, String> kafkaParams = new HashMap<String,String>(); kafkaParams.put("bootstrap.servers", "cheny0.field.hortonworks.com:6667"); kafkaParams.put("group.id", "test_sparkstreaming_kafka"); kafkaParams.put("auto.offset.reset", "largest"); kafkaParams.put("security.protocol", "PLAINTEXTSASL"); System.setProperty("java.security.auth.login.config","/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf"); System.setProperty("java.security.krb5.conf", "/etc/krb5.conf"); // Create direct kafka stream with brokers and topics JavaPairInputDStream<String, String> messages = KafkaUtils.createDirectStream( jssc, String.class, String.class, StringDecoder.class, StringDecoder.class, kafkaParams, topicsSet ); // Get the lines, split them into words, count the words and print JavaDStream<String> lines = messages.map( new Function<Tuple2<String, String>, String>() { @Override public String call(Tuple2<String, String> tuple2) { return tuple2._2(); } } ); JavaDStream<String> words = lines.flatMap( new FlatMapFunction<String, String>() { @Override public Iterable<String> call(String x) { return Lists.newArrayList( SPACE.split( x ) ); } } ); JavaPairDStream<String, Integer> wordCounts = words.mapToPair( new PairFunction<String, String, Integer>() { @Override public Tuple2<String, Integer> call(String s) { return new Tuple2<String, Integer>( s, 1 ); } } ).reduceByKey( new Function2<Integer, Integer, Integer>() { @Override public Integer call(Integer i1, Integer i2) { return i1 + i2; } } ); wordCounts.print(); // Start the computation jssc.start(); jssc.awaitTermination(); } } The jaas configuration file wrap principal and its keytab path, here I'm lazy directly use kafka service account. Should build user account for use in this. [root@cheny0 ~]# cat /usr/hdp/current/kafka-broker/config/kafka_client_jaas.confKafkaClient KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true doNotPrompt=true principal="kafka/cheny0.field.hortonworks.com@FIELD.HORTONWORKS.COM" keyTab="kafka.service.keytab" useTicketCache=true renewTicket=true serviceName="kafka"; }; Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true doNotPrompt=true principal="kafka/cheny0.field.hortonworks.com@FIELD.HORTONWORKS.COM" keyTab="kafka.service.keytab" useTicketCache=true renewTicket=true serviceName="kafka"; }; And here need say more on submission command. In the convenience of you to check log in spark history server, if in HDP 2.5 and prior version, you have to specify the server address in the command: //spark job for kerberized hdp 2.5 spark-submit --master=yarn --deploy-mode=cluster \ --files /usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf,/etc/security/keytabs/kafka.service.keytab \ --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf" \ --conf "spark.driver.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf" \ --conf "spark.yarn.historyServer.address=http://cheny0.field.hortonworks.com:18080" \ --conf "spark.eventLog.dir=hdfs:///spark-history" \ --conf "spark.eventLog.enabled=true" \ --jars /usr/hdp/current/spark-client/lib/spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar,/usr/hdp/current/kafka-broker/libs/kafka_2.10-0.10.1.2.6.1.0-129.jar,/usr/hdp/current/spark-client/lib/spark-examples-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar \ --class spark.example.JavaDirectKafkaWordCount kafka-producer-1.0-SNAPSHOT.jar There is no need such in HDP 2.6 //spark job for kerberized hdp 2.6 spark-submit --master=yarn --deploy-mode=cluster \ --files /usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf,/etc/security/keytabs/kafka.service.keytab \ --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf" \ --conf "spark.driver.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf" \ --jars /usr/hdp/current/spark-client/lib/spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar,/usr/hdp/current/kafka-broker/libs/kafka_2.10-0.10.1.2.6.1.0-129.jar,/usr/hdp/current/spark-client/lib/spark-examples-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar \ --class spark.example.JavaDirectKafkaWordCount kafka-producer-1.0-SNAPSHOT.jar For those of you who expected explanation on above code and command, can be referred to document link https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_spark-guide/content/spark-streaming-kafka-kerb.html Now finish background introduce, let's step back to issue topics, I will walk you through Zookeeper access right check Because Kafka need to read topics and offset information from zookeeper znode, check if the acl satisfies [root@cheny0 tmp]# cd /usr/hdp/current/zookeeper-client/bin [root@cheny0 bin]# ./zkCli.sh -server cheny0:2181 [zk: cheny0:2181(CONNECTED) 0] getAcl / 'world,'anyone : cdrwa [zk: cheny0:2181(CONNECTED) 2] getAcl /consumers 'world,'anyone : cdrwa [zk: cheny0:2181(CONNECTED) 4] getAcl /brokers 'world,'anyone : cdrwa HDFS access right check [root@cheny0 ~]# hdfs dfs -ls /user Found 10 items drwxrwx--- - ambari-qa hdfs0 2017-09-17 13:34 /user/ambari-qa drwxr-xr-x - hbase hdfs0 2017-08-20 00:08 /user/hbased rwxr-xr-x - hcathdfs0 2017-08-06 00:55 /user/hcat drwx------ - hdfshdfs0 2017-09-28 03:10 /user/hdfs drwxr-xr-x - hivehdfs0 2017-08-06 00:55 /user/hive drwxr-xr-x - kafka hdfs0 2017-09-28 12:23 /user/kafka drwxrwxr-x - oozie hdfs0 2017-08-06 00:56 /user/oozie drwxr-xr-x - hdfshdfs0 2017-08-13 11:47 /user/root drwxrwxr-x - spark hdfs0 2017-09-08 16:54 /user/spark drwxr-xr-x - yarnhdfs0 2017-09-16 22:17 /user/yarn Note by default there did not create path "/user/kafka", so you need do it by hand and give right access [root@cheny0 ~]# hdfs dfs -mkdir /user/kafka [root@cheny0 ~]# hdfs dfs -chown kafka:hdfs /user/kafka Jaas configuration file and Keytab file access right Sparkstreaming job here is submitted in yarn cluster, which then plays as yarn user to read jaas and keytab files, should check the file acl. By default keytab file only its principal user to read, other users cannot access. You should grant other user to read those two files. [root@cheny0 ~]# ls -l /usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf -rw-r--r-- 1 kafka hadoop 560 Sep 26 13:35 /usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf [root@cheny0 ~]# ls -l /etc/security/keytabs/kafka.service.keytab -r--r--r-- 1 kafka hadoop 208 Aug6 01:53 /etc/security/keytabs/kafka.service.keytab Specify path of Jaas and Keytab files As emphasized in most Spark articles, Sparkstreaming driver wrap Jaas and Keytab files and sent to executor to parse. The very important step is specify the path on the driver machine. There is no need to hand-copy those files to all slave nodes. --files /usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf,/etc/security/keytabs/kafka.service.keytab \ --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf" \ --conf "spark.driver.extraJavaOptions=-Djava.security.auth.login.config=kafka_client_jaas.conf" \ If not sufficient paying attention to above points, you will likely come across error saying org.apache.spark.SparkException: Couldn't connect to leader for topic or javax.security.auth.login.LoginException:Unable to obtain password from user or org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException:Jaas configuration not found If all go well, we move to some points need to note when debugging Use Yarn to kill application The test code is non-stop because its of streaming process. Whenever submit and failed but still be running on yarn queue resource. If not to clear, successive submission will be block and queued. Better use yarn command to kill them. [root@cheny0 tmp]# yarn application -list -appStates RUNNING 17/09/30 22:22:08 INFO client.RMProxy: Connecting to ResourceManager at cheny1.field.hortonworks.com/172.26.197.243:8050 17/09/30 22:22:09 INFO client.AHSProxy: Connecting to Application History server at cheny1.field.hortonworks.com/172.26.197.243:10200 Total number of applications (application-types: [] and states: [RUNNING]):3 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1505572583831_0056spark.example.JavaDirectKafkaWordCount SPARK kafka default RUNNING UNDEFINED 10% http://172.26.197.246:37394 application_1505572583831_0057spark.example.JavaDirectKafkaWordCount SPARK kafka default RUNNING UNDEFINED 10% http://172.26.197.246:35247 application_1505572583831_0058spark.example.JavaDirectKafkaWordCount SPARK kafka default RUNNING UNDEFINED 10% http://172.26.197.246:42456 [root@cheny0 tmp]# yarn application -kill application_1505572583831_0056 17/09/30 22:22:33 INFO client.RMProxy: Connecting to ResourceManager at cheny1.field.hortonworks.com/172.26.197.243:8050 17/09/30 22:22:34 INFO client.AHSProxy: Connecting to Application History server at cheny1.field.hortonworks.com/172.26.197.243:10200 Killing application application_1505572583831_0056 17/09/30 22:22:34 INFO impl.YarnClientImpl: Killed application application_1505572583831_0056 Use Yarn resource manager and History server webpage to check log Yarn resource manager helps you dive into containers' logs since spark execute tasks distributed across contained executors. While History server enable you to check spark DAG and full stack printed on each executor, and the more is collecting metrics help to do problem analysis. Maven: specify HDP repository and dependent jar for compiling purpose Should align with HDP release package and have jar files searched from HDP official repository <repositories> <repository> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>true</enabled> </snapshots> <id>hortonworks.extrepo</id> <name>Hortonworks HDP</name> <url>http://repo.hortonworks.com/content/repositories/releases</url> </repository> <repository> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>true</enabled> </snapshots> <id>hortonworks.other</id> <name>Hortonworks Other Dependencies</name> <url>http://repo.hortonworks.com/content/groups/public</url> </repository> </repositories><br> For this test code, we have following dependent package <dependencies> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>0.11.0.0</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.7.3</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>2.7.3</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming-kafka_2.10</artifactId> <version>1.6.3.2.6.1.0-129</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_2.10</artifactId> <version>1.6.3.2.6.1.0-129</version> <scope>provided</scope> </dependency> </dependencies><br> Once compiled the target jar, only pick non-dependencies to execute, and specify path of dependent jars in command. --jars /usr/hdp/current/spark-client/lib/spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar,/usr/hdp/current/kafka-broker/libs/kafka_2.10-0.10.1.2.6.1.0-129.jar,/usr/hdp/current/spark-client/lib/spark-examples-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar \ --class spark.example.JavaDirectKafkaWordCount kafka-producer-1.0-SNAPSHOT.jar Use Kafka tool to test After starting the above streaming process, meanwhile initiate an producer using [root@cheny0 bin]# ./kafka-console-producer.sh --broker-list cheny0:6667 --topic chen --security-protocol PLAINTEXTSASL Other Notable Pits for Sparkstreaming Connecting Kafka in Kerberized HDP 2.6 Here typed the title of this article as input. And checked output log generated from SparkStreaming process, either from yarn or spark history server. ------------------------------------------- Time: 1506791000000 ms ------------------------------------------- ------------------------------------------- Time: 1506791200000 ms ------------------------------------------- (Kafka,1) (2.6,1) (HDP,1) (Other,1) (,1) (Connecting,1) (Kerberized,1) (Notable,1) (Issue,1) (in,1) ... ------------------------------------------- Time: 1506791400000 ms ------------------------------------------- When you witness the above result, that means success through. Last but not least, for debugging spark stream accessing kafka in kerberized environment, you may encounter many issues from all aspect unexpected. If you are not much sure about your code, better to split into parts and verify one by one, do not scramble it and make you lost. That is my summary from experienced tortures. Hope you guys will do smoother than me. Stay awesome! ... View more

Hadoop release itself wraps typical testing tools, and there are many other popular benchmarks for testing Apache Hadoop. When it comes to HDP, some of the testing tool needs to be configured before can be used, and more need to revise the coding for it can work on HDP, for example in order have more metrics supported or in case of kerberos environment. This article introduce configuration or modification for testing tools working in HDP. Let's start by going through typical service components form HDP, now I'm using HDP 2.6.1.0 with FreeIPA kerberized. HBase - YCSB we begin with HBase because of more coding revise effort in bechmarking tools. Besides using HBase in-build evaluation tools org.apache.hadoop.hbase.PerformanceEvaluation to evaluation throughput and read/write performance, more popular tool is YCSB, which you download from git://github.com/brianfrankcooper/YCSB.git git clone git://github.com/brianfrankcooper/YCSB.git before compiling we need to do following code modification before it runs in kerberos environment. Here I just quick hard-code the necessary configuration for it getting credential. Maybe you can put all those info as parameters following command 'java' or './ycsb' which exactly executes the benchmarking. However, for saving time on error-prone command typing, hard-code can help us on focusing tuning benchmarking parameters instead of those credential configurations. //modify source code com.yahoo.ycsb.db.HBaseClient10.java //add config.set("hadoop.security.authentication", "Kerberos"); config.set("hbase.zookeeper.quorum", "cheny0.field.hortonworks.com,cheny2.field.hortonworks.com,cheny1.field.hortonworks.com"); config.set("zookeeper.znode.parent", "/hbase-secure"); //add getProperties().setProperty("principal", "hbase/cheny0.field.hortonworks.com@FIELD.HORTONWORKS.COM"); getProperties().setProperty("keytab", "/etc/security/keytabs/hbase.service.keytab"); if ((getProperties().getProperty("principal")!=null) && (getProperties().getProperty("keytab")!=null) && "kerberos".equalsIgnoreCase(config.get("hadoop.security.authentication"))) { try { UserGroupInformation.setConfiguration(config); UserGroupInformation userGroupInformation = UserGroupInformation.loginUserFromKeytabAndReturnUGI( getProperties().getProperty("principal"), getProperties().getProperty("keytab")); UserGroupInformation.setLoginUser(userGroupInformation); } catch (IOException e) { System.err.println("Keytab file is not readable or not found"); throw new DBException(e); } } And then remember aligned the dependent package with the same version it actually used in HDP. Be noted that pom.xml in sub-project 'hbase12', it references variable 'hbase12.version' <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-shaded-client</artifactId> <version>${hbase12.version}</version></dependency> And then check the root pom, and modify the version to align with your HBase from HDP 2.6.1.0 <hbase12.version>1.1.2</hbase12.version> Finally compiling to obtain executable jar file, and then launch it. sh-3.2# cd /Users/chen.yang/workspace/YCSB sh-3.2# mvn -DskipTests package sh-3.2# ssh cheny0 [root@cheny0 ~]# cd /tmp [root@cheny0 tmp]# tar zxvf ycsb-hbase12-binding-0.13.0-SNAPSHOT.tar.gz [root@cheny0 tmp]# mv ycsb-hbase12-binding-0.13.0-SNAPSHOT /ycsb //build conf dir [root@cheny0 tmp]# mkdir -p /ycsb/hbase12-binding/conf [root@cheny0 tmp]# scp /etc/hbase/conf/* ./ //execute load [root@cheny0 tmp]# cd /ycsb/bin [root@cheny0 tmp]# ./ycsb load hbase12 -P /ycsb/workloads/workloada -p columnfamily=f1 -p table=ycsb //internally it translate into command [root@cheny0 tmp]# java -Djavax.security.auth.useSubjectCredsOnly=false -cp "/etc/hbase/conf:/ycsb/lib/*" com.yahoo.ycsb.Client -db com.yahoo.ycsb.db.hbase12.HBaseClient12 -P /ycsb/workloads/workloada -p columnfamily=f1 -p table=ycsb -load HBase - YRegionsStats.rb This is not powerful testing tool, but its useful for help you arrange out data distribution status across regions, it help you get insight of data skew which intensively affect your Hbase application and services performance. Its a ruby script that can be downloaded form https://gist.github.com/nihed/f9ade8e6e8da7134aba4 The pity is it fails to runs at very first time with following error hints [root@cheny0 tmp]# hbase org.jruby.Main RegionsStats.rb NameError: cannot load Java class org.apache.hadoop.hbase.HServerLoadget_proxy_or_package_under_package at org/jruby/javasupport/JavaUtilities.java:54method_missing at file:/usr/hdp/2.6.1.0-129/hbase/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/java.rb:51(root) at RegionsStats.rb:15 The failure is because it try to load a deprecated class. So after deleting the 15th line, and it can work //use ruby script to check balance [root@cheny0 keytabs]# kinit -kt hbase.headless.keytab hbase-cheny@FIELD.HORTONWORKS.COM [root@cheny0 keytabs]# cd /tmp [root@cheny0 tmp]# hbase org.jruby.Main RegionsStats.rb 2017-08-24 00:49:18,816 INFO [main] Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS ******************************************************************* Hostname | regions count | regions size cheny1.field.hortonworks.com | 3 | 691 cheny2.field.hortonworks.com | 3 | 323 cheny0.field.hortonworks.com | 3 | 325 ******************************************************************* 0 | hbase:acl,,1501955861021.8b9119de437d98b0e5ffdf2475d1493b. 0 | hbase:meta,,1 0 | hbase:namespace,,1501952068264.30b243b85880dd40396d7940d6813605. 161 | TestTable,000000000000000000005244,1503383284564.22f23efda0244448340e62d2d457ae3b. 162 | TestTable,00000000000000000000654925,1503383284564.d71893801deac51ee626a5900320f2de. 162 | TestTable,00000000000000000000917303,1503383284403.912d9ab2d345097f51cf2c1a6e29b7b9. 163 | TestTable,0000000000000000000078582,1503383284403.0d5b67dfc510b9dd4e940116cc5509fa. 345 | TestTable,,1503159051542.63000b3e289f8d32ee37ce6632e3e0ef. 346 | TestTable,00000000000000000000261843,1503159051542.f34eea103da422ab90607a06eb4d350f. Kafka - kafka-producer-perf-test.sh Besides tuning disk I/O performance of Kafka broker, another pretty big influence on performance coms from producer and subscribe implementation. Here we only talk about producer, since for each partition it keeps maintaining a queue for accumulate message in order to save rpc with broker by sending in batch. With this the threshold for message sent out is highly affect the I/O performance. I recommend using patch from https://issues.apache.org/jira/browse/KAFKA-3554 Down the project and do compiling to get 'kafka-tools-1.0.0-SNAPSHOT.jar', then replace its counterpart in folder '/usr/hdp/current/kafka-broker/libs'. Before executing, do the following modification in kafka-producer-perf-test.sh replace exec $(dirname $0)/kafka-run-class.sh kafka.tools.ProducerPerformance by exec $(dirname $0)/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance execute the benchmark //for security testing [kafka@cheny0 bin]$ kinit -kt /etc/security/keytabs/kafka.service.keytab kafka/cheny0.field.hortonworks.com@FIELD.HORTONWORKS.COM [root@cheny0 bin]$ ./kafka-producer-perf-test.sh --num-records 1000000 --record-size 1000 --topic test_topic --throughput 100000 --num-threads 2 --value-bound 50000 --print-metrics --producer-props bootstrap.servers=cheny0.field.hortonworks.com:6667 compression.type=gzip max.in.flight.requests.per.connection=1 linger.ms=5 security.protocol=SASL_PLAINTEXT Its a really good tool for you to emulate and get sufficient metrics for studying the health. And there accompanies a bunch of equation to judge being healthy or not. For details, can be referred to youtube video https://www.youtube.com/watch?v=oQe7PpDDdzA and the slides in https://www.slideshare.net/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600?qid=ebe42992-c4bb-4337-83dc-c5492043aa34&v=&b=&from_search=2 And believe me, the beauty lies in small modulate can raise big difference, try to use it and get more appreciation. HDFS and MapReduce Here I just wanna refer you to http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/ where has sufficient explain and tutorials. But one more thing should be alerted, the default testing scale is based on 1 terabytes, so if your cluster nodes has limited disk capacity, its strongly require you first considering decrease the value setting, and then do the benchmark. Or, it will ruin your good mood. Okay guys, I know this article is not new to you, here I just do the collection of my memory for sharing you guys to better carry out those benchmark. Since with confidence in your tuned cluster through checking health is only by means of benchmarking and learn to parse the results. The above is what I'v done during several projects I ever did. And if later can I learn more, I will do more complementary. Hope you also do it well ! ... View more