In previous post we have seen how to Automate HDP installation with Kerberos authencation on single node cluster using Ambari Blueprints. In this post, we will see how to Deploy multinode node HDP Cluster with Kerberos authentication via Ambari blueprint. . Note - For Ambari 2.6.X onwards, we will have to register VDF to register internal repository, or else Ambari will pick up latest version of HDP and use the public repos. please see below document for more information. For Ambari version less than 2.6.X, this guide will work without any modifications. Document - https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-release-notes/content/ambari_relnotes-2.6.0.0-behavioral-changes.html . Below are simple steps to install HDP multi node cluster with Kerberos Authentication(MIT KDC) using internal repository via Ambari Blueprints. . Step 1: Install Ambari server using steps mentioned under below link http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/ch_Installing_Ambari.html . Step 2: Register ambari-agent manually Install ambari-agent package on all the nodes in the cluster and modify hostname to ambari server host(fqdn) in /etc/ambari-agent/conf/ambari-agent.ini . Step 3: Install and configure MIT KDC Detailed Steps(Demo on HDP Sandbox 2.4): 3.1 Clone our github repository Ambari server in your HDP Cluster Note - This script will install and configure KDC on your Ambari Server. git clone https://github.com/crazyadmins/useful-scripts.git Sample Output: [root@sandbox ~]# git clone https://github.com/crazyadmins/useful-scripts.git Initialized empty Git repository in /root/useful-scripts/.git/ remote: Counting objects: 29, done. remote: Compressing objects: 100% (25/25), done. remote: Total 29 (delta 4), reused 25 (delta 3), pack-reused 0 Unpacking objects: 100% (29/29), done. 2. Goto useful-scripts/ambari directory [root@sandbox ~]# cd useful-scripts/ambari/ [root@sandbox ambari]# ls -lrt total 16 -rw-r--r-- 1 root root 5701 2016-04-23 20:33 setup_kerberos.sh -rw-r--r-- 1 root root 748 2016-04-23 20:33 README -rw-r--r-- 1 root root 366 2016-04-23 20:33 ambari.props [root@sandbox ambari]# 3. Copy setup_only_kdc.sh and ambari.props to the host where you want to setup KDC Server 4. Edit and modify ambari.props file according to your cluster environment Note - In case of multinode cluster, Please don't forget to add comma separated list of hosts as a value of KERBEROS_CLIENTS variable(Not applicable for this post). Sample output for my Sandbox [root@sandbox ambari]# cat ambari.props CLUSTER_NAME=Sandbox --> You can skip this variable AMBARI_ADMIN_USER=admin --> Not required AMBARI_ADMIN_PASSWORD=admin --> Note required AMBARI_HOST=sandbox.hortonworks.com --> Required KDC_HOST=sandbox.hortonworks.com --> Required REALM=HWX.COM --> Required KERBEROS_CLIENTS=sandbox.hortonworks.com --> Not required ##### Notes ##### #1. KERBEROS_CLIENTS - Comma separated list of Kerberos clients in case of multinode cluster #2. Admin princial is admin/admin and password is hadoop [root@sandbox ambari]# 5. Start installation by simply executing setup_only_kdc.sh Notes: 1. Please run setup_only_kdc.sh from KDC_HOST only, you don’t need to setup or configure KDC, this script will do everything for you. . Step 4: Configure blueprints Please follow below steps to create Blueprints . 4.1 Create hostmap.json(cluster creation template) file as shown below: Note – This file will have information related to all the hosts which are part of your HDP cluster. { "blueprint": "hdptest", "default_password": "hadoop", "host_groups": [{ "name": "kerbnode1", "hosts": [{ "fqdn": "kerbnode1.openstacklocal" }] }, { "name": "kerbnode2", "hosts": [{ "fqdn": "kerbnode2.openstacklocal" }] }, { "name": "kerbnode3", "hosts": [{ "fqdn": "kerbnode3.openstacklocal" }] }], "credentials": [{ "alias": "kdc.admin.credential", "principal": "admin/admin", "key": "hadoop", "type": "TEMPORARY" }], "security": { "type": "KERBEROS" }, "Clusters": { "cluster_name": "kerberosCluster" } } 4.2 Create cluster_config.json(blueprint) file, it contents mapping of hosts to HDP components { "configurations": [{ "kerberos-env": { "properties_attributes": {}, "properties": { "realm": "HWX.COM", "kdc_type": "mit-kdc", "kdc_host": "kerbnode1.openstacklocal", "admin_server_host": "kerbnode1.openstacklocal" } } }, { "krb5-conf": { "properties_attributes": {}, "properties": { "domains": "HWX.COM", "manage_krb5_conf": "true" } } }], "host_groups": [{ "name": "kerbnode1", "components": [{ "name": "NAMENODE" }, { "name": "NODEMANAGER" }, { "name": "DATANODE" }, { "name": "ZOOKEEPER_CLIENT" }, { "name": "HDFS_CLIENT" }, { "name": "YARN_CLIENT" }, { "name": "MAPREDUCE2_CLIENT" }, { "name": "ZOOKEEPER_SERVER" }], "cardinality": 1 }, { "name": "kerbnode2", "components": [{ "name": "SECONDARY_NAMENODE" }, { "name": "NODEMANAGER" }, { "name": "DATANODE" }, { "name": "ZOOKEEPER_CLIENT" }, { "name": "ZOOKEEPER_SERVER" }, { "name": "HDFS_CLIENT" }, { "name": "YARN_CLIENT" }, { "name": "MAPREDUCE2_CLIENT" }], "cardinality": 1 }, { "name": "kerbnode3", "components": [{ "name": "RESOURCEMANAGER" }, { "name": "APP_TIMELINE_SERVER" }, { "name": "HISTORYSERVER" }, { "name": "NODEMANAGER" }, { "name": "DATANODE" }, { "name": "ZOOKEEPER_CLIENT" }, { "name": "ZOOKEEPER_SERVER" }, { "name": "HDFS_CLIENT" }, { "name": "YARN_CLIENT" }, { "name": "MAPREDUCE2_CLIENT" }], "cardinality": 1 }], "Blueprints": { "blueprint_name": "hdptest", "stack_name": "HDP", "stack_version": "2.5", "security": { "type": "KERBEROS" } } } . Step 5: Create an internal repository map . 5.1: hdp repository – copy below contents, modify base_url to add hostname/ip-address of your internal repository server and save it in repo.json file. { "Repositories" : { "base_url" : "http://172.26.64.249/hdp/centos6/HDP-2.5.3.0/", "verify_base_url" : true } } . 5.2: hdp-utils repository – copy below contents, modify base_url to add hostname/ip-address of your internal repository server and save it in hdputils-repo.json file. { "Repositories" : { "base_url" : "http://172.26.64.249/hdp/centos6/HDP-UTILS-1.1.0.20/", "verify_base_url" : true } } . Step 6: Register blueprint with Ambari server by executing below command curl -H X-Requested-By: ambari -X POST -u admin:admin http://<ambari-server>:8080/api/v1/blueprints/hdptest -d @cluster_config.json Step 7: Setup Internal repo via REST API. Execute below curl calls to setup internal repositories. curl -H "X-Requested-By: ambari"-X PUT -u admin:admin http://<ambari-server-hostname>:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat6/repositories/HDP-2.5 -d @repo.json curl -H "X-Requested-By: ambari"-X PUT -u admin:admin http://<ambari-server-hostname>:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat6/repositories/HDP-UTILS-1.1.0.21 -d @hdputils-repo.json . Step 8: Pull the trigger! Below command will start cluster installation. curl -H "X-Requested-By: ambari" -X POST -u admin:admin http://<ambari-server-hostname>:8080/api/v1/clusters/hdptest -d @hostmap.json . You should see that Ambari has already marked Kerberos as enabled and started installing required services: . Please feel free to comment if you need any further help on this. Happy Hadooping!! ... View more

SYMPTOM While adding new service to the HDP cluster, it fails with below error: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/hook.py", line 37, in <module> AfterInstallHook().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/hook.py", line 31, in hook setup_hdp_symlinks() File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py", line 49, in setup_hdp_symlinks hdp_select.select_all(version) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/hdp_select.py", line 122, in select_all Execute(command, only_if = only_if_command) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh /usr/bin/hdp-select set all `ambari-python-wrap /usr/bin/hdp-select versions | grep ^2.4.2.0-258 | tail -1`' returned 1. Traceback (most recent call last): File "/usr/bin/hdp-select", line 382, in <module> printVersions() File "/usr/bin/hdp-select", line 239, in printVersions result[tuple(map(int, versionRegex.split(f)))] = f ValueError: invalid literal for int() with base 10: 'smartsense' ERROR: set command takes 2 parameters, instead of 1 usage: hdp-select [-h] [<command>] [<package>] [<version>] Set the selected version of HDP. positional arguments: <command> One of set, status, versions, or packages <package> the package name to set <version> the HDP version to set optional arguments: -h, --help show this help message and exit -r, --rpm-mode if true checks if there is symlink exists and creates the symlink if it doesn't Commands: set : set the package to a specified version status : show the version of the package versions : show the currently installed versions packages : show the individual package names . ROOT CAUSE There is an extra unwanted directory under /usr/hdp on Ambari server Please see below output: [root@prodnode1 hdp]# ls -lrt /usr/hdp/ total 16 drwxr-xr-x. 19 root root 4096 Sep 26 08:31 2.4.2.0-258 drwxr-xr-x. 2 root root 4096 Nov 21 13:28 current drwxr-xr-x. 2 root root 4096 Jan 2 15:37 smartsense drwxr-xr-x. 3 root root 4096 Jan 2 15:39 share [root@prodnode1 hdp]# Note - It's not recommended to put any directory under /usr/hdp/ except 'share', 'current', 'versioned directory' . RESOLUTION Move /usr/hdp/<unwanted-directory> to some other location than '/usr/hdp' See below output: #Before moving directory [root@prodnode1 hdp]# hdp-select versions Traceback (most recent call last): File "/usr/bin/hdp-select", line 382, in <module> printVersions() File "/usr/bin/hdp-select", line 239, in printVersions result[tuple(map(int, versionRegex.split(f)))] = f ValueError: invalid literal for int() with base 10: 'smartsense' [root@prodnode1 hdp]# #Move directory outside [root@prodnode1 hdp]# mv /usr/hdp/smartsense/ /root/ [root@prodnode1 hdp]# #After moving directory [root@prodnode1 hdp]# hdp-select versions 2.4.2.0-258 . Please comment if you have any question. Happy Hadooping!! 🙂 ... View more

SYMPTOM We get below error while installing new HDP version packages before upgrading to latest HDP version on SUSE linux. 2016-12-21 13:46:47,919 - Package Manager failed to install packages. Error: Execution of '/usr/bin/zypper --quiet install --auto-agree-with-licenses --no-confirm livy_2_3_2_0_2950' returned 104. File 'repomd.xml' from repository 'AMBARI-2.4.1.0.repo' is unsigned, continue? [yes/no] (no): no Error building the cache: [|] Valid metadata not found at specified URL(s) Warning: Disabling repository 'AMBARI-2.4.1.0.repo' because of the above error. File 'repomd.xml' from repository 'HDP.repo' is unsigned, continue? [yes/no] (no): no Error building the cache: [|] Valid metadata not found at specified URL(s) Warning: Disabling repository 'HDP.repo' because of the above error. No provider of 'livy_2_3_2_0_2950' found. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 376, in install_packages retry_count=agent_stack_retry_count File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 58, in action_upgrade self.upgrade_package(package_name, self.resource.use_repos, self.resource.skip_repos) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/zypper.py", line 62, in upgrade_package return self.install_package(name, use_repos, skip_repos, is_upgrade) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/zypper.py", line 57, in install_package self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput()) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 83, in checked_call_with_retries return self._call_with_retries(cmd, is_checked=True, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 91, in _call_with_retries code, out = func(cmd, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 71, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 93, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 141, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 294, in _call raise Fail(err_msg) Fail: Execution of '/usr/bin/zypper --quiet install --auto-agree-with-licenses --no-confirm livy_2_3_2_0_2950' returned 104. File 'repomd.xml' from repository 'AMBARI-2.4.1.0.repo' is unsigned, continue? [yes/no] (no): no Error building the cache: [|] Valid metadata not found at specified URL(s) Warning: Disabling repository 'AMBARI-2.4.1.0.repo' because of the above error. File 'repomd.xml' from repository 'HDP.repo' is unsigned, continue? [yes/no] (no): no Error building the cache: . ROOT CAUSE This is a BUG reported under https://issues.apache.org/jira/browse/AMBARI-19186 for SUSE linux if we are using unsigned repo. . WORKAROUND N/A . RESOLUTION Apply patch given at https://issues.apache.org/jira/browse/AMBARI-19186 Steps to Apply the patch: 1. Take a backup of /usr/lib/ambari-agent/lib/resource_management/libraries/functions/packages_analyzer.py 2. Edit /usr/lib/ambari-agent/lib/resource_management/libraries/functions/packages_analyzer.py with your favorite editor(I use vim) 3. Find the line with "--installed-only" E.g ["sudo", "zypper", "search", "--installed-only", "--details"], 4. Replace it with: ["sudo", "zypper", "--no-gpg-checks", "search", "--installed-only", "--details"], 5. Find the line with "--uninstalled-only" ["sudo", "zypper", "search", "--uninstalled-only", "--details"], 6. Replace it with: ["sudo", "zypper", "--no-gpg-checks", "search", "--uninstalled-only", "--details"], . Note - If the host where you are having this issue is a ambari-agent, you only need to apply patch on below file: /usr/lib/ambari-agent/lib/resource_management/libraries/functions/packages_analyzer.py If the host where you are having an issue is ambari-server, you need to apply patch on below files: /usr/lib/ambari-server/lib/resource_management/libraries/functions/packages_analyzer.py /usr/lib/ambari-agent/lib/resource_management/libraries/functions/packages_analyzer.py . Hope this information helps! Please comment if you have any questions. Happy Hadooping!! 🙂 ... View more

SYMPTOM Running java action via Oozie workflow fails with below error: Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, Could not find Yarn tags property (mapreduce.job.tags) java.lang.RuntimeException: Could not find Yarn tags property (mapreduce.job.tags) at org.apache.oozie.action.hadoop.LauncherMainHadoopUtils.getChildYarnJobs(LauncherMainHadoopUtils.java:52) at org.apache.oozie.action.hadoop.LauncherMainHadoopUtils.killChildYarnJobs(LauncherMainHadoopUtils.java:87) at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:44) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38) at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:36) . ROOT CAUSE Missing or Yarn related jar file conflict in Oozie sharelib. . RESOLUTION Complete the following steps as oozie user in the Oozie node: 1. Recreate Oozie sharelib using below command /usr/hdp/<hdp-version>/oozie/bin/oozie-setup.sh sharelib create -locallib /usr/hdp/<hdp-version>/oozie/oozie-sharelib.tar.gz -fs hdfs://<namenode-host>:8020 2. Update Oozie sharelib using below command oozie admin -oozie http://<oozie-host>:11000/oozie -sharelibupdate 3. Restart oozie service using Ambari and resubmit the workflow. . Note - If you have put any custom jars in Oozie sharelib, please make sure to copy them back again after re-creating Oozie sharelib. ... View more

SYMPTOM Beeline fails with below error: $ beeline --verbose Beeline version 0.14.0.2.2.6.0-2800 by Apache Hive beeline> !connect jdbc:hive2://prodnode1.crazyadmins.com:10000/default;principal=hive/prodnode1.crazyadmins.com@CRAZYADMINS.COM scan complete in 8ms Connecting to jdbc:hive2://prodnode1.crazyadmins.com:10000/default;principal=hive/prodnode1.crazyadmins.com@CRAZYADMINS.COM Enter username for jdbc:hive2://prodnode1.crazyadmins.com:10000/default;principal=hive/prodnode1.crazyadmins.com@CRAZYADMINS.COM: kuldeepk Enter password for jdbc:hive2://prodnode1.crazyadmins.com:10000/default;principal=hive/prodnode1.crazyadmins.com@CRAZYADMINS.COM: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.2.6.0-2800/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.2.6.0-2800/hive/lib/hive-jdbc-0.14.0.2.2.6.0-2800-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 16/02/02 00:35:55 [main]: ERROR transport.TSaslTransport: SASL negotiation failure javax.security.sasl.SaslException: No common protection layer between client and server at com.sun.security.sasl.gsskerb.GssKrb5Client.doFinalHandshake(GssKrb5Client.java:252) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:187) at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:507) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:264) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:190) at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:163) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:187) at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:138) at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:179) at org.apache.hive.beeline.Commands.connect(Commands.java:1078) at org.apache.hive.beeline.Commands.connect(Commands.java:999) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:45) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:936) at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:801) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:762) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:476) at org.apache.hive.beeline.BeeLine.main(BeeLine.java:459) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAc . ROOT CAUSE SSL was enabled on this cluster for Hiveserver2 --> Further customer disabled it however forgot to revert below property hive.server2.thrift.sasl.qop=auth-conf . WORKAROUND N/A . RESOLUTION Revert value of this property as below via Ambari and restart required services. hive.server2.thrift.sasl.qop=auth ... View more

In this post, we will see how to configure node labels on YARN. Before we go for the configuration part, let’s understand what is node label in YARN. Node labels allows us to divide our cluster in different parts and we can use those parts individually as per our requirements. More specifically, we can create a group of node-managers using node labels, for example group of node managers which are having high amount of RAM and use them to process only critical production jobs! This is cool, isn’t it? So lets see how we can configure node labels on YARN. . Types of node labels: Exclusive – In this type of node labels, only associated/mapped queues can access the resources of node label. Non Exclusive(sharable) – If resources are not in use for this node label then it can be shared with other running applications in a cluster. . Configuring node labels: . Step 1: Create required directory structure on HDFS Note – You can run below commands from any of the hdfs client. sudo su hdfs hadoop fs -mkdir -p /yarn/node-labels hadoop fs -chown -R yarn:yarn /yarn hadoop fs -chmod -R 700 /yarn . Step 2: Make sure that you have user directory for ‘yarn’ user on HDFS, if not then please create it using below commands Note – You can run below commands from any of the hdfs client. sudo su hdfs hadoop fs -mkdir -p /user/yarn hadoop fs -chown -R yarn:yarn /user/yarn hadoop fs -chmod -R 700 /user/yarn . Step 3: Configure below properties in yarn-site.xml via Ambari UI. If you don’t have Ambari UI, please add it manually to /etc/hadoop/conf/yarn-site.xml and restart required services. yarn.node-labels.enabled=true yarn.node-labels.fs-store.root-dir=hdfs://<namenode-host>:<namenode-rpc-port>/<complete-path_to_node_label_directory> Note – Please restart required services after above configuration changes! . Step 4: Create node labels using below commands sudo -u yarn yarn rmadmin -addToClusterNodeLabels "<node-label1>(exclusive=<true|false>),<node-label2>(exclusive=<true|false>)" For example, to add 2 node labels x and y: sudo -u yarn yarn rmadmin -addToClusterNodeLabels "x(exclusive=true),y(exclusive=false)" You can verify if node labels have been created by looking at Resource manager UI under ‘Node Lables’ option in the left pane or you can also run below command on any of the Yarn client yarn cluster --list-node-labels Sample output: [yarn@prodnode1 ~]$ yarn cluster --list-node-labels 16/12/14 15:45:56 INFO impl.TimelineClientImpl: Timeline service address: http://prodnode3.openstacklocal:8188/ws/v1/timeline/ 16/12/14 15:45:56 INFO client.RMProxy: Connecting to ResourceManager at prodnode3.openstacklocal/172.26.74.211:8050 Node Labels: <x:exclusivity=true>,<y:exclusivity=false> . Step 5: Allocate node labels to the node managers using below command: sudo -u yarn yarn rmadmin -replaceLabelsOnNode "<node-manager1>:<port>=<node-label1> <node-manager2>:<port>=<node-label2>" Example: sudo -u yarn yarn rmadmin -replaceLabelsOnNode "prodnode1.openstacklocal=x prodnode2.openstacklocal=y" Note – Don’t worry about port if you have only one node manager running per host. . Step 6: Map node labels to the queues: I have created 2 queues ‘a’ and ‘b’ in such a way that, queue ‘a’ can access nodes with label ‘x’ and ‘y’ where queue ‘b’ can only access the nodes with label ‘y’. By default, all the queues can access nodes with ‘default’ label. Below is my capacity scheduler configuration: yarn.scheduler.capacity.maximum-am-resource-percent=0.2 yarn.scheduler.capacity.maximum-applications=10000 yarn.scheduler.capacity.node-locality-delay=40 yarn.scheduler.capacity.queue-mappings-override.enable=false yarn.scheduler.capacity.root.a.a1.accessible-node-labels=x,y yarn.scheduler.capacity.root.a.a1.accessible-node-labels.x.capacity=30 yarn.scheduler.capacity.root.a.a1.accessible-node-labels.x.maximum-capacity=100 yarn.scheduler.capacity.root.a.a1.accessible-node-labels.y.capacity=50 yarn.scheduler.capacity.root.a.a1.accessible-node-labels.y.maximum-capacity=100 yarn.scheduler.capacity.root.a.a1.acl_administer_queue=* yarn.scheduler.capacity.root.a.a1.acl_submit_applications=* yarn.scheduler.capacity.root.a.a1.capacity=40 yarn.scheduler.capacity.root.a.a1.maximum-capacity=100 yarn.scheduler.capacity.root.a.a1.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.a.a1.ordering-policy=fifo yarn.scheduler.capacity.root.a.a1.state=RUNNING yarn.scheduler.capacity.root.a.a1.user-limit-factor=1 yarn.scheduler.capacity.root.a.a2.accessible-node-labels=x,y yarn.scheduler.capacity.root.a.a2.accessible-node-labels.x.capacity=70 yarn.scheduler.capacity.root.a.a2.accessible-node-labels.x.maximum-capacity=100 yarn.scheduler.capacity.root.a.a2.accessible-node-labels.y.capacity=50 yarn.scheduler.capacity.root.a.a2.accessible-node-labels.y.maximum-capacity=100 yarn.scheduler.capacity.root.a.a2.acl_administer_queue=* yarn.scheduler.capacity.root.a.a2.acl_submit_applications=* yarn.scheduler.capacity.root.a.a2.capacity=60 yarn.scheduler.capacity.root.a.a2.maximum-capacity=60 yarn.scheduler.capacity.root.a.a2.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.a.a2.ordering-policy=fifo yarn.scheduler.capacity.root.a.a2.state=RUNNING yarn.scheduler.capacity.root.a.a2.user-limit-factor=1 yarn.scheduler.capacity.root.a.accessible-node-labels=x,y yarn.scheduler.capacity.root.a.accessible-node-labels.x.capacity=100 yarn.scheduler.capacity.root.a.accessible-node-labels.x.maximum-capacity=100 yarn.scheduler.capacity.root.a.accessible-node-labels.y.capacity=50 yarn.scheduler.capacity.root.a.accessible-node-labels.y.maximum-capacity=100 yarn.scheduler.capacity.root.a.acl_administer_queue=* yarn.scheduler.capacity.root.a.acl_submit_applications=* yarn.scheduler.capacity.root.a.capacity=40 yarn.scheduler.capacity.root.a.maximum-capacity=40 yarn.scheduler.capacity.root.a.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.a.ordering-policy=fifo yarn.scheduler.capacity.root.a.queues=a1,a2 yarn.scheduler.capacity.root.a.state=RUNNING yarn.scheduler.capacity.root.a.user-limit-factor=1 yarn.scheduler.capacity.root.accessible-node-labels=x,y yarn.scheduler.capacity.root.accessible-node-labels.x.capacity=100 yarn.scheduler.capacity.root.accessible-node-labels.x.maximum-capacity=100 yarn.scheduler.capacity.root.accessible-node-labels.y.capacity=100 yarn.scheduler.capacity.root.accessible-node-labels.y.maximum-capacity=100 yarn.scheduler.capacity.root.acl_administer_queue=* yarn.scheduler.capacity.root.b.accessible-node-labels=y yarn.scheduler.capacity.root.b.accessible-node-labels.y.capacity=50 yarn.scheduler.capacity.root.b.accessible-node-labels.y.maximum-capacity=100 yarn.scheduler.capacity.root.b.acl_administer_queue=* yarn.scheduler.capacity.root.b.acl_submit_applications=* yarn.scheduler.capacity.root.b.b1.accessible-node-labels=y yarn.scheduler.capacity.root.b.b1.accessible-node-labels.y.capacity=100 yarn.scheduler.capacity.root.b.b1.accessible-node-labels.y.maximum-capacity=100 yarn.scheduler.capacity.root.b.b1.acl_administer_queue=* yarn.scheduler.capacity.root.b.b1.acl_submit_applications=* yarn.scheduler.capacity.root.b.b1.capacity=100 yarn.scheduler.capacity.root.b.b1.maximum-capacity=100 yarn.scheduler.capacity.root.b.b1.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.b.b1.ordering-policy=fifo yarn.scheduler.capacity.root.b.b1.state=RUNNING yarn.scheduler.capacity.root.b.b1.user-limit-factor=1 yarn.scheduler.capacity.root.b.capacity=60 yarn.scheduler.capacity.root.b.maximum-capacity=100 yarn.scheduler.capacity.root.b.minimum-user-limit-percent=100 yarn.scheduler.capacity.root.b.ordering-policy=fifo yarn.scheduler.capacity.root.b.queues=b1 yarn.scheduler.capacity.root.b.state=RUNNING yarn.scheduler.capacity.root.b.user-limit-factor=1 yarn.scheduler.capacity.root.capacity=100 yarn.scheduler.capacity.root.queues=a,b . Please visit http://crazyadmins.com/configure-node-labels-on-yarn/ for more details and FAQs. . Please comment if you need any further help on this. Happy Hadooping!! 🙂 ... View more

My Environment details: HDP Version - 2.4.2.0 Ambari - 2.2.2.0 Mysql Version - 5.1.34 . Hive shell was taking lot of time to load and when I checked Hive Metastore logs, I found below exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1 . Complete StackTrace: NestedThrowablesStackTrace: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.Util.getInstance(Util.java:386) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1052) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619) at com.mysql.jdbc.ConnectionImpl.unsetMaxRows(ConnectionImpl.java:5421) at com.mysql.jdbc.StatementImpl.realClose(StatementImpl.java:2441) at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3079) at com.mysql.jdbc.PreparedStatement.close(PreparedStatement.java:1156) at com.jolbox.bonecp.StatementHandle.close(StatementHandle.java:138) at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.close(ParamLoggingPreparedStatement.java:318) at org.datanucleus.store.rdbms.SQLController.closeStatement(SQLController.java:568) at org.datanucleus.store.rdbms.query.SQLQuery.performExecute(SQLQuery.java:357) at org.datanucleus.store.query.Query.executeQuery(Query.java:1786) at org.datanucleus.store.query.AbstractSQLQuery.executeWithArray(AbstractSQLQuery.java:339) at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:312) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1628) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:466) at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:393) at org.apache.hadoop.hive.metastore.ObjectStore$2.getSqlResult(ObjectStore.java:1738) at org.apache.hadoop.hive.metastore.ObjectStore$2.getSqlResult(ObjectStore.java:1734) at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2394) at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsInternal(ObjectStore.java:1734) at org.apache.hadoop.hive.metastore.ObjectStore.getPartitions(ObjectStore.java:1728) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114) at com.sun.proxy.$Proxy10.getPartitions(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.dropPartitionsAndGetLocations(HiveMetaStore.java:1700) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1539) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1744) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) at com.sun.proxy.$Proxy11.drop_table_with_environment_context(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.drop_table_with_environment_context(HiveMetaStoreClient.java:2062) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.drop_table_with_environment_context(SessionHiveMetaStoreClient.java:118) . Root cause: 1. MySql server version was patched(upgraded to 5.1.38) however mysql-java-connector was not upgraded. 2. Hive was configured to use Mysql as Metastore DB. 3. More details - http://bugs.mysql.com/bug.php?id=66659 . Resolution: After checking release notes on MySql community, found that this BUG has been fixed in mysql-java-connector-5.1.22 Please read more details here(Second last point) - https://dev.mysql.com/doc/relnotes/connector-j/5.1/en/news-5-1-22.html To fix this - I had to upgrade my mysql-java-connector from mysql-java-connector-5.1.17 to mysql-java-connector-5.1.22. Here is the download link for mysql-java-connector-5.1.22 . Where to place upgraded connector? On Ambari server(as root user): mkdir ~/backups mv /var/lib/ambari-server/resources/mysql* ~/backups/ cp $some_location/mysql-connector-java-5.1.22.jar /var/lib/ambari-server/resources/ ln -s /var/lib/ambari-server/resources/mysql-connector-java-5.1.22.jar /var/lib/ambari-server/resources/mysql-jdbc-driver.jar mv /usr/share/java/mysql* ~/backups/ cp /var/lib/ambari-server/resources/mysql-connector-java-5.1.22.jar /usr/share/java/ ln -s /usr/share/java/mysql-connector-java-5.1.22.jar /usr/share/java/mysql-connector-java.jar #Please run below command carefully (Please Don't copy and paste, or copy it carefully!) rm -rf /var/lib/ambari-agent/tmp/mysql-* . On HiveMetastore mkdir ~/backupsmv /usr/share/java/mysql* ~/backups/ cp $local_path/mysql-connector-java-5.1.22.jar /usr/share/java/ ln -s /usr/share/java/mysql-connector-java-5.1.22.jar /usr/share/java/mysql-connector-java.jar #Please run below command carefully (Please Don't copy and paste, or copy it carefully!) rm -rf /var/lib/ambari-agent/tmp/mysql-* . On Hiveserver2 mkdir ~/backupsmv /usr/share/java/mysql* ~/backups/ cp $local_path/mysql-connector-java-5.1.22.jar /usr/share/java/ ln -s /usr/share/java/mysql-connector-java-5.1.22.jar /usr/share/java/mysql-connector-java.jar #Please run below command carefully (Please Don't copy and paste, or copy it carefully!) rm -rf /var/lib/ambari-agent/tmp/mysql-* . Restart HiveMestastore and Hiveserver2 via Ambari and this should fix your issue! 🙂 . Please comment if you have any quesiton or feedback. Happy Hadooping!! 🙂 ... View more