This has been observed for Ambari 1.7. I know Ambari 1.7 is very old now however if some people are still using it and facing same issue then this post can save your lot of time! 🙂 . We know that Ambari 1.7 has Ganglia for reporting metrics. . Our issue was, we were unable to get service metrics for YARN service, below was the exact scenario: 1. In Ganglia UI, I was able to get graphs for the YARN Metrics. 2. In Ambari UI, I was able to see metrics for other services like HDFS etc. 3. This issue was observed on one of our customer's cluster. 4. I did setup same cluster on my local system however I was not able to reproduce this issue. 5. Only difference in customer's and my cluster was, customer was having RM HA and I had installed single node instance. 6. There was no error while getting metrics, please see below screenshot when metrics were not visible. . How to troubleshoot: 1. Click on any of the metrics, say 'Nodemanagers' --> This will open graph in big window. 2. Open the developer tools from Chrome/Firefox and inspect the network activities. 3. Notice the REST call from where its trying to fetch metrics. 4. Now do the RM failover and try to follow step 1-3 again. 5. Same REST call? No difference. 6. If you flip the RMs, Graphs will start populating the Data. . Root Cause: If you look at hadoop-metrics2.properties file, it has only one RM host(initial rm1) hardcoded for resourcemanager.sink.ganglia.servers resourcemanager.sink.ganglia.servers=rm1.crazyadmins.com:8664 . Workaround: Make same RM host as active RM which you get in REST API as a output of troubleshooting step 4. . Permanent fix: Edit /etc/hadoop/conf/hadoop-metrics2.properties file and add another RM host: E.g. resourcemanager.sink.ganglia.servers=rm1.crazyadmins.com:8664,rm2.crazyadmins.com:8664 Note - This file is not managed by Ambari 1.7. Feel free to modify on both the RM hosts and restart RMs via Ambari after the modifications. . Hope you enjoyed this article! Please comment if you have any questions. Happy Hadooping!! 🙂 ... View more

In previous post we have seen how to Automate HDP installation with Namenode HA using Ambari Blueprints. In this post, we will see how to Deploy single node HDP Cluster with Kerberos authentication via Ambari blueprint . Note - For Ambari 2.6.X onwards, we will have to register VDF to register internal repository, or else Ambari will pick up latest version of HDP and use the public repos. please see below document for more information. For Ambari version less than 2.6.X, this guide will work without any modifications. Document - https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-release-notes/content/ambari_relnotes-2.6.0.0-behavioral-changes.html . Below are simple steps to install HDP single node cluster with Kerberos Authentication(MIT KDC) using internal repository via Ambari Blueprints. . Step 1: Install Ambari server using steps mentioned under below link http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Installing_HDP_AMB/content/_download_the_ambari_repo_lnx6.html . Step 2: Register ambari-agent manually Install ambari-agent package on all the nodes in the cluster and modify hostname to ambari server host(fqdn) in /etc/ambari-agent/conf/ambari-agent.ini . Step 3: Install and configure MIT KDC Detailed Steps(Demo on HDP Sandbox 2.4): 3.1 Clone our github repository Ambari server in your HDP Cluster Note - This script will install and configure KDC on your Ambari Server. git clone https://github.com/crazyadmins/useful-scripts.git Sample Output: [root@sandbox ~]# git clone https://github.com/crazyadmins/useful-scripts.git Initialized empty Git repository in /root/useful-scripts/.git/ remote: Counting objects: 29, done. remote: Compressing objects: 100% (25/25), done. remote: Total 29 (delta 4), reused 25 (delta 3), pack-reused 0 Unpacking objects: 100% (29/29), done. 2. Goto useful-scripts/ambari directory [root@sandbox ~]# cd useful-scripts/ambari/ [root@sandbox ambari]# ls -lrt total 16 -rw-r--r-- 1 root root 5701 2016-04-23 20:33 setup_kerberos.sh -rw-r--r-- 1 root root 748 2016-04-23 20:33 README -rw-r--r-- 1 root root 366 2016-04-23 20:33 ambari.props [root@sandbox ambari]# 3. Copy setup_only_kdc.sh and ambari.props to the host where you want to setup KDC Server 4. Edit and modify ambari.props file according to your cluster environment Note - In case of multinode cluster, Please don't forget to add comma separated list of hosts as a value of KERBEROS_CLIENTS variable(Not applicable for this post). Sample output for my Sandbox [root@sandbox ambari]# cat ambari.props CLUSTER_NAME=Sandbox AMBARI_ADMIN_USER=admin AMBARI_ADMIN_PASSWORD=admin AMBARI_HOST=sandbox.hortonworks.com KDC_HOST=sandbox.hortonworks.com REALM=HWX.COM KERBEROS_CLIENTS=sandbox.hortonworks.com ##### Notes ##### #1. KERBEROS_CLIENTS - Comma separated list of Kerberos clients in case of multinode cluster #2. Admin princial is admin/admin and password is hadoop [root@sandbox ambari]# 5. Start installation by simply executing setup_only_kdc.sh Notes: 1. Please run setup_only_kdc.sh from KDC_HOST only, you don’t need to setup or configure KDC, this script will do everything for you. . Step 4: Configure blueprints Please follow below steps to create Blueprints . 4.1 Create hostmapping.json file as shown below: Note – This file will have information related to all the hosts which are part of your HDP cluster. { "blueprint" : "hdptest", "default_password" : "hadoop", "host_groups" :[ { "name" : "bluetest", "hosts" : [ { "fqdn" : "bluetest.openstacklocal" } ] } ], "credentials" : [ { "alias" : "kdc.admin.credential", "principal" : "admin/admin", "key" : "hadoop", "type" : "TEMPORARY" } ], "security" : { "type" : "KERBEROS" }, "Clusters" : {"cluster_name":"kerberosCluster"} } 4.2 Create cluster_configuration.json file, it contents mapping of hosts to HDP components { "configurations": [{ "kerberos-env": { "properties_attributes": {}, "properties": { "realm": "HWX.COM", "kdc_type": "mit-kdc", "kdc_host": "bluetest.openstacklocal", "admin_server_host": "bluetest.openstacklocal" } } }, { "krb5-conf": { "properties_attributes": {}, "properties": { "domains": "HWX.COM", "manage_krb5_conf": "true" } } }], "host_groups": [{ "name": "bluetest", "components": [{ "name": "NAMENODE" }, { "name": "NODEMANAGER" }, { "name": "DATANODE" }, { "name": "ZOOKEEPER_CLIENT" }, { "name": "HDFS_CLIENT" }, { "name": "YARN_CLIENT" }, { "name": "MAPREDUCE2_CLIENT" }, { "name": "ZOOKEEPER_SERVER" }, { "name": "SECONDARY_NAMENODE" }, { "name": "RESOURCEMANAGER" }, { "name": "APP_TIMELINE_SERVER" }, { "name": "HISTORYSERVER" }], "cardinality": 1 }], "Blueprints": { "blueprint_name": "hdptest", "stack_name": "HDP", "stack_version": "2.4", "security": { "type": "KERBEROS" } } } . Step 5: Create an internal repository map . 5.1: hdp repository – copy below contents, modify base_url to add hostname/ip-address of your internal repository server and save it in repo.json file. { "Repositories" : { "base_url" : "http://172.26.64.249/hdp/centos6/HDP-2.4.2.0/", "verify_base_url" : true } } . 5.2: hdp-utils repository – copy below contents, modify base_url to add hostname/ip-address of your internal repository server and save it in hdputils-repo.json file. { "Repositories" : { "base_url" : "http://172.26.64.249/hdp/centos6/HDP-UTILS-1.1.0.20/", "verify_base_url" : true } } . Step 6: Register blueprint with Ambari server by executing below command curl -H "X-Requested-By: ambari" -X POST -u admin:admin http://<ambari-server>:8080/api/v1/blueprints/multinode-hdp -d @cluster_config.json . Step 7: Setup Internal repo via REST API. Execute below curl calls to setup internal repositories. curl -H "X-Requested-By: ambari"-X PUT -u admin:admin http://<ambari-server-hostname>:8080/api/v1/stacks/HDP/versions/2.4/operating_systems/redhat6/repositories/HDP-2.4 -d @repo.json curl -H "X-Requested-By: ambari"-X PUT -u admin:admin http://<ambari-server-hostname>:8080/api/v1/stacks/HDP/versions/2.4/operating_systems/redhat6/repositories/HDP-UTILS-1.1.0.20 -d @hdputils-repo.json . Step 8: Pull the trigger! Below command will start cluster installation. curl -H "X-Requested-By: ambari" -X POST -u admin:admin http://<ambari-server-hostname>:8080/api/v1/clusters/multinode-hdp -d @hostmap.json . Please refer Next Part for Automated HDP installation using Ambari blueprint with Kerberos authentication for multi-node cluster. . Please feel free to comment if you need any further help on this. Happy Hadooping!! ... View more

After completing Ambari upgrade from Ambari 1.7 to Ambari 2.0.2, Ambari failed to start because of below error(s): [root@xyz ~]# service ambari-server start Using python /usr/bin/python2.6 Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start.................... ERROR: Exiting with exit code -1. REASON: Ambari Server java process died with exitcode 255. Check /var/log/ambari-server/ambari-server.out for more information. . If you check /var/log/ambari-server/ambari-server.out logs, you get below error(s): [root@xyz ~]# cat /var/log/ambari-server/ambari-server.out [EL Warning]: metadata: 2016-12-01 10:01:23.436--ServerSession(306824177)--The reference column name [resource_type_id] mapped on the element [field permissions] does not correspond to a valid id or basic field/column on the mapping reference. Will use referenced column name as provided. [EL Info]: 2016-12-01 10:01:24.412--ServerSession(306824177)--EclipseLink, version: Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd [EL Info]: connection: 2016-12-01 10:01:24.603--ServerSession(306824177)--file:/usr/lib/ambari-server/ambari-server-2.0.2.25.jar_ambari-server_url=jdbc:postgresql://localhost/ambari_user=ambari login successful . Note - Above error is complex and confusing. You should always check /var/log/ambari-server/ambari-server.log. In my case, I was getting below error in ambari-server.log org.apache.ambari.server.StackAccessException: Stack data, stackName=HDP, stackVersion=2.2, serviceName=SMARTSENSE at org.apache.ambari.server.api.services.AmbariMetaInfo.getService(AmbariMetaInfo.java:497) at org.apache.ambari.server.api.services.AmbariMetaInfo.getComponent(AmbariMetaInfo.java:265) at org.apache.ambari.server.controller.utilities.DatabaseChecker.checkDBConsistency(DatabaseChecker.java:96) at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:217) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:665) . Why this has happened? This is one of the Known issue - Sometimes stack component for SmartSense gets removed from Ambari DB after an upgrade. . How to resolve this? 1. Check what version of SmartSense you have installed on your cluster. 2. Download RPM from Customer Support portal 3. Run below command to re-add the SmartSense view and Stack to the Ambari DB. [root@xyz crazyadmins.com]# hst add-to-ambari Enter SmartSense distributable path: /home/kuldeepk/smartsense-hst-1.2.1.0-161.x86_64.rpm Added SmartSense service definition to Ambari 4. Restart 'ambari-server'. [root@xyz crazyadmins.com]# ambari-server restart Using python /usr/bin/python Restarting ambari-server Using python /usr/bin/python Stopping ambari-server Ambari Server is not running Using python /usr/bin/python Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start.................... Ambari Server 'start' completed successfully. [root@prodnode1 ~]# . Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! 🙂 ... View more

Here is the scenario: 1. I have workflow.xml which contains hive action. . 2. I have added <job-xml> tag inside hive action and provided path to the hive-site.xml (say /tmp/hive-site.xml) . 3. I have added hive-site.xml to the ${wf.application.path}/lib directory as well. . 4. I have hive-site.xml to the oozie sharelib under: /user/oozie/sharelib/lib_<timestamp>/oozie/hive-site.xml /user/oozie/sharelib/lib_<timestamp>/sqoop/hive-site.xml /user/oozie/sharelib/lib_<timestamp>/hive/hive-site.xml . 5. My simple hive workflow is failing with below error: Oozie Hive action configuration ================================================================= Using action configuration file /hadoop/data01/hadoop/yarn/local/usercache/root/appcache/application_1443111597609_2691/container_1443111597609_2691_01_000002/action.xml ------------------------ Setting env property for mapreduce.job.credentials.binary to: /hadoop/data01/hadoop/yarn/local/usercache/root/appcache/application_1443111597609_2691/container_1443111597609_2691_01_000002/container_tokens ------------------------ ------------------------ Setting env property for tez.credentials.path to: /hadoop/data01/hadoop/yarn/local/usercache/root/appcache/application_1443111597609_2691/container_1443111597609_2691_01_000002/container_tokens ------------------------ <<< Invocation of Main class completed <<< Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw exception, hive-site.xml (Permission denied) java.io.FileNotFoundException: hive-site.xml (Permission denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:221) at java.io.FileOutputStream.<init>(FileOutputStream.java:110) at org.apache.oozie.action.hadoop.HiveMain.setUpHiveSite(HiveMain.java:166) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:196) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Oozie Launcher failed, finishing Hadoop job gracefully . How to resolve? . As we know that having multiple jar files with different version in oozie sharelib can cause classnotfound exceptions, in a similar way having multiple copies of configuration file also can cause conflicts. In this case Oozie might be trying to override hive-site.xml in nodemanager's local filesystem (/hadoop/yarn/local/usercache/<username>/appcache/application_id/blah/blah with one of the copy taken from sharelib or ${wf.application.path}/lib or from the <job-xml>. To resolve such conflicts, We have to remove extra copies of hive-site.xml from all the above mentioned location. Oozie uses hive-site.xml from /etc/oozie/conf/action-conf/hive/hive-site.xml 🙂 . To repeat, This issue was resolved by removing hive-site.xml from below locations: 1. oozie sharelib (it was present at multiple locations in oozie sharelib) . 2. from ${wf.application.path}/lib/ directory. . 3. From workflow.xml (removed <job-xml> part) . By default Oozie takes this file from /etc/oozie/conf/action-conf/hive/hive-site.xml . With Oozie nothing is Easy 😉 Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! 🙂 ... View more

Sometimes, after disabling Kerberos successfully, you can get below error while accessing NN or RM UIs. . Because of this, all the alerts, Ambari metrics data can be broken because JMX data won't be available to Ambari Metrics or Alerts via HTTP. . Resolution: Modify below propery in core-site.xml via Ambari and restart required services. hadoop.http.authentication.simple.anonymous.allowed=true Note - Most of the times issue happens because of this property being set as false after disabling Kerberos . Please also compare below property and see if it is correct. hadoop.security.authentication=simple .Issue is simple but it takes lot of time to troubleshoot. Hence posting this article to reduce someone's time 🙂 Please comment if you face any issues or need any further help on this. Happy Hadooping!! 🙂 ... View more

Step1: Please allow below ports from your OS firewall for Ambari. https://ambari.apache.org/1.2.5/installing-hadoop-using-ambari/content/reference_chap2_7.html . Step2: Please go though the required component and allow below ports from your OS firewall for HDP. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_HDP_Reference_Guide/content/accumulo-ports.html . Step 3: In order to allow YARN jobs to run successfully, we need to add custom TCP port range to the YARN configuration. Login to Ambari UI --> Select Mapreduce2 --> Configs --> Customer mapred-site --> Add/Modify below property yarn.app.mapreduce.am.job.client.port-range=32000-65000 Notes: 1. 32000-65000 is the port range which will be used by Application Master to connect to Node Managers. . 2. You can increase the number of ports based on job volume. . How to add exception in Centos7 firewall? Example for Step 3. #firewall-cmd --permanent --zone=public --add-port=32000-65000/tcp #firewall-cmd --reload Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! ... View more

Below are the steps for Oozie database migration from Derby to Mysql. Step 1 - Have MySQL server installed and ready to be configured. . Step 2 - Stop Oozie service from Ambari UI. . Step 3 - On Ambari Server, run below command ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar Note - Please pass appropriate driver if /usr/share/java/mysql-connector-java.jar does not exists. . . Step 4 - Login to Mysql server as root user and create a blank 'oozie' database and grant required permissions to the 'oozie' user. #create database oozie; #grant all privileges on oozie.* to 'oozie’@‘<oozie-server>' identified by 'oozie’; Note - 'oozie' is your oozie database password, if you want you can change it in above command . . Step 5 - Add mysql database server details in Oozie configuration via Ambari UI. . Step 6 - Make sure that you have mysql connector jar under Oozie libext (/usr/hdp/<version>/oozie/libext/mysql-connector-java.jar), if not then copy it from available machine. . Step 7 - Prepare Oozie war file /usr/hdp/<version>/oozie/bin/oozie-setup.sh prepare-war Note - Run above command on oozie server as oozie user. . Step 8 - Prepare Oozie schema using below command (Run below command on Oozie host as oozie user) /usr/hdp/<version>/oozie/bin/oozie-setup.sh db create -run Please note - Above steps does not have information to migrate historical data from Derby to Mysql. Basically Oozie stores workflows/coordinator configuration related information as MEDIUMBLOB datatype and there is no straightforward way to convert these information in mysql compatible format. . If you take a sql dump from Derby and import directly into mysql, Oozie server will start, you will see all historical data however whenever Oozie will try to change the coordinator action state/workflow action state, you might see parsing issues like below(because of incorrect BLOBs in Mysql): 2016-08-22 06:48:19,036 WARN CoordMaterializeTransitionXCommand:523 - SERVER[oozienode2.openstacklocal] USER[root] GROUP[-] TOKEN[] APP[test] JOB[0000003-160822025623541-oozie-oozi-C] ACTION[-] Configuration parse error. read from DB :4f424a00000001000000010005636f6465630002677a1f8b0800000000000000c5945d4bc3301486ef05ffc32ebc6d9276acce520afe8121526fbc8be9a9ab4b73e269639de27f37eb9c30d6099d8237219fcffbe6cd47aad094d5a323d95668b2f3b3c924b58416a85df72ddf36b286ac5e378a2adbdec87699f2beeb6bf84java.io.IOException: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog. at org.apache.oozie.util.XConfiguration.parse(XConfiguration.java:289) at org.apache.oozie.util.XConfiguration.<init>(XConfiguration.java:80) at Happy Hadooping!! Please comment your feedback or questions in the comment section. ... View more