Note - I did receive few complaints about this script that it's failing on new Ambari versions, It happened because of change in kerberos payload. I have modified the script and it works like a charm again for new versions. Please continue to provide me the feedback! 🙂 Disclaimer - "Please do not use unless you know what u r doing! Test on Dev. cluster before using on production systems" 🙂 Automated Kerberos Installation and Configuration – For this post, I have written a shell script which uses Ambari APIs to configure Kerberos on HDP Single or Multinode clusters. You just need to clone our github repository and modify property file according to your cluster environment, execute setup script and phew!! Within 5-10 minutes you should have your cluster completely secured by Kerberos! Cool isn’t it? Detailed Steps(Demo on HDP Sandbox 2.4): 1. Clone our github repository Ambari server in your HDP Cluster Note - This script will install and configure KDC on your Ambari Server. git clone https://github.com/crazyadmins/useful-scripts.git Sample Output: [root@sandbox ~]# git clone https://github.com/crazyadmins/useful-scripts.git Initialized empty Git repository in /root/useful-scripts/.git/ remote: Counting objects: 29, done. remote: Compressing objects: 100% (25/25), done. remote: Total 29 (delta 4), reused 25 (delta 3), pack-reused 0 Unpacking objects: 100% (29/29), done. 2. Goto useful-scripts/ambari directory [root@sandbox ~]# cd useful-scripts/ambari/ [root@sandbox ambari]# ls -lrt total 16 -rw-r--r-- 1 root root 5701 2016-04-23 20:33 setup_kerberos.sh -rw-r--r-- 1 root root 748 2016-04-23 20:33 README -rw-r--r-- 1 root root 366 2016-04-23 20:33 ambari.props [root@sandbox ambari]# 3. Copy setup_kerberos.sh and ambari.props to the host where you want to setup KDC Server 4. Edit and modify ambari.props file according to your cluster environment Note - In case of multinode cluster, Please don't forget to add comma separated list of hosts as a value of KERBEROS_CLIENTS variable. Sample output for my Sandbox [root@sandbox ambari]# cat ambari.props CLUSTER_NAME=Sandbox AMBARI_ADMIN_USER=admin AMBARI_ADMIN_PASSWORD=admin AMBARI_HOST=sandbox.hortonworks.com KDC_HOST=sandbox.hortonworks.com REALM=HWX.COM KERBEROS_CLIENTS=sandbox.hortonworks.com ##### Notes ##### #1. KERBEROS_CLIENTS - Comma separated list of Kerberos clients in case of multinode cluster #2. Admin princial is admin/admin and password is hadoop [root@sandbox ambari]# 5. Start installation by simply executing setup_kerberos.sh Notes: 1. Please run setup_kerberos.sh from KDC_HOST only, you don’t need to setup or configure KDC, this script will do everything for you. 2. If you are running script on Sandbox then please turn OFF maintenance mode for HDFS and turn ON maintenance mode for Zepplin Notebook before executing the script. sh setup_kerberos.sh Screenshots: 1. Before Script execution 2. Script execution is in progress 3. Script execution has finished 4. Voila!! Please comment if you have any feedback/questions/suggestions. Happy Hadooping!! ... View more

If you are getting below error while refreshing client configs via Ambari UI then here are the steps to fix it! 1. Remove KERBEROS_CLIENT for problematic hosts using below curl call curl -H "X-Requested-By:ambari"-u admin:admin -X DELETE "http://<AMBARI-SERVER>:8080/api/v1/clusters/<cluster-name>/hosts/<host-name>/host_components/KERBEROS_CLIENT" 2. Ensure from Ambari UI that KERBEROS_CLIENT got removed. 3. From Ambari UI --> goto the problematic host --> click on Add button --> select Kerberos Client 4. Installation should complete without any issue. ... View more

Here is an example of scheduling Oozie co-ordinator based on input data events. it starts Oozie workflow when input data is available. In this example coordinator will start at 2016-04-10, 6:00 GMT and will keep running till 2017-02-26, 23:25GMT (please note start and end time in xml file) start="2016-04-10T06:00Z" end="2017-02-26T23:25Z" timezone="GMT" Frequency is 1 day frequency="${coord:days(1)}" Below ETL function gives same value as start time which means coordinator will look for input data which has value same as start data in /user/root/output/YYYYMMDD format <instance>${coord:current(0)}</instance> Below are the working configuration files. coordinator.xml: <coordinator-app name="test" frequency="${coord:days(1)}" start="2016-04-10T06:00Z" end="2017-02-26T23:25Z" timezone="GMT" xmlns="uri:oozie:coordinator:0.2"> <datasets> <dataset name="inputdataset" frequency="${coord:days(1)}" initial-instance="2016-04-10T06:00Z" timezone="GMT"> <uri-template>${nameNode}/user/root/input/${YEAR}${MONTH}${DAY}</uri-template> <done-flag></done-flag> </dataset> <dataset name="outputdataset" frequency="${coord:days(1)}" initial-instance="2016-04-10T06:00Z" timezone="GMT"> <uri-template>${nameNode}/user/root/output/${YEAR}${MONTH}${DAY}</uri-template> <done-flag></done-flag> </dataset> </datasets> <input-events> <data-in name="inputevent" dataset="inputdataset"> <instance>${coord:current(0)}</instance> </data-in> </input-events> <output-events> <data-out name="outputevent" dataset="outputdataset"> <instance>${coord:current(0)}</instance> </data-out> </output-events> <action> <workflow> <app-path>${workflowAppUri}</app-path> <configuration> <property> <name>inputDir</name> <value>${coord:dataIn('inputevent')}</value> </property> <property> <name>outputDir</name> <value>${coord:dataOut('outputevent')}</value> </property> </configuration> </workflow> </action> </coordinator-app> workflow.xml <workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-wf"> <start to="shell-node"/> <action name="shell-node"> <shell xmlns="uri:oozie:shell-action:0.2"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <exec>${myscript}</exec> <argument>${inputDir}</argument> <argument>${outputDir}</argument> <file>${myscriptPath}</file> <capture-output/> </shell> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <kill name="fail-output"> <message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message> </kill> <end name="end"/> </workflow-app> job.properties nameNode=hdfs://sandbox.hortonworks.com:8020 start=2016-04-12T06:00Z end=2017-02-26T23:25Z jobTracker=sandbox.hortonworks.com:8050 queueName=default examplesRoot=examples oozie.coord.application.path=${nameNode}/user/root workflowAppUri=${oozie.coord.application.path} myscript=myscript.sh myscriptPath=${oozie.wf.application.path}/myscript.sh myscript.sh #!/bin/bash echo "I'm receiving input as $1" > /tmp/output echo "I can store my output at $2" >> /tmp/output How to schedule this? 1. Edit above files as per your environment. 2. Validate your workflow.xml and cordinator.xml files using below command #oozie validate workflow.xml #oozie validate cordinator.xml 3. Upload your script and these xml files to oozie.coord.application.path and workflowAppUri mentioned in the job.properties 4. Submit coordinator using below command. oozie job -oozie http://<oozie-server>:11000/oozie -config $local/path/job.properties -run Note - You will see that some coordinator actions are in WAITING state, that's because they are still waiting for input data to be available on hdfs If you check /var/log/oozie.log and grep for WAITING coordinator actions: 2016-04-14 05:54:05,850 INFO CoordActionInputCheckXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000038-160408193600784-oozie-oozi-C] ACTION[0000038-160408193600784-oozie-oozi-C@3] [0000038-160408193600784-oozie-oozi-C@3]::ActionInputCheck:: In checkListOfPaths: hdfs://sandbox.hortonworks.com:8020/user/root/input/20160412 is Missing. [..] 2016-04-14 05:54:15,601 INFO CoordActionInputCheckXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000038-160408193600784-oozie-oozi-C] ACTION[0000038-160408193600784-oozie-oozi-C@4] [0000038-160408193600784-oozie-oozi-C@4]::ActionInputCheck:: In checkListOfPaths: hdfs://sandbox.hortonworks.com:8020/user/root/input/20160413 is Missing. On HDFS: [root@sandbox coord]# hadoop fs -ls /user/root/input/ Found 3 items -rw-r--r-- 3 root hdfs 0 2016-04-13 13:16 /user/root/input/20160410 drwxr-xr-x - root hdfs 0 2016-04-13 13:07 /user/root/input/20160411 Output: [root@sandbox coord]# cat /tmp/output I'm receiving input as hdfs://sandbox.hortonworks.com:8020/user/root/input/20160411 I can store my output at hdfs://sandbox.hortonworks.com:8020/user/root/output/20160411 ... View more

What is the Role of Journal nodes in Namenode HA ? I know many of us are aware that Role of Journal nodes is to keep both the Namenodes in sync and avoid hdfs split brain scenario by allowing only Active NN to write into journals. Have you ever wonder how does it works? Here you go! Journal nodes are distributed system to store edits. Active Namenode as a client writes edits to journal nodes and commit only when its replicated to all the journal nodes in a distributed system. Standby NN need to read data from edits to be in sync with Active one. It can read from any of the replica stored on journal nodes. ZKFC will make sure that only one Namenode should be active at a time. However, when a failover occurs, it is still possible that the previous Active NameNode could serve read requests to clients, which may be out of date until that NameNode shuts down when trying to write to the JournalNodes. For this reason, we should configure fencing methods even when using the Quorum Journal Manager. How quorum journal manager work with fencing ? To work with fencing journal manager uses epoc numbers. Epoc numbers are integer which always gets increased and have unique value once assigned. Namenode generate epoc number using simple algorithm and uses it while sending RPC requests to the QJM. When you configure Namenode HA, the first Active Namenode will get epoc value 1. In case of failover or restart, epoc number will get increased. The Namenode with higher epoc number is considered as newer than any Namenode with earlier epoc number. Now both Namenode thinks that they are active and sends write request to quorum journal manager with their epoc number, how QJM handles this situation? Quorum journal manager stores epoc number locally which called as promised epoc. Whenever JournalNode receives RPC request along with epoc number from Namenode, it compares the epoch number with promised epoch. If request is coming from newer node which means epoc number is greater than promised epoc then itrecords new epoc number as promised epoc. If the request is coming from Namenode with older epoc number, then QJM simply rejects the request. When QJM rejects the requests from Namenode with older epoc value then you get below lines in the Namenode logs WARN client.QuorumJournalManager (IPCLoggerChannel.java:call(388)) - Remote journal <journal-node-hostname>:<port> failed to write txns 2397121201-2397121201. Will try to write to this JN again after the next log roll. org.apache.hadoop.ipc.RemoteException(java.io.IOException): IPC's epoch 112 is less than the last promised epoch 113 ... View more

Generally you will get permission denied error as below when you run oozie shell action without setting environment variable to username who is submitting the oozie workflow. Permission denied: user=yarn, access=WRITE oozie shell action Below are my sample configs workflow.xml <workflow-app xmlns="uri:oozie:workflow:0.3" name="shell-wf"> <start to="shell-node"/> <action name="shell-node"> <shell xmlns="uri:oozie:shell-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <exec>test.sh</exec> <file>/user/root/test.sh</file> </shell> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app> job.properties nameNode=hdfs://sandbox.hortonworks.com:8020 jobTracker=sandbox.hortonworks.com:8050 queueName=default examplesRoot=examples oozie.wf.application.path=${nameNode}/user/${user.name} test.sh shell script #!/bin/bash hadoop fs -mkdir /user/root/testdir Error: mkdir: Permission denied: user=yarn, access=WRITE, inode="/user/root/testdir":root:hdfs:drwxr-xr-x Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1] log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.impl.MetricsSystemImpl). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. How to fix this? To tell oozie not to run container as yarn but as the user who has submitted the workflow, we need to add below variable environment variable in our workflow.xml file <env-var>HADOOP_USER_NAME=${wf:user()}</env-var> Modified working workflow.xml file: <workflow-app xmlns="uri:oozie:workflow:0.3" name="shell-wf"> <start to="shell-node"/> <action name="shell-node"> <shell xmlns="uri:oozie:shell-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <exec>test.sh</exec> <env-var>HADOOP_USER_NAME=${wf:user()}</env-var> <file>/user/root/test.sh</file> </shell> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app> Ouptut: [root@sandbox ~]# hadoop fs -ls -d /user/root/testdir drwxr-xr-x - root hdfs 0 2016-04-08 07:59 /user/root/testdir [root@sandbox ~]# ... View more