About KuldeepK

KuldeepK · ‎12-24-2015

Thanks @Artem Ervits - will try this

KuldeepK · ‎12-24-2015

Thanks @Ana Gillan

KuldeepK · ‎12-22-2015

Q1. Need more clarification on below variable substitution, Is it $username who submits the workflow ? HADOOP_USER_NAME=${wf:user()}

KuldeepK · ‎12-22-2015

I have a shell action defined in my workflow which moves data from one hdfs location to another, my action is failing because of below error Permission denied: user=xyz, access=EXECUTE, inode="/user/yarn/.staging":yarn:supergroup:drwx------ Below are some useful configurations (got these from application log) yarn.app.mapreduce.am.staging-dir=/user user.name=yarn mapreduce.job.user.name=xyz I also have added below variable in my workflow.xml file under shell action <env-var>HADOOP_USER_NAME=${wf:user()}</env-var> I'm not sure why user "xyz" is trying to read/write at /user/yarn/.staging location. Oozie experts, any idea whats going on here ?

KuldeepK · ‎12-21-2015

@Shigeru Takehara Please refer this - https://developer.ibm.com/hadoop/blog/2015/11/05/r...

KuldeepK · ‎12-21-2015

This article is an update of one on http://hadooped.blogspot.com/2013/10/apache-oozie-part-13-oozie-ssh-action_30.html, authored by @Anagha Khanolkar. Below are the steps to setup Oozie workflow using ssh-action: Step 1. Create job.properties. Example: #************************************************* # job.properties #************************************************* nameNode=hdfs://<namenode-machine-fqdn>:8020 jobTracker=<resource-manager-fqdn>:8050 queueName=default oozie.libpath=${nameNode}/user/oozie/share/lib oozie.use.system.libpath=true oozie.wf.rerun.failnodes=true oozieProjectRoot=${nameNode}/user/${user.name} appPath=${oozieProjectRoot} oozie.wf.application.path=${appPath} inputDir=${oozieProjectRoot} focusNodeLogin=<username>@<remote-host-where-you-have-your-shell-script(s)> shellScriptPath=~/uploadFile.sh emailToAddress=<email-id> Step 2. Write workflow.xml Example:    <workflow-app name="WorkFlowForSshAction" xmlns="uri:oozie:workflow:0.1"> <start to="sshAction"/> <action name="sshAction"> <ssh xmlns="uri:oozie:ssh-action:0.1"> <host>${focusNodeLogin}</host> <command>${shellScriptPath}</command> <capture-output/> </ssh> <ok to="sendEmail"/> <error to="killAction"/> </action> <action name="sendEmail"> <email xmlns="uri:oozie:email-action:0.1"> <to>${emailToAddress}</to> <subject>Output of workflow ${wf:id()}</subject> <body>Status of the file move: ${wf:actionData('sshAction')['STATUS']}</body> </email> <ok to="end"/> <error to="end"/> </action> <kill name="killAction"> <message>"Killed job due to error"</message> </kill> <end name="end"/> </workflow-app> Step 3. Write sample uploadFile.sh script Example: #!/bin/bash hadoop fs -put ~/test /user/<username>/uploadedbyoozie Step 4. Upload workflow.xml to ${appPath} defined in job.properties. Step 5. Login to Oozie host by "oozie" user. Step 6. Generate a key pair (if you don't have already ) using 'ssh-keygen' command Step 7. On Oozie host copy ~/.ssh/id_rsa.pub and paste it on <remote-host>'s ~/.ssh/authorized_keys file (focus node) Step 8. Test password-less ssh from oozie@oozie-host to <username>@<remote-host> Step 9. if step 7 succeeds then go ahead and run oozie job. it should complete without error Note - In order to get password-less ssh working please make sure that: 1. You have 700 permissions on ~/.ssh directory 2. 600 permissions on ~/.ssh/authorized_keys file on remote-host 3. 600 to ~/.ssh/id_rsa 4. 644 to ~/.ssh/id_rsa.pub

KuldeepK · ‎12-18-2015

@Rajesh Balamohan helped me to find answer of this question. As per code it always converts given database name in lowercase format Answer of this questions is - Database names in hive will always be in lowecase. https://svn.apache.org/repos/asf/hive/trunk/metast... @Override public void createDatabase(Database db) throws InvalidObjectException, MetaException { boolean commited = false; MDatabase mdb = new MDatabase(); mdb.setName(db.getName().toLowerCase()); mdb.setLocationUri(db.getLocationUri()); mdb.setDescription(db.getDescription()); mdb.setParameters(db.getParameters()); mdb.setOwnerName(db.getOwnerName()); PrincipalType ownerType = db.getOwnerType(); mdb.setOwnerType((null == ownerType ? PrincipalType.USER.name() : ownerType.name())); try { openTransaction(); pm.makePersistent(mdb); commited = commitTransaction(); } finally { if (!commited) { rollbackTransaction(); } } }

KuldeepK · ‎12-18-2015

Thanks @Neeraj Sabharwal

KuldeepK · ‎12-17-2015

@bdurai - Thats correct! Thank you.

KuldeepK · ‎12-17-2015

Goto terminal of your sandbox in oracle virtualBox --> Press <Alt+F5> --> enter username - root --> enter password - hadoop --> it will ask you to set new password --> set new password --> then try to login via putty using new credentials

Online	Offline
Last Visited	‎04-07-2022 05:11 PM

Member Since	‎04-03-2019 04:03 PM
Last Visited	‎04-07-2022 05:11 PM
Posts	962
Kudos received	1733

Cloudera Community

Re: oozie shell action

Re: Oozie Service Check fails after upgrading to ...

Re: oozie - mr container fails to start on rhel6 n...

Re: Not able to run docker container on yarn even ...

Re: Oozie Pig action doesn't appear in Tez UI

Re: Oozie shell action - Permission denied: user=x...

Re: Oozie shell action - Permission denied: user=x...

What is the value of ${wf:user()} and where it is ...

Oozie shell action - Permission denied: user=xyz, ...

Re: How to run Spark job from Oozie Workflow on HD...

Oozie ssh action

Re: Uppercase hive DB name

Re: Uppercase hive DB name

Re: Ranger authorization for HDFS - Unable to chan...

Re: How to SSH into HortonWorks Sandbox using Putt...