Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

OOZIE - credentials in job.properties are not used by hive action?

OOZIE - credentials in job.properties are not used by hive action?

Contributor

Dear oozie experts, I've been working a lot with oozie the last month and am pretty confident I created a proper workflow with lots of parameterizations. I am now at a stage where I'm removing redundant text (changing fullpaths to relative ones (where allowed) etc).

One thing I can't get my head around though is the requirement for a hive action to have the credentials part in the top part of your workflow.xml. As these credentials are already part of the job.properties files it doesn't seem logic to duplicate those properties in the workflow.xml. anyone knows why this is required?

Is this just a limitation of the hive-action schema?

7 REPLIES 7

Re: OOZIE - credentials in job.properties are not used by hive action?

? Not sure what you mean with "as these credentials are already part of the job.properties file"

The Credentials tag is an XML tag that needs to be there in your workflow.xml for oozie to do the kerberos ticket initialization. He then sends uses these tickets for all actions that reference them. The job.properties is only there to parametrize tags. It cannot replace tags in the workflow.xml

Re: OOZIE - credentials in job.properties are not used by hive action?

Contributor

As Ben said there is no functionality from items within the job.properties, without referencing the tags within the workflow they will never be used. You'd want to store these tags in the job.properties file if they change often(such as across different environments) but you still want to use the same workflow, saves you having to go in and edit the workflow each time. Plus it looks a bit nicer :)

Re: OOZIE - credentials in job.properties are not used by hive action?

Contributor

I think I get what you are saying but how do I reference a variable like credentials in the job.properties for say the hive action? The tags in the job.properties files are stored like this (at least when created by the hue oozie editor)

credentials={u'hcat': {'xml_name': u'hcat', 'properties': [('hcat.metastore.uri', u'thrift://[SERVER]:[PORT]'), ('hcat.metastore.principal', u'hive/[SERVER]')]}, u'hive2': {'xml_name': u'hive2', 'properties': [('hive2.jdbc.url', 'jdbc:hive2://[SERVER]:10000/default'), ('hive2.server.principal', 'hive/[SERVER]')]}, None: {'xml_name': None, 'properties': []}}

all other properties are stored like:

VARIABLE1=VALUE1
VARIABLE2=VALUE2

and are referenced by the ${VARIABLE_NAME} but the credentials are stored differently.

Re: OOZIE - credentials in job.properties are not used by hive action?

Contributor

Hi Rene,

I don't normally use hue so i'm not entirely sure how it creates the job properties. When I store the hive credentials in job.properties I like to do it like this:

hivePrincipalJdbcUrl=<yourHiveUrl>;
hivePrincipal=<yourPrincipal>

Then at the top of the Oozie workflow.xml declare these credentials

<credentials>
<credential name="hive2Cred" type="hive2">
<property>
<name>hive2.jdbc.url</name>
<value>${hivePrincipalJdbcUrl}</value>
</property>
<property>
<name>hive2.server.principal</name>
<value>${hivePrincipal}</value>
</property>
</credential>
</credentials>

Then finally reference these credentials in your Hive Action:

<action name="exampleHive" cred="hive2Cred">
<hive2 xmlns="uri:oozie:hive2-action:0.1">
<job-tracker></job-tracker>
<name-node></name-node>
<jdbc-url></jdbc-url>
</hive2>
<ok to="end" />
<error to="fail" />
</action>

Re: OOZIE - credentials in job.properties are not used by hive action?

Contributor

Thanks for your detailed explanation Daniel! I know how to use variables from my job.properties but somehow the credentials part is stored completely different by the hue editor. It looks like it's converted the credentials xml part into json and put that part in the job.properties...

Re: OOZIE - credentials in job.properties are not used by hive action?

@Rene Sluiter Hue is the devil for oozie development ( great for management and administration though ) The start is easy but afterwards it only makes problems. A good XML editor is the way to go I think :-)

Re: OOZIE - credentials in job.properties are not used by hive action?

Contributor

any thoughts about my comment @Daniel Perry or @Benjamin Leonhardi?

Don't have an account?
Coming from Hortonworks? Activate your account here