Reply
Champion Alumni
Posts: 196
Registered: ‎11-18-2014
Accepted Solution

oozie- attach the job logs to email (email action)

Hello,

 

Is there a way to attach or add in the content of the email (email action in oozie) the job logs? (the logs from all the actions of the job)

I didn't find any wf parameter that could help ..

 

thank you!

GHERMAN Alina
Explorer
Posts: 21
Registered: ‎09-09-2015

Re: oozie- attach the job logs to email (email action)

Oozie Rest APIs to the resucue (Which Hue too uses)

 

Else

 

one can write a shell or .ksh which captures external_Id and its related launcher attemp ids, and get the logs of individual attemp ids.

 

We did the later one.

Champion Alumni
Posts: 196
Registered: ‎11-18-2014

Re: oozie- attach the job logs to email (email action)

[ Edited ]

Thank you for your answer. I will try to do a schell script for gathering the logs. 

However, is there a way to add the logs in the attachement?

 

 

Thank you!

 

Alina

GHERMAN Alina
Explorer
Posts: 21
Registered: ‎09-09-2015

Re: oozie- attach the job logs to email (email action)

email action has <attachment> element which can be used to attach an hdfs file or a comma sperated hdfs files.

 

However, make sure the logs are added/put to an hdfs file/files before attaching thereof.

Champion Alumni
Posts: 196
Registered: ‎11-18-2014

Re: oozie- attach the job logs to email (email action)

Hello,

 

I saw that there is an attachement element, however, I cannot add it in hue... 

 

 

Thank you!

 

GHERMAN Alina
Explorer
Posts: 21
Registered: ‎09-09-2015

Re: oozie- attach the job logs to email (email action)

[ Edited ]

Could be related to CDH version/environment as mentioned here. and its related jira.

 

However, if possible take a chance to edit manually your workflow for the email action to include <attachment>.. element and check whether attachment functionality works fine for your CDH environment.

 

Best regards

Champion Alumni
Posts: 196
Registered: ‎11-18-2014

Re: oozie- attach the job logs to email (email action)

You were right, this is linked to :

  • OOZIE-2160 - Oozie email action now supports attachments with an <attachment> element

 

I am in CDH 5.3

 

 

Thank you!

GHERMAN Alina
Champion Alumni
Posts: 196
Registered: ‎11-18-2014

Re: oozie- attach the job logs to email (email action)

Hello,

 

I told myself that if I cannot add an attachement, I'll just add a link to my log files from HDFS.

So I did a :

 

yarn application -list -appStates FINISHED |grep 'my_workflow_name' |grep -Po 'application_\d+_\d+' | sed 's/.*application://' | tail -n 1

in order to find the application id ($my_application_id) that I needed. Afterwards I wanted to do a :

 

 

 

yarn logs -applicationId $my_application_id

However, this doesn't return any logs if it is not executed with a user that has the rights to read the logs. So I wanted to change it into:

sudo -u hdfs yarn logs -applicationId $application_id

but then I got the ERROR:

 

 sudo: sorry, you must have a tty to run sudo

 

 

Is there a proper way to find out the logs without changing the level of securiyt security?

(http://unix.stackexchange.com/questions/122616/why-do-i-need-a-tty-to-run-sudo-if-i-can-sudo-without...

 

 

Thank you!

 

 

GHERMAN Alina
Posts: 1,894
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: oozie- attach the job logs to email (email action)

On non-kerberized (insecure) clusters, you can do the below:

export HADOOP_USER_NAME=username
[command]

For ex:

export HADOOP_USER_NAME=hdfs
yarn logs -applicationId $application_id
Champion Alumni
Posts: 196
Registered: ‎11-18-2014

Re: oozie- attach the job logs to email (email action)

Hello,

 

I tried to do :

 

export HADOOP_USER_NAME=my_user
load_events=`yarn logs -applicationId $application_id` 

I also tried:

export HADOOP_USER_NAME=hdfs
load_events=`yarn logs -applicationId $application_id` 

 

However I get :

 

 

Logs not available at /tmp/logs/hdfs/logs/application_1449728267224_0138 Log aggregation has not completed or is not enabled.

And this is the message that I get when I do this command with un unauthorised user..

 

 

 

GHERMAN Alina
Announcements