Reply
Explorer
Posts: 21
Registered: ‎04-15-2014

Oozie launcher never ends

Hello,

I'm doing a simple oozie's workflow with sqqop's action.
this is the xml's code

 

<workflow-app name="Sqoop" xmlns="uri:oozie:workflow:0.4">
    <start to="Sqoop"/>
    <action name="Sqoop">
        <sqoop xmlns="uri:oozie:sqoop-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <command>import --connect jdbc:mysql://10.0.0.221/db --username cloudera --password cloudera --table test -m 1 --target-dir /user/albert</command>
        </sqoop>
        <ok to="end"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

When i launch my job with hue or line-command, starts the oozie-launcher.
My oozie laucher, never finish, it is always stay at 95%.
If i kill the job oozie-launcher, starts the oozie-sqoop and it do his work very good.
But i don't understand why the oozie-launcher doesnt' work well!

 

This is my output

 

>>> Invoking Sqoop command line now >>>

12280 [main] WARN  org.apache.sqoop.tool.SqoopTool  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
12510 [main] INFO  org.apache.sqoop.Sqoop  - Running Sqoop version: 1.4.4-cdh5.0.1
12640 [main] WARN  org.apache.sqoop.tool.BaseSqoopTool  - Setting your password on the command-line is insecure. Consider using -P instead.
12748 [main] WARN  org.apache.sqoop.ConnFactory  - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
13131 [main] INFO  org.apache.sqoop.manager.MySQLManager  - Preparing to use a MySQL streaming resultset.
13131 [main] INFO  org.apache.sqoop.tool.CodeGenTool  - Beginning code generation
15107 [main] INFO  org.apache.sqoop.manager.SqlManager  - Executing SQL statement: SELECT t.* FROM `test` AS t LIMIT 1
15202 [main] INFO  org.apache.sqoop.manager.SqlManager  - Executing SQL statement: SELECT t.* FROM `test` AS t LIMIT 1
15206 [main] INFO  org.apache.sqoop.orm.CompilationManager  - HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH-5.0.1-1.cdh5.0.1.p0.47/lib/hadoop-mapreduce
23112 [main] INFO  org.apache.sqoop.orm.CompilationManager  - Writing jar file: /tmp/sqoop-yarn/compile/ab1d242ba1b3a0869bab06c1eb20c02f/test.jar
23140 [main] WARN  org.apache.sqoop.manager.MySQLManager  - It looks like you are importing from mysql.
23140 [main] WARN  org.apache.sqoop.manager.MySQLManager  - This transfer can be faster! Use the --direct
23140 [main] WARN  org.apache.sqoop.manager.MySQLManager  - option to exercise a MySQL-specific fast path.
23140 [main] INFO  org.apache.sqoop.manager.MySQLManager  - Setting zero DATETIME behavior to convertToNull (mysql)
23151 [main] INFO  org.apache.sqoop.mapreduce.ImportJobBase  - Beginning import of test
23232 [main] WARN  org.apache.sqoop.mapreduce.JobBase  - SQOOP_HOME is unset. May not be able to find all job dependencies.
25731 [main] INFO  org.apache.sqoop.mapreduce.db.DBInputFormat  - Using read commited transaction isolation
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat

 

 

Someone can help me?

Thank-you

Cloudera Employee
Posts: 34
Registered: ‎07-08-2013

Re: Oozie launcher never ends

Hi,

 

The launcher job sticks around the entire time the sqoop job is running (because it's running the sqoop CLI).  So when using Oozie, you need to make sure that your cluster has capacity.  In MR1 this was quite common if you didn't have enough map slots.  In MR2, I've seen this happen if you haven't configured your Scheduler properly or if you don't have enough memory in your node manager.

Software Engineer | Cloudera, Inc. | http://cloudera.com
Explorer
Posts: 21
Registered: ‎04-15-2014

Re: Oozie launcher never ends

I think could be right idea!
I change on cloudera manager max running apps for user root in cluster->Dynamic resource pools.
i also change in YARN configuration the java-heap-space.

Is there something else to change, may be in oozie configuration?

Explorer
Posts: 21
Registered: ‎04-15-2014

Re: Oozie launcher never ends

I'm trying this workflow under AWS-ec2 machines, for testing.
I have changed my mini-cluster with 3 nodes, with machines powerful (x.medium -> x2.large) and my workflow it's ok.
I search on internet some impo, but i don't find a tutorial-rules, there is some link for this? now i set virtual core, and other impostation with my mind but i want some more efficient.

New Contributor
Posts: 1
Registered: ‎08-20-2014

Re: Oozie launcher never ends

Was there ever resolution to this ? I am having the same problem.
Explorer
Posts: 21
Registered: ‎04-15-2014

Re: Oozie launcher never ends

Hi,

 

I changed the power of the machines in aws.
I have also some parameter of yarn, but the real change was to have more powerful machines.

Posts: 1,524
Kudos: 265
Solutions: 232
Registered: ‎07-31-2013

Re: Oozie launcher never ends

The problem was most likely that the NodeManager's exposed container and cpu container configs were set too low by default in the YARN configuration. Increasing this to appropriate values, such as ones suggested by http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_mapreduce_to_yar... will most likely resolve your 'hanging jobs' issue.
Backline Customer Operations Engineer
New Contributor
Posts: 2
Registered: ‎01-09-2015

Re: Oozie launcher never ends

I tried below solution it works perfectly for me.

1) Change the Hadoop schedule type from capacity scheduler to fair scheduler. Because for small cluster each queue assign some memory size (2048MB) to complete single map reduce job. If more than one map reduce job run in single queue mean it met deadlock.

Solution: add below property to yarn-site.xml

  <property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
  </property>
  <property>
    <name>yarn.scheduler.fair.allocation.file</name>
    <value>file:/%HADOOP_HOME%/etc/hadoop/fair-scheduler.xml</value>
  </property>

2) By default Hadoop Total memory size was allot as 8GB.

So if we run two mapreduce program memory used by Hadoop get more than 8GB so it met deadlock.

Solution: Increase the size of Total Memory of nodemanager using following properties at yarn-site.xml

<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>20960</value>
  </property>
  <property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>1024</value>
  </property>
  <property>
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>2048</value>
  </property>

So If user try to run more than two mapreduce program mean he need to increase nodemanager or he need to increase the size of total memory of Hadoop (note: Increasing the size will reduce the system usage memory. Above property file able to run 10 map reduce program concurrently.)

New Contributor
Posts: 4
Registered: ‎11-05-2016

Re: Oozie launcher never ends

Hi Kawin,

Your suggestion worked for me as well. I'm still wondering how a simple job which is expected to receive less than 10 records got stuck.


Many Thanks

Cloudera Employee
Posts: 22
Registered: ‎06-17-2016

Re: Oozie launcher never ends

Hi, 

 

Oozie uses an MR job to execute the actual action logic:

In your case when you execute the sqoop action in command line, Sqoop executes an MR job and uses two containers to complete the job (A container for the ApplicationMaster and one for the Mapper).

When you execute the same thing from Oozie, it uses four containers:

One AM and a Mapper for the Launcher in Oozie and Sqoop launches an other AM and a Mapper to do the actual job.

You may reduce the AM and Mapper memory for the Launchers as they don't need much memory ( except if you use Spark action with yarn-client mode that rund Spark in the Launcher Mapper) 

 

All the best, 

 

gezapeti

 

Announcements