Reply
New Contributor
Posts: 3
Registered: ‎05-23-2015

Re: Oozie-sqoop action in CDH 5.2-Heart beat issue

Thanks!!
Looks like you haven't seen my previous email. I was able to resolve the issue as it was due to lack of RAM on my system after closing some few unwanted applications. The Heartbeat issue was over and the pig script worked in hue.
Although, the entire process as I have mentioned is extremely slow again due to lack of RAM 4GB :
Hopefully, In the pursuit of happiness,  in the coming future. I will also climb the expensive 16GB ladder :)
But, I once again appreciate your effort!
Cheers,vibhor
New Contributor
Posts: 4
Registered: ‎06-22-2016

Re: Oozie-sqoop action in CDH 5.2-Heart beat issue

I am running sqoop jobs in paralell in a workflow and facing the same error. Since I am running the whole thing on the AWS cluster, I am convinced that it is not a memory problem. If any other please suggest.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<workflow-app xmlns="uri:oozie:workflow:0.1" name="WorkflowWithSqoopAction">
<start to="fork-node"/>
<fork name="fork-node">
<path start="BOOKS"/>
<path start="SALES"/>
<path start="EMPLOYEE"/>
</fork>
<action name="BOOKS">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<command>import --connect jdbc:oracle:thin:@ip:1521/ORCL --username=x --password=x --table=BOOKS</command>
</sqoop>
<ok to="joining"/>
<error to="fail"/>
</action>
<action name="SALES">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<command>import --connect jdbc:oracle:thin:@ip:1521/ORCL --username=x --password=x --table=SALES</command>
</sqoop>
<ok to="joining"/>
<error to="fail"/>
</action>
<action name="EMPLOYEE">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<command>import --connect jdbc:oracle:thin:@ip:1521/ORCL --username=x --password=x --table=EMPLOYEE</command>
</sqoop>
<ok to="joining"/>
<error to="fail"/>
</action>
<join name="joining" to="end"/>
<kill name="fail">
<message>Killed job due to error: ${wf:errorMessage(wf:lastErrorNode())}</message>
</kill>
<end name="end"/>
</workflow-app> 

 

 

 

 

4514 [uber-SubtaskRunner] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
4538 [uber-SubtaskRunner] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 1.4.6-cdh5.5.2
4550 [uber-SubtaskRunner] WARN org.apache.sqoop.tool.BaseSqoopTool - Setting your password on the command-line is insecure. Consider using -P instead.
4561 [uber-SubtaskRunner] WARN org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
4636 [uber-SubtaskRunner] INFO org.apache.sqoop.manager.oracle.OraOopManagerFactory - Data Connector for Oracle and Hadoop is disabled.
4650 [uber-SubtaskRunner] INFO org.apache.sqoop.manager.SqlManager - Using default fetchSize of 1000
4650 [uber-SubtaskRunner] INFO org.apache.sqoop.tool.CodeGenTool - Beginning code generation
5065 [uber-SubtaskRunner] INFO org.apache.sqoop.manager.OracleManager - Time zone has been set to GMT
5140 [uber-SubtaskRunner] INFO org.apache.sqoop.manager.SqlManager - Executing SQL statement: SELECT t.* FROM BOOKS t WHERE 1=0
5166 [uber-SubtaskRunner] INFO org.apache.sqoop.orm.CompilationManager - HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/lib/hadoop-mapreduce
6719 [uber-SubtaskRunner] INFO org.apache.sqoop.orm.CompilationManager - Writing jar file: /tmp/sqoop-yarn/compile/82166d0efd936226575485974d82d7b8/BOOKS.jar
6731 [uber-SubtaskRunner] INFO org.apache.sqoop.manager.OracleManager - Time zone has been set to GMT
6738 [uber-SubtaskRunner] INFO org.apache.sqoop.manager.OracleManager - Time zone has been set to GMT
6743 [uber-SubtaskRunner] INFO org.apache.sqoop.mapreduce.ImportJobBase - Beginning import of BOOKS
6764 [uber-SubtaskRunner] INFO org.apache.sqoop.manager.OracleManager - Time zone has been set to GMT
6781 [uber-SubtaskRunner] WARN org.apache.sqoop.mapreduce.JobBase - SQOOP_HOME is unset. May not be able to find all job dependencies.
7497 [uber-SubtaskRunner] INFO org.apache.sqoop.mapreduce.db.DBInputFormat - Using read commited transaction isolation
7497 [uber-SubtaskRunner] INFO org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat - BoundingValsQuery: SELECT MIN(BOOK_ID), MAX(BOOK_ID) FROM BOOKS
7501 [uber-SubtaskRunner] WARN org.apache.sqoop.mapreduce.db.TextSplitter - Generating splits for a textual index column.
7501 [uber-SubtaskRunner] WARN org.apache.sqoop.mapreduce.db.TextSplitter - If your database sorts in a case-insensitive order, this may result in a partial import or duplicate records.
7501 [uber-SubtaskRunner] WARN org.apache.sqoop.mapreduce.db.TextSplitter - You are strongly encouraged to choose an integral split column.
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat

 

New Contributor
Posts: 4
Registered: ‎11-05-2016

Re: Oozie-sqoop action in CDH 5.2-Heart beat issue

Thanks. it works after retaining the default values for memory and cpu cores using cloudera manager. But I'm not sure which xml file I should change as I have same set of files inside conf.cloudera.yarn and conf.pseudo. Someone pls tell me which xml file is modified for the below given screenshots. Tnx in advance.

$ ls -alt /etc/hadoop
total 36
drwxr-xr-x 2 root root 4096 Nov 26 04:42 conf.cloudera.yarn
drwxr-xr-x 2 root root 4096 Nov 26 04:41 conf.pseudo
drwxr-xr-x. 122 root root 12288 Nov 26 04:21 ..
drwxr-xr-x 7 root root 4096 Nov 23 13:49 .
lrwxrwxrwx 1 root root 29 Nov 23 13:49 conf -> /etc/alternatives/hadoop-conf
drwxr-xr-x 2 root root 4096 Nov 23 13:49 conf.cloudera.hdfs
drwxr-xr-x 2 root root 4096 Aug 10 14:07 conf.empty
drwxr-xr-x 2 root root 4096 Aug 10 14:06 conf.impala
lrwxrwxrwx 1 root root 10 Aug 10 14:06 conf.dist -> conf.empty

 

Capture.PNG

Highlighted
Explorer
Posts: 8
Registered: ‎03-10-2017

Re: Oozie-sqoop action in CDH 5.2-Heart beat issue

Hi BigdataFunda,
I Have same Probleme, so i work with cloudera quickstart 5.10 .
i would now, whzt do you mean increase cores !! increase vcores in file conf yearn.xml !!!
thank you !!
Announcements