Reply
New Contributor
Posts: 1
Registered: ‎11-19-2013

job only submit to local not to cluster

I am testing CDH5(HDFS2 + MRV1) with 11 virtual Nodes(1 master(namenode + jobtracker) + 10 worker node(datanode+ tasktracker)) with CentOS5.8

Following the CDH5 installion guide  http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/c...

However hit a weird issue that “all jobs only can submit to localhost not to cluster(jobtracker)” try with two case (client combine with master and worker node).

Even if run job with “–jt jobtracker:8021” still can’t resolve this issue.

Do you hit the same issue?

 

mapred-site.xml:

<property>
   <name>mapred.job.tracker</name>
   <value>namenode:8021</value>
   <description>The host and port that the MapReduce job tracker runsat.</description>
</property>

 

core-site.xml:

<property>
        <name>fs.defaultFS</name>
        <value>hdfs://namenode:8020</value>
        <description>
            The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation.
        </description>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
    </property>

 

hdfs-site.xml

   <property>
       <name>dfs.replication</name>
       <value>3</value>
   </property>
   <property>
       <name>dfs.block.size</name>
       <value>67108864</value>
   </property>
   <property>
        <name>dfs.http.address</name>
        <value>namenode:8087</value>
   </property>
   <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///hadoop/hadoop-data/namenodeyum</value>
   </property>
   <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
   </property> 

Highlighted
Posts: 416
Topics: 51
Kudos: 89
Solutions: 49
Registered: ‎06-26-2013

Re: job only submit to local not to cluster

I have moved this topic to the MapReduce discussion board.

 

Usually when this happens, it's because you have not set up your environment correctly so that when you submit the MR job, it's not picking up your mapred-site.xml file and therefore selects the local job runner.

 

Maybe this page will help you get the environment variables/conf files set up correctly:

 

http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/SingleCluster.html#Setting_up...