Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

I get error when in execute the Hadoop c++ Pipes Wordcount example in hadoop 2.7.3

I get error when in execute the Hadoop c++ Pipes Wordcount example in hadoop 2.7.3

Contributor

I followed this link to run a c++ pipes wordcount mapreduce program in my hadoop 2.7.3,But it does not work for me !!

My code wordcount.cpp

#include <algorithm>
#include <limits>
#include <string>

#include  "stdint.h"  // <--- to prevent uint64_t errors! 
#include "Pipes.hh"
#include "TemplateFactory.hh"
#include "StringUtils.hh"


class WordCountMapper : public HadoopPipes::Mapper {
 public:
  // Constructor: does nothing
  WordCountMapper(HadoopPipes::TaskContext& context) {}
  // Map function: Receives a line, outputs (word,"1") to reducer.
  void map(HadoopPipes::MapContext& context) {
    // Get line of text
    std::string line = context.getInputValue();
    // Split into words
    std::vector<std::string> words = HadoopUtils::splitString(line, " ");
    // Emit each word tuple (word, "1" )
    for (unsigned int i = 0; i < words.size(); i++) {
      context.emit(words[i], HadoopUtils::toString(1));
    }
  }
};


class WordCountReducer : public HadoopPipes::Reducer {
 public:
  // Constructor: does nothing
  WordCountReducer(HadoopPipes::TaskContext& context) {}
  // Reduce function
  void reduce(HadoopPipes::ReduceContext& context) {
    int count = 0;
    // Get all tuples with the same key, and count their numbers
    while (context.nextValue()) {
      count += HadoopUtils::toInt(context.getInputValue());
    }
    // Emit (word, count)
    context.emit(context.getInputKey(), HadoopUtils::toString(count));
  }
};


int main(int argc, char *argv[]) {
  return HadoopPipes::runTask(
    HadoopPipes::TemplateFactory<WordCountMapper, WordCountReducer>()
  );
}

makefile

CC = g++
HADOOP_INSTALL = /usr/local/lib/hadoop-2.7.3
INC = -I$(HADOOP_INSTALL)/include
LIBS = -L$(HADOOP_INSTALL)/lib/native -lhadooppipes -lhadooputils -lpthread -lssl -lcrypto
CPPFLAGS = $(INC) -Wall -g -O2


wordcount: wordcount.cpp
	$(CC) $(CPPFLAGS) -o $@ $< $(LIBS)

Error

root@ubuntu:/usr/local/lib/hadoop-2.7.3# bin/mapred pipes -D hadoop.pipes.java.recordreader=true \
>                  -D hadoop.pipes.java.recordwriter=true \
>                  -input /user/root/wordcount/sotu_2015.txt   \
>                  -output /user/root/output                         \
>                  -program /user/root/wordcount/wordcount
17/01/11 06:28:49 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
17/01/11 06:28:49 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
17/01/11 06:28:49 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized


17/01/11 06:28:49 WARN mapreduce.JobResourceUploader: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
17/01/11 06:28:50 INFO mapred.FileInputFormat: Total input paths to process : 1
17/01/11 06:28:50 INFO mapreduce.JobSubmitter: number of splits:1
17/01/11 06:28:50 INFO Configuration.deprecation: hadoop.pipes.java.recordreader is deprecated. Instead, use mapreduce.pipes.isjavarecordreader
17/01/11 06:28:50 INFO Configuration.deprecation: hadoop.pipes.java.recordwriter is deprecated. Instead, use mapreduce.pipes.isjavarecordwriter
17/01/11 06:28:50 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local624096181_0001
17/01/11 06:28:51 INFO mapred.LocalDistributedCacheManager: Localized hdfs://Hadoop:54310/user/root/wordcount/wordcount as file:/app/hadoop/tmp/mapred/local/1484144930842/wordcount
17/01/11 06:28:51 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
17/01/11 06:28:51 INFO mapreduce.Job: Running job: job_local624096181_0001
17/01/11 06:28:51 INFO mapred.LocalJobRunner: OutputCommitter set in config null
17/01/11 06:28:51 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
17/01/11 06:28:51 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/01/11 06:28:51 INFO mapred.LocalJobRunner: Waiting for map tasks
17/01/11 06:28:51 INFO mapred.LocalJobRunner: Starting task: attempt_local624096181_0001_m_000000_0
17/01/11 06:28:52 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/01/11 06:28:52 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
17/01/11 06:28:52 INFO mapred.MapTask: Processing split: hdfs://Hadoop:54310/user/root/wordcount/sotu_2015.txt:0+37562
17/01/11 06:28:52 INFO mapred.MapTask: numReduceTasks: 1
17/01/11 06:28:52 INFO mapreduce.Job: Job job_local624096181_0001 running in uber mode : false
17/01/11 06:28:52 INFO mapreduce.Job:  map 0% reduce 0%
17/01/11 06:29:40 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/01/11 06:29:40 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/01/11 06:29:40 INFO mapred.MapTask: soft limit at 83886080
17/01/11 06:29:40 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/01/11 06:29:40 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/01/11 06:29:40 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/01/11 06:29:40 INFO mapred.LocalJobRunner: map task executor complete.
17/01/11 06:29:40 WARN mapred.LocalJobRunner: job_local624096181_0001
java.lang.Exception: java.lang.NullPointerException
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:104)
	at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:69)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
17/01/11 06:29:41 INFO mapreduce.Job: Job job_local624096181_0001 failed with state FAILED due to: NA
17/01/11 06:29:41 INFO mapreduce.Job: Counters: 0
Exception in thread "main" java.io.IOException: Job failed!
	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:873)
	at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:264)
	at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:503)
	at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:518)


Hadoop works well and the data is on hdfs

I need your help, thanks in advance