Reply
New Contributor
Posts: 1
Registered: ‎03-31-2015

program works for hadoop 1.x but when i alter the code to hadoop 2.x program and i am getting below

I have program works for hadoop 1.x but when i alter the code to hadoop 2.x program and i am getting below error message

 
 
15/03/31 15:14:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/03/31 15:14:54 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
15/03/31 15:14:54 INFO mapred.JobClient: Cleaning up the staging area file:/tmp/hadoop-vm4learning/mapred/staging/vm4learning1651638436/.staging/job_local1651638436_0001
15/03/31 15:14:54 ERROR security.UserGroupInformation: PriviledgedActionException as:vm4learning cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/home/vm4learning/WorkSpace/BigData/mrassignment/output
Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/home/vm4learning/WorkSpace/BigData/mrassignment/output
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:235)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1054)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1071)
at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
at pageindex.main(pageindex.java:76)
 
 
code:
import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.FileSplit;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;


public class pageindex

{
public class mapperindex extends Mapper<LongWritable,Text,Text,Text>
{
public void mapind(LongWritable key, Text value, Context context) throws IOException, InterruptedException
{
StringTokenizer itr= new StringTokenizer(value.toString());
FileSplit filesplit = (FileSplit)context.getInputSplit();
String filename =filesplit.getPath().getName();

while(itr.hasMoreTokens())
{
context.write(new Text(itr.nextToken()), new Text(filename));

}
}
}

public class reducerindex extends Reducer<Text,Text,Text,Text>
{


public void reduceind(Text key, Iterable<Text> values,Context context ) throws IOException, InterruptedException
{

String s="";
while(values.iterator() != null)
{
s= s.concat((values.iterator().toString())+ "{");
}
context.write(key, new Text(s));

}
}
public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException

{
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();

if (otherArgs.length != 2) {
System.err.println("Usage: wordcount <in> <out>");
System.exit(2);
}
Job job = new Job(conf, "word count");

job.setJarByClass(pageindex.class);

job.setMapperClass(mapperindex.class);
/*job.setCombinerClass(IntSumReducer.class);*/
job.setReducerClass(reducerindex.class);

job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));

FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));

job.setNumReduceTasks(2);

//Submit the job and wait for it to complete
System.exit(job.waitForCompletion(true) ? 0 : 1);

}
}


suggestion pl...
 

 

Cloudera Employee
Posts: 13
Registered: ‎08-19-2015

Re: program works for hadoop 1.x but when i alter the code to hadoop 2.x program and i am getting be

It looks like there are some permissions issues or missing directory:

 

5/03/31 15:14:54 ERROR security.UserGroupInformation: PriviledgedActionException as:vm4learning cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/home/vm4learning/WorkSpace/BigData/mrassignment/output
Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/home/vm4learning/WorkSpace/BigData/mrassignment/output
 
Can you verify that the location:
/home/vm4learning/WorkSpace/BigData/mrassignment/output
 
Exists and also is it the correct path for the Input?
 
Can you share the command you used to initate the job?
 
Thanks
Josh
Announcements