Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

YARN Script Terasort Error

SOLVED Go to solution

YARN Script Terasort Error

Explorer

Hi Community,

 

I am trying to run a YARN script to test tune my cluster using teraGen and TeraSort. I have modified the script and got the teraGen part Running with no problems. However I keep getting an error when the teraSort begins:

 

Below is the error:

 

18/04/04 21:29:47 INFO terasort.TeraSort: starting
java.net.URISyntaxException: Illegal character in path at index 0:  /_partition.lst#_partition.lst
        at java.net.URI$Parser.fail(URI.java:2829)
        at java.net.URI$Parser.checkChars(URI.java:3002)
        at java.net.URI$Parser.parseHierarchical(URI.java:3086)
        at java.net.URI$Parser.parse(URI.java:3044)
        at java.net.URI.<init>(URI.java:595)
        at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:300)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

And below is the Script

 

#!/bin/sh
# Confirm the path values given below correspond to your installation

MR=/opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce
HADOOP=/opt/cloudera/parcels/CDH/bin

# Mark start of the loop
echo Testing loop started on `date`

# Mapper containers
for i in 8
do
   # Reducer containers
   for j in 1
   do
      # Container memory
      for k in 512 1024
      do
         # Set mapper JVM heap
         MAP_MB=`echo "($k*0.8)/1" | bc`

         # Set reducer JVM heap
         RED_MB=`echo "($k*0.8)/1" | bc`

        time ${HADOOP}/hadoop jar ${MR}/hadoop-examples.jar teragen \
                     -Dmapreduce.job.maps=$i \
                     -Dmapreduce.map.memory.mb=$k \
                     -Dmapreduce.map.java.opts.max.heap=$MAP_MB \
                     100000000 /user/devops/results/tg-10GB-${i}-${j}-${k} 1>teragen_${i}_${j}_${k}.out 2>teragen_${i}_${j}_${k}.err

       time ${HADOOP}/hadoop jar ${MR}/hadoop-examples.jar terasort \
                     -Dmapreduce.job.maps=$i \
                     -Dmapreduce.job.reduces=$j \
                     -Dmapreduce.map.memory.mb=$k \
                     -Dmapreduce.map.java.opts.max.heap=$MAP_MB \
                     -Dmapreduce.reduce.memory.mb=$k \
                     -Dmapreduce.reduce.java.opts.max.heap=$RED_MB \
                      /user/devops/results/tg-10GB-${i}-${j}-${k} \
                      /user/devops/results/ts-10GB-${i}-${j}-${k} 1>>tera_${i}_${j}_${k}.out 2>>tera_${i}_${j}_${k}.err

        $HADOOP/hadoop fs -rm -r -skipTrash /user/devops/results/tg-10GB-${i}-${j}-${k}
        $HADOOP/hadoop fs -rm -r -skipTrash /user/devops/results/ts-10GB-${i}-${j}-${k}
      done
   done
done

echo Testing loop ended on `date`

Been at this for a couple of days so any help is more than welcome!

 

thanks

1 ACCEPTED SOLUTION

Accepted Solutions

Re: YARN Script Terasort Error

Explorer

Ok so I finally found an answer. It seems copying from notepad to vi adds hidden characters had to code everything on vi and I had no issue.

1 REPLY 1

Re: YARN Script Terasort Error

Explorer

Ok so I finally found an answer. It seems copying from notepad to vi adds hidden characters had to code everything on vi and I had no issue.