YARN Script Terasort Error

CrispinVV — Fri, 16 Sep 2022 13:04:00 GMT

Hi Community,

I am trying to run a YARN script to test tune my cluster using teraGen and TeraSort. I have modified the script and got the teraGen part Running with no problems. However I keep getting an error when the teraSort begins:

Below is the error:

18/04/04 21:29:47 INFO terasort.TeraSort: starting
java.net.URISyntaxException: Illegal character in path at index 0:  /_partition.lst#_partition.lst
        at java.net.URI$Parser.fail(URI.java:2829)
        at java.net.URI$Parser.checkChars(URI.java:3002)
        at java.net.URI$Parser.parseHierarchical(URI.java:3086)
        at java.net.URI$Parser.parse(URI.java:3044)
        at java.net.URI.<init>(URI.java:595)
        at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:300)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

And below is the Script

#!/bin/sh
# Confirm the path values given below correspond to your installation

MR=/opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce
HADOOP=/opt/cloudera/parcels/CDH/bin

# Mark start of the loop
echo Testing loop started on `date`

# Mapper containers
for i in 8
do
   # Reducer containers
   for j in 1
   do
      # Container memory
      for k in 512 1024
      do
         # Set mapper JVM heap
         MAP_MB=`echo "($k*0.8)/1" | bc`

         # Set reducer JVM heap
         RED_MB=`echo "($k*0.8)/1" | bc`

        time ${HADOOP}/hadoop jar ${MR}/hadoop-examples.jar teragen \
                     -Dmapreduce.job.maps=$i \
                     -Dmapreduce.map.memory.mb=$k \
                     -Dmapreduce.map.java.opts.max.heap=$MAP_MB \
                     100000000 /user/devops/results/tg-10GB-${i}-${j}-${k} 1>teragen_${i}_${j}_${k}.out 2>teragen_${i}_${j}_${k}.err

       time ${HADOOP}/hadoop jar ${MR}/hadoop-examples.jar terasort \
                     -Dmapreduce.job.maps=$i \
                     -Dmapreduce.job.reduces=$j \
                     -Dmapreduce.map.memory.mb=$k \
                     -Dmapreduce.map.java.opts.max.heap=$MAP_MB \
                     -Dmapreduce.reduce.memory.mb=$k \
                     -Dmapreduce.reduce.java.opts.max.heap=$RED_MB \
                      /user/devops/results/tg-10GB-${i}-${j}-${k} \
                      /user/devops/results/ts-10GB-${i}-${j}-${k} 1>>tera_${i}_${j}_${k}.out 2>>tera_${i}_${j}_${k}.err

        $HADOOP/hadoop fs -rm -r -skipTrash /user/devops/results/tg-10GB-${i}-${j}-${k}
        $HADOOP/hadoop fs -rm -r -skipTrash /user/devops/results/ts-10GB-${i}-${j}-${k}
      done
   done
done

echo Testing loop ended on `date`

Been at this for a couple of days so any help is more than welcome!

thanks

Re: YARN Script Terasort Error

CrispinVV — Mon, 07 May 2018 16:43:03 GMT

Ok so I finally found an answer. It seems copying from notepad to vi adds hidden characters had to code everything on vi and I had no issue.

question Re: YARN Script Terasort Error in Support Questions

YARN Script Terasort Error

Re: YARN Script Terasort Error