Support Questions
Find answers, ask questions, and share your expertise

bad substitution error with ${hdp.version}

New Contributor

Hi,

I am trying to run Spring XD with hortonworks cluster (2.2.4.2-2). I am getting the following error. I am trying to run a sqoop job with the help of Spring XD. I can run the same sqoop job from the edge node through shell. I am not getting any issue

Stack trace: ExitCodeException exitCode=1: /grid/06/hadoop/yarn/local/usercache/mradmin/appcache/application_1454021423311_7817/container_1454021423311_7817_01_000002/launch_container.sh: line 26: $PWD:/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64: bad substitution
/grid/06/hadoop/yarn/local/usercache/mradmin/appcache/application_1454021423311_7817/container_1454021423311_7817_01_000002/launch_container.sh: line 58: $JAVA_HOME/bin/java -Dpepperdata.private.yarnTaskAttemptId=attempt_1454021423311_7817_m_000000_0 -Dorg.jboss.netty.epollBugWorkaround=true -Djava.protocol.handler.pkgs=com.pepperdata.supervisor.protocol.handler -Dpepperdata.private.taskAttemptId=container_1454021423311_7817_01_000002 -server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version} -Xmx4916m -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/grid/05/hadoop/yarn/log/application_1454021423311_7817/container_1454021423311_7817_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 172.27.30.132 59947 attempt_1454021423311_7817_m_000000_0 2 1>/grid/05/hadoop/yarn/log/application_1454021423311_7817/container_1454021423311_7817_01_000002/stdout 2>/grid/05/hadoop/yarn/log/application_1454021423311_7817/container_1454021423311_7817_01_000002/stderr : bad substitution
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.PepperdataContainerExecutor.launchContainer(SourceFile:267)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
6 REPLIES 6

Re: bad substitution error with ${hdp.version}

Re: bad substitution error with ${hdp.version}

New Contributor

@Neeraj Sabharwal: I refered that document only.

Re: bad substitution error with ${hdp.version}

Mentor
@Reju Ramachandran Pillai

you do need to refer to Spring XD documentation but what's happening is your variable for ${hdp.version} is not being accepted. Here's what I found in Spring XD docs, so when you launch the app, pass hdp22

There's more info at their github page.

Using Hadoop

Spring XD supports the following Hadoop distributions:

  • hadoop27 - Apache Hadoop 2.7.1 (default)
  • phd21 - Pivotal HD 2.1 and 2.0
  • phd30 - Pivotal HD 3.0
  • cdh5 - Cloudera CDH 5.3.0
  • hdp22 - Hortonworks Data Platform 2.2

To specify the distribution libraries to use for Hadoop client connections, use the option --hadoopDistro for the xd-container and xd-shell commands:

xd/bin>$ ./xd-shell --hadoopDistro <distribution>
xd/bin>$ ./xd-admin
xd/bin>$ ./xd-container --hadoopDistro <distribution>

Re: bad substitution error with ${hdp.version}

New Contributor

@Artem Ervits: I have tried with the option ./xd-container --hadoopDistro hdp22. Still, I am getting the same error. I also tried to add in the xd-container file with JAVA_OPTS. It also didn't work. It seems like hadoop distribution value is not passing when we are running through spring XD. A little bit background about this. We have added pepperdata.jar file in the cluster. So we have added that classpath in mapreduce classpath in server.yml file. From that time onwards, we are facing this issue. Before that, we didn't face any issue. It was running fine before that. We have to add the pepperdata.jar file.

Re: bad substitution error with ${hdp.version}

Mentor

What's the output of $PATH ? It makes sense that you are having such issues. Confirm with pepperdata folks how to modify classpath. Most likely you need to append to path and you just overwrote it. There is no documentation on their site, I cannot help there. @Reju Ramachandran Pillai

Re: bad substitution error with ${hdp.version}

Mentor

@Reju Ramachandran Pillai did it address you're issue?