Member since
04-28-2015
7
Posts
0
Kudos Received
0
Solutions
10-13-2015
09:27 AM
I have an Oozie shell action that calls impala-shell and has been working fine. I wanted to set some defaults in ~/.impalarc which works fine from the shell but doesn't seem to get applied during workflow execution. When I look at the environment variable values in the shell action log it says: . . . USER=etl-user: . . . HOME=/home/: . . . USER looks right but I expect HOME to be either /home/$USER or /home/etl-user. Does this influence where impala-shell looks for .impalarc when running under an Oozie shell action? I was hoping to get this done by distributing my .impalarc to ~etl-user on all of the cluster nodes and not chenge the workflow definition but no joy. The objective is to change the default --impalad=host:port argument to point at the Impala high availabilty proxy that we have set up.
... View more
Labels:
- Labels:
-
Apache Oozie
09-03-2015
01:46 PM
From where should I copy hive-site.xml? I tried a couple that I found under /opt/cloudera/parcels individually but no joy.
... View more
04-28-2015
01:22 PM
Two solutions that I reasoned out by staring at the various scripts in /usr/lib/spark/bin SPARK_CLASSPATH=/usr/share/java/slf4j-simple.jar spark-shell --master local spark-shell --driver-class-path /usr/share/java/slf4j-simple.jar --master local The first approach is deprecated according to the output.
... View more
04-28-2015
01:06 PM
I have a similar problem trying to run spark-shell from CDH 5.4 under 14.04. I was able to get around this by saying: SPARK_PRINT_LAUNCH_COMMAND=true spark-shell :to get the underlying java invocation and then tack an SLF4J jar onto the classpath there: /usr/lib/jvm/java-7-oracle/bin/java -cp :/usr/lib/spark/conf:/usr/lib/spark/lib/spark-assembly-1.3.0-cdh5.4.0-hadoop2.6.0-cdh5.4.0.jar:/etc/hadoop/conf::/usr/lib/spark/lib/spark-assembly.jar::/usr/lib/hadoop/lib/*:/usr/lib/hadoop/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/*:/usr/lib/hive/lib/*:/usr/lib/flume-ng/lib/*:/usr/lib/paquet/lib/*:/usr/lib/avro/lib/*:/usr/share/java/slf4j-simple.jar -XX:MaxPermSize=128m -Dscala.usejavacp=true -Xms512m -Xmx512m org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main spark-shell There must be something missing from one of the scripts or JARs in the CDH Ubuntu repo. Any suggestions on how to hack it so that spark-shell works until this is addressed?
... View more