Hi guys, I have read the article about testing and I would like to test spark-testing-base with spark. Unfortunately I am not an expert with Maven so I don't get the tests to run. My project looks like this: pom.xml src/main/scala/com/test/spark/mycode.scala src/test/scala/com/test/spark/test.scala I can run 'mvn package' without problems. But it says [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ mycode --- [INFO] Nothing to compile - all classes are up to date Why does my test not run? I have added the dependency as explained on the git for scala-testing-base and used the first example from its wiki. My pom.xml looks like this: <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.test.spark</groupId> <artifactId>mycode</artifactId> <version>0.0.1</version> <name>${project.artifactId}</name> <description>Simple wtest</description> <inceptionYear>2017</inceptionYear> <!-- change from 1.6 to 1.7 depending on Java version --> <properties> <maven.compiler.source>1.6</maven.compiler.source> <maven.compiler.target>1.6</maven.compiler.target> <encoding>UTF-8</encoding> <scala.version>2.11.5</scala.version> <scala.compat.version>2.11</scala.compat.version> <spark.version>1.6.1</spark.version> </properties> <dependencies> <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> <version>${scala.version}</version> </dependency> <!-- Spark dependency --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_${scala.compat.version}</artifactId> <version>${spark.version}</version> </dependency> <!-- Spark sql dependency --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_${scala.compat.version}</artifactId> <version>${spark.version}</version> </dependency> <!-- Spark hive dependency --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-hive_${scala.compat.version}</artifactId> <version>${spark.version}</version> </dependency> <!-- spark-testing-base dependency --> <dependency> <groupId>com.holdenkarau</groupId> <artifactId>spark-testing-base_2.11</artifactId> <version>${spark.version}_0.6.0</version> <scope>test</scope> </dependency> <dependency> <groupId>org.scalactic</groupId> <artifactId>scalactic_2.11</artifactId> <version>3.0.1</version> </dependency> <dependency> <groupId>org.scalatest</groupId> <artifactId>scalatest_2.11</artifactId> <version>3.2.0-SNAP5</version> <scope>test</scope> </dependency> </dependencies> <build> <sourceDirectory>src/main/scala</sourceDirectory> <testSourceDirectory>src/test/scala</testSourceDirectory> <!-- Create JAR with all dependencies --> <plugins> <plugin> <artifactId>maven-assembly-plugin</artifactId> <version>3.0.0</version> <executions> <execution> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> </configuration> </plugin> <plugin> <!-- see http://davidb.github.com/scala-maven-plugin --> <groupId>net.alchim31.maven</groupId> <artifactId>scala-maven-plugin</artifactId> <version>3.2.2</version> <configuration> <scalaVersion>${scala.version}</scalaVersion> <scalaCompatVersion>${scala.compat.version}</scalaCompatVersion> </configuration> <executions> <execution> <phase>compile</phase> <goals> <goal>compile</goal> </goals> </execution> </executions> </plugin> <!-- for testing scala code --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.20</version> <configuration> <argLine>-Xmx2048m -XX:MaxPermSize=2048m</argLine> </configuration> </plugin> </plugins> </build> </project> Do you have any advise what I need to change in order to have my test run?? Best, Ken ... View more

Hi, how can I start a Sqoop2 server on my Sandbox and use the Sqoop Client API to import data into HDFS? ... View more

Hi everybody, I have a question regarding Oozie and stabilyzing my development process. First of all to write safe code I found whis article about test driven development in hadoop. As far as I understand this means that I have to provide developer tests sperately for each tool (i.e. Sqoop, Hive, Spark,..). So how would a typical developement process look like in your opinion? Should all code for Hive and Spark be written first and then be tested by unit tests which were defined before developing the actual code? This means using Beeline or HiveRunner as well as Spark-Testing-Base. And only then test the Oozie Worklfow with minioozie? In addition I would like to know how I can handle errors in Oozie appropriately. I had the feeling that sometimes an error occured (maybe in Hive or something else) and the complete workflow was stopped at that point. So the action was stopped and did not even reach the point where Oozie decides whether to use the OK or ERROR branch. So al my error handling in Oozie was not useful. When and how can that happen? Is is a type of error which has to be tested before in the tool itself and not in Oozie? Maybe I do not really understand how Oozie delegates the action to YARN and where those errors rise. Any help on this topic is really appreciated. Brwosing the web I didn't come up with many input which refers precisely on this topic. Thanks in advance! ... View more

Hi guys, I am on HDP 2.5 and I would like to run a Spark action within Oozie. The jar was tested with spark-submit beofre. My action looks like this: (Namenode etc. from global settings) <action name="spark-action" retry-max="1"> <spark xmlns="uri:oozie:spark-action:0.2"> <master>yarn-cluster</master> <name>Spark Test</name> <class>org.SparkTest</class> <jar>/myhdfs/spark_test.jar</jar> <spark-opts>--executor-memory 2G</spark-opts> </spark> <ok to="end"/> <error to="kill"/> </action> At first I tried with mode="locale" but then I get an error that the master and mode don't fit together. (Client deploy mode is not compatible with master "yarn-cluster") When I leave it open, so no declaration of mode I get an error that the file does not exist. But the file is definetly in the HDFS in the right place. What do I need to change?? The error is: Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Application application_1491479953113_0414 finished with failed status org.apache.spark.SparkException: Application application_1491479953113_0414 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:1122) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1169) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:289) at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:211) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51) at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:59) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. In my job.properties I added the following: oozie.use.system.libpath=trueoozie.action.sharelib.for.spark = spark,hcatalog,hive Thanks for your help!! ... View more