Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Testing with spark-testing-base and maven

Testing with spark-testing-base and maven

Contributor

Hi guys,

I have read the article about testing and I would like to test spark-testing-base with spark. Unfortunately I am not an expert with Maven so I don't get the tests to run. My project looks like this:

pom.xml
src/main/scala/com/test/spark/mycode.scala
src/test/scala/com/test/spark/test.scala

I can run 'mvn package' without problems. But it says

[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ mycode ---
[INFO] Nothing to compile - all classes are up to date

Why does my test not run? I have added the dependency as explained on the git for scala-testing-base and used the first example from its wiki.

My pom.xml looks like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.test.spark</groupId>
  <artifactId>mycode</artifactId>
  <version>0.0.1</version>
  <name>${project.artifactId}</name>
  <description>Simple wtest</description>
  <inceptionYear>2017</inceptionYear>
  
  <!-- change from 1.6 to 1.7 depending on Java version -->
  <properties>
    <maven.compiler.source>1.6</maven.compiler.source>
    <maven.compiler.target>1.6</maven.compiler.target>
    <encoding>UTF-8</encoding>
    <scala.version>2.11.5</scala.version>
    <scala.compat.version>2.11</scala.compat.version>
    <spark.version>1.6.1</spark.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>${scala.version}</version>
    </dependency>

    <!-- Spark dependency -->
    <dependency>         
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_${scala.compat.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <!-- Spark sql dependency -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.compat.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <!-- Spark hive dependency -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_${scala.compat.version}</artifactId>
        <version>${spark.version}</version>
    </dependency>
    <!-- spark-testing-base dependency -->
    <dependency>
        <groupId>com.holdenkarau</groupId>
        <artifactId>spark-testing-base_2.11</artifactId>
        <version>${spark.version}_0.6.0</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.scalactic</groupId>
        <artifactId>scalactic_2.11</artifactId>
        <version>3.0.1</version>
    </dependency>
    <dependency>
        <groupId>org.scalatest</groupId>
        <artifactId>scalatest_2.11</artifactId>
        <version>3.2.0-SNAP5</version>
        <scope>test</scope>
    </dependency>
 </dependencies>

  <build>
    <sourceDirectory>src/main/scala</sourceDirectory>
    <testSourceDirectory>src/test/scala</testSourceDirectory>
    <!-- Create JAR with all dependencies -->
    <plugins>
       <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>3.0.0</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
        </configuration>
      </plugin>
      <plugin>
        <!-- see http://davidb.github.com/scala-maven-plugin -->
        <groupId>net.alchim31.maven</groupId>
        <artifactId>scala-maven-plugin</artifactId>
        <version>3.2.2</version>
        <configuration>
            <scalaVersion>${scala.version}</scalaVersion>
            <scalaCompatVersion>${scala.compat.version}</scalaCompatVersion>
        </configuration>
        <executions>
          <execution>
            <phase>compile</phase>
            <goals>
              <goal>compile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <!-- for testing scala code -->
      <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>2.20</version>
          <configuration>
            <argLine>-Xmx2048m -XX:MaxPermSize=2048m</argLine>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>


Do you have any advise what I need to change in order to have my test run??

Best,

Ken

2 REPLIES 2
Highlighted

Re: Testing with spark-testing-base and maven

New Contributor

Hi Ken,

Spark 1.6.1 is built on scala 2.10.x. You have to use the right scala version to compile and run tests on your spark-scala code. I see the scala version mentioned in your pom.xml is 2.11.5. Can you replace the scala version with 2.10 and build your project ?

Re: Testing with spark-testing-base and maven

Contributor

Unfortunately it does not change it. The tests are still not run. :(

Don't have an account?
Coming from Hortonworks? Activate your account here