Support Questions
Find answers, ask questions, and share your expertise

issue in spark-solr building snapshot.jar using maven

Highlighted

issue in spark-solr building snapshot.jar using maven

Explorer

Hi

I am struggling with building jar files using maven for spark-solr. Spark-Solr provides integration between spark and solr and maven builds the source to provide necessary jar file.

I have scala 2.10.5 installed with spark 1.6.2 and solr 5.2.1 on top of HDP 2.3

I am running mvn clean package -DskipTests -e which gives me below error

[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-solr: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. CompileFailed -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-solr: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:144)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
        ... 19 more

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

Also find below pom.xml

<?xml version="1.0"?><projectxmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"><modelVersion>4.0.0</modelVersion><groupId>com.lucidworks.spark</groupId><artifactId>spark-solr</artifactId><version>2.2.0</version><packaging>jar</packaging><name>spark-solr</name><description>Tools for reading data from Spark into Solr</description><url>https://github.com/LucidWorks/spark-solr</url><licenses><license><name>Apache License, Version 2.0</name><url>http://www.apache.org/licenses/LICENSE-2.0.txt</url><distribution>repo</distribution><comments>A business-friendly OSS license</comments></license></licenses><developers><developer><name>Lucidworks</name><organization>Lucidworks</organization><organizationUrl>www.lucidworks.com</organizationUrl></developer></developers><scm><connection>scm:git:git@github.com:LucidWorks/spark-solr.git</connection><developerConnection>scm:git:git@github.com:LucidWorks/spark-solr.git</developerConnection><url>https://github.com/LucidWorks/spark-solr</url></scm><distributionManagement><snapshotRepository><id>ossrh</id><url>https://oss.sonatype.org/content/repositories/snapshots</url></snapshotRepository></distributionManagement><properties><project.build.sourceEncoding>UTF-8</project.build.sourceEncoding><java.version>1.8</java.version><spark.version>1.6.2</spark.version><solr.version>6.1.0</solr.version><fasterxml.version>2.4.0</fasterxml.version><scala.version>2.10.5</scala.version><scala.binary.version>2.10</scala.binary.version><scoverage.plugin.version>1.1.1</scoverage.plugin.version><MaxPermSize>128m</MaxPermSize></properties><profiles><profile><id>release</id><build><plugins><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-javadoc-plugin</artifactId><version>2.10.3</version><executions><execution><id>attach-javadocs</id><goals><goal>jar</goal></goals></execution></executions></plugin><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-gpg-plugin</artifactId><version>1.6</version><executions><execution><id>sign-artifacts</id><phase>verify</phase><goals><goal>sign</goal></goals></execution></executions></plugin></plugins></build></profile><profile><id>coverage</id><activation><activeByDefault>false</activeByDefault></activation><build><plugins><plugin><groupId>org.jacoco</groupId><artifactId>jacoco-maven-plugin</artifactId><version>0.7.4.201502262128</version><configuration><excludes><exclude>com/lucidworks/spark/example/**</exclude><exclude>**/*Test*</exclude></excludes></configuration><executions><execution><id>default-prepare-agent</id><goals><goal>prepare-agent</goal></goals></execution><execution><id>default-report</id><phase>package</phase><goals><goal>report</goal></goals></execution></executions></plugin><plugin><groupId>org.scoverage</groupId><artifactId>scoverage-maven-plugin</artifactId><version>${scoverage.plugin.version}</version><configuration><scalaVersion>${scala.version}</scalaVersion></configuration></plugin><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.18.1</version><configuration><argLine>${argLine} -XX:MaxPermSize=${MaxPermSize}</argLine></configuration></plugin></plugins></build></profile></profiles><build><plugins><plugin><groupId>net.alchim31.maven</groupId><artifactId>scala-maven-plugin</artifactId><version>3.2.2</version><executions><execution><id>eclipse-add-source</id><goals><goal>add-source</goal></goals></execution><execution><id>scala-compile-first</id><phase>process-resources</phase><goals><goal>compile</goal></goals></execution><execution><id>scala-test-compile-first</id><phase>process-test-resources</phase><goals><goal>testCompile</goal></goals></execution><execution><id>attach-scaladocs</id><phase>verify</phase><goals><goal>doc-jar</goal></goals></execution></executions><configuration><scalaVersion>${scala.version}</scalaVersion><recompileMode>incremental</recompileMode><useZincServer>true</useZincServer><javacArgs><javacArg>-source</javacArg><javacArg>${java.version}</javacArg><javacArg>-target</javacArg><javacArg>${java.version}</javacArg></javacArgs></configuration></plugin><plugin><artifactId>maven-compiler-plugin</artifactId><version>3.0</version><configuration><source>${java.version}</source><target>${java.version}</target></configuration></plugin><plugin><groupId>org.codehaus.mojo</groupId><artifactId>versions-maven-plugin</artifactId><version>2.3</version></plugin><!-- Surefire runs Java tests --><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.18.1</version><configuration><argLine> -XX:MaxPermSize=${MaxPermSize}</argLine></configuration></plugin><!-- ScalaTest runs Scala tests --><plugin><groupId>org.scalatest</groupId><artifactId>scalatest-maven-plugin</artifactId><version>1.0</version><configuration><stderr/><!-- Log to stderr --><junitxml>.</junitxml><!-- Directory/ies for JUnit XML --><reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory><argLine>-ea -XX:MaxPermSize=${MaxPermSize}</argLine><systemProperties><java.awt.headless>true</java.awt.headless><java.io.tmpdir>${project.build.directory}/tmp</java.io.tmpdir></systemProperties></configuration><executions><execution><id>test</id><goals><goal>test</goal></goals></execution></executions></plugin><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-shade-plugin</artifactId><version>2.3</version><executions><execution><phase>package</phase><goals><goal>shade</goal></goals><configuration><shadedArtifactAttached>true</shadedArtifactAttached><shadedClassifierName>shaded</shadedClassifierName><filters><filter><artifact>*:*</artifact><excludes><exclude>META-INF/*.SF</exclude><exclude>META-INF/*.DSA</exclude><exclude>META-INF/*.RSA</exclude></excludes></filter></filters><artifactSet><excludes><exclude>org.apache.spark:spark-core_${scala.binary.version}</exclude><exclude>org.apache.spark:spark-streaming_${scala.binary.version}</exclude><exclude>org.apache.spark:spark-hive_${scala.binary.version}</exclude><exclude>org.apache.hadoop:*</exclude><exclude>com.google.protobuf_spark:*</exclude><exclude>com.twitter:chill_${scala.binary.version}</exclude><exclude>com.twitter:chill-java</exclude><exclude>org.spark-project.akka:*</exclude><exclude>org.spark-project:*</exclude><exclude>net.java.dev.jets3t:jets3t</exclude><exclude>org.eclipse.jetty:*</exclude><exclude>org.eclipse.jetty.orbit:*</exclude><exclude>org.slf4j:*</exclude><exclude>org.scala-lang:scala-library</exclude><exclude>commons-httpclient:commons-httpclient</exclude><exclude>org.apache.curator:*</exclude><exclude>org.apache.commons:commons-lang3</exclude><exclude>com.google.code.findbugs</exclude><exclude>com.ning:compress-lzf</exclude><exclude>org.xerial.snappy:snappy-java</exclude><exclude>net.jpountz.lz4:lz4</exclude><exclude>com.esotericsoftware.kryo:*</exclude><exclude>log4j:log4j</exclude><exclude>com.esotericsoftware.reflectasm:reflectasm</exclude><exclude>com.esotericsoftware.minlog:minlog</exclude><exclude>org.objenesis:objenesis</exclude><exclude>commons-net:commons-net</exclude><exclude>org.apache.james:*</exclude><exclude>org.ow2.asm:asm-debug-all</exclude><exclude>javax.xml.stream:stax-api</exclude><exclude>javax.ws.rs:javax.ws.rs-api</exclude><exclude>xml-apis:xml-apis</exclude><exclude>org.ow2.asm:asm</exclude><exclude>com.codahale.metrics:*</exclude><exclude>io.dropwizard.metrics:*</exclude><exclude>com.sun.xml.bind:jaxb-impl</exclude><exclude>com.sun.jersey:jersey-client</exclude><exclude>org.mortbay.jetty:jetty-util</exclude></excludes></artifactSet><relocations><relocation><pattern>org.apache.http</pattern><shadedPattern>shaded.apache.http</shadedPattern></relocation><relocation><pattern>org.codehaus.jackson</pattern><shadedPattern>shaded.codehaus.jackson</shadedPattern></relocation><relocation><pattern>com.fasterxml.jackson</pattern><shadedPattern>shaded.fasterxml.jackson</shadedPattern></relocation><relocation><pattern>org.objectweb.asm</pattern><shadedPattern>shaded.objectweb.asm</shadedPattern></relocation><relocation><pattern>org.joda.time</pattern><shadedPattern>shaded.joda.time</shadedPattern></relocation><relocation><pattern>com.google.guava</pattern><shadedPattern>shaded.google.guava</shadedPattern></relocation><relocation><pattern>com.sun.jersey</pattern><shadedPattern>shaded.sun.jersey</shadedPattern></relocation><relocation><pattern>javax.ws.rs</pattern><shadedPattern>shaded.javax.ws.rs</shadedPattern></relocation><!-- Commented out because the maven-shade-plugin does not work for Scala classes due to ScalaSig--><!--<relocation>--><!--<pattern>org.apache.solr</pattern>--><!--<shadedPattern>shaded.apache.solr</shadedPattern>--><!--</relocation>--><relocation><pattern>org.apache.commons.cli</pattern><shadedPattern>shaded.apache.commons.cli</shadedPattern></relocation></relocations></configuration></execution></executions></plugin><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-source-plugin</artifactId><version>2.4</version><executions><execution><id>attach-sources</id><goals><goal>jar</goal></goals></execution></executions></plugin><plugin><groupId>org.sonatype.plugins</groupId><artifactId>nexus-staging-maven-plugin</artifactId><version>1.6.6</version><extensions>true</extensions><configuration><serverId>ossrh</serverId><nexusUrl>https://oss.sonatype.org/</nexusUrl><autoReleaseAfterClose>true</autoReleaseAfterClose></configuration></plugin></plugins></build><dependencies><dependency><groupId>org.apache.spark</groupId><artifactId>spark-core_${scala.binary.version}</artifactId><version>${spark.version}</version><scope>provided</scope><exclusions><exclusion><groupId>org.eclipse.jetty.orbit</groupId><artifactId>javax.servlet</artifactId></exclusion></exclusions></dependency><dependency><groupId>com.esotericsoftware.kryo</groupId><artifactId>kryo</artifactId><version>2.21</version><scope>compile</scope></dependency><dependency><groupId>com.codahale.metrics</groupId><artifactId>metrics-core</artifactId><version>3.0.0</version><scope>compile</scope></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-streaming_${scala.binary.version}</artifactId><version>${spark.version}</version><scope>provided</scope></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-mllib_${scala.binary.version}</artifactId><version>${spark.version}</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-streaming-twitter_${scala.binary.version}</artifactId><version>${spark.version}</version></dependency><dependency><groupId>commons-cli</groupId><artifactId>commons-cli</artifactId><version>1.3.1</version></dependency><dependency><groupId>commons-io</groupId><artifactId>commons-io</artifactId><version>2.3</version></dependency><dependency><groupId>org.apache.solr</groupId><artifactId>solr-solrj</artifactId><version>${solr.version}</version><type>jar</type></dependency><dependency><groupId>org.apache.solr</groupId><artifactId>solr-core</artifactId><version>${solr.version}</version><type>jar</type></dependency><dependency><groupId>org.apache.solr</groupId><artifactId>solr-test-framework</artifactId><version>${solr.version}</version><type>jar</type><scope>test</scope></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>2.6.0</version><scope>compile</scope><exclusions><exclusion><groupId>org.slf4j</groupId><artifactId>slf4j-log4j12</artifactId></exclusion><exclusion><groupId>javax.servlet</groupId><artifactId>servlet-api</artifactId></exclusion><exclusion><groupId>org.restlet.jee</groupId><artifactId>org.restlet.ext.servlet</artifactId></exclusion></exclusions></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-common</artifactId><version>2.6.0</version><scope>compile</scope><exclusions><exclusion><groupId>org.slf4j</groupId><artifactId>slf4j-log4j12</artifactId></exclusion><exclusion><groupId>javax.servlet</groupId><artifactId>servlet-api</artifactId></exclusion><exclusion><groupId>org.restlet.jee</groupId><artifactId>org.restlet.ext.servlet</artifactId></exclusion><exclusion><groupId>javax.servlet.jsp</groupId><artifactId>jsp-api</artifactId></exclusion></exclusions></dependency><dependency><groupId>org.apache.commons</groupId><artifactId>commons-lang3</artifactId><version>3.3.2</version></dependency><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.11</version><scope>test</scope></dependency><dependency><groupId>com.fasterxml.jackson.core</groupId><artifactId>jackson-core</artifactId><version>${fasterxml.version}</version></dependency><dependency><groupId>com.fasterxml.jackson.core</groupId><artifactId>jackson-databind</artifactId><version>${fasterxml.version}</version></dependency><dependency><groupId>com.fasterxml.jackson.module</groupId><artifactId>jackson-module-scala_${scala.binary.version}</artifactId><version>${fasterxml.version}</version></dependency><dependency><groupId>org.scala-lang</groupId><artifactId>scala-library</artifactId><version>${scala.version}</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-core_${scala.binary.version}</artifactId><version>${spark.version}</version><classifier>tests</classifier><scope>test</scope></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-mllib_${scala.binary.version}</artifactId><version>${spark.version}</version><classifier>tests</classifier><scope>test</scope></dependency><dependency><groupId>org.scalatest</groupId><artifactId>scalatest_${scala.binary.version}</artifactId><version>2.2.1</version><scope>test</scope></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-hive_${scala.binary.version}</artifactId><version>${spark.version}</version></dependency><dependency><groupId>com.databricks</groupId><artifactId>spark-csv_${scala.binary.version}</artifactId><version>1.4.0</version></dependency><dependency><groupId>org.apache.commons</groupId><artifactId>commons-compress</artifactId><version>1.4.1</version></dependency><dependency><!-- optional dependency of joda-time that is mandatory in Scala --><groupId>org.joda</groupId><artifactId>joda-convert</artifactId><version>1.2</version></dependency><dependency><groupId>com.github.tomakehurst</groupId><artifactId>wiremock</artifactId><version>1.56</version><!-- Include everything below here if you have dependency conflicts --><classifier>standalone</classifier><exclusions><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>jetty</artifactId></exclusion><exclusion><groupId>com.google.guava</groupId><artifactId>guava</artifactId></exclusion><exclusion><groupId>com.fasterxml.jackson.core</groupId><artifactId>jackson-core</artifactId></exclusion><exclusion><groupId>com.fasterxml.jackson.core</groupId><artifactId>jackson-annotations</artifactId></exclusion><exclusion><groupId>com.fasterxml.jackson.core</groupId><artifactId>jackson-databind</artifactId></exclusion><exclusion><groupId>org.apache.httpcomponents</groupId><artifactId>httpclient</artifactId></exclusion><exclusion><groupId>org.skyscreamer</groupId><artifactId>jsonassert</artifactId></exclusion><exclusion><groupId>xmlunit</groupId><artifactId>xmlunit</artifactId></exclusion><exclusion><groupId>com.jayway.jsonpath</groupId><artifactId>json-path</artifactId></exclusion><exclusion><groupId>net.sf.jopt-simple</groupId><artifactId>jopt-simple</artifactId></exclusion><exclusion><groupId>org.apache.httpcomponents</groupId><artifactId>httpclient</artifactId></exclusion><exclusion><groupId>org.apache.httpcomponents</groupId><artifactId>httpcore</artifactId></exclusion></exclusions><scope>test</scope></dependency></dependencies></project>

Thanks in advance for your help.

Omer

1 REPLY 1

Re: issue in spark-solr building snapshot.jar using maven

Explorer

Ahh... ok nobody answered but issue resolved.

we need to older branch. spark-solr 2.0.4 for my configuration