STORM : How do I fix the google.guava dependency while running mvn clean install -DskipTests=true ?

[#1]

I'm getting this error :

The POM for com.google.guava:guava-testlib:jar:16.0.1 is invalid, transitive dependencies (if any) will not be available: 1 problem was encountered while building the effective model for com.google.guava:guava-testlib:[unknown-version]

[#2]

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project storm-elasticsearch: Compilation failure: Compilation failure:
[ERROR] /root/storm/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/common/EsConfigTest.java:[23,33] package com.google.common.collect does not exist
[ERROR] /root/storm/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/common/EsConfigTest.java:[23,33] package com.google.common.collect does not exist
[ERROR] /root/storm/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/common/EsConfigTest.java:[57,16] cannot find symbol
[ERROR] symbol:   variable ImmutableMap
[ERROR] location: class org.apache.storm.elasticsearch.common.EsConfigTest
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project storm-elasticsearch: Compilation failure
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: Compilation failure
        at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:858)
        at org.apache.maven.plugin.compiler.TestCompilerMojo.execute(TestCompilerMojo.java:152)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
        ... 19 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :storm-elasticsearch

[#3]

@keerthana gajarajakumar

what version of maven do you have and please explain what you're doing to reproduce.

[#4]

Maven version is 3.2.5. I'm trying to execute storm-starter ExclamationTopology.

storm jar ./storm/examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar org.apache.storm.starter.ExclamationTopology

[#5 · Accepted solution]

Leave storm/lib vanilla and package the higher versions of those libs with your topology jar using maven shade to relocate the necessary packages to avoid conflict. Here is an example from a storm topology that relocates guava (com.google.common).

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>${version.shade}</version>
    <configuration>
        <relocations>
            <relocation>
                <pattern>com.google.common</pattern>
                <shadedPattern>com.cisco.com.google.common</shadedPattern>
            </relocation>
        </relocations>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
                <artifactSet>
                    <excludes>
                        <exclude>org.datanucleus</exclude>
                    </excludes>
                </artifactSet>
                <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
                <transformers>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"
/>
                </transformers>
            </configuration>
        </execution>
    </executions>
</plugin>

[#6]

@Paul Hargis -- Thanks for your time and suggestions. I was able to solve the guava dependency issue. Now I'm getting another error due to external Kafka dependency. Can you guide me?

Running org.apache.storm.kafka.bolt.KafkaBoltTest Tests run: 16, Failures: 0, Errors: 16, Skipped: 0, Time elapsed: 9.338 sec <<< FAILURE! - in org.apache.storm.kafka.bolt.KafkaBoltTest executeWithBrokerDown(org.apache.storm.kafka.bolt.KafkaBoltTest) Time elapsed: 1.65 sec <<< ERROR!

[#7]

@keerthana gajarajakumar please post this as a new question on HCC

[#8]

@Ancil McBarnett -- Hey I made this a seperate question. Thanks!

[#9]

@Paul Hargis Follow up question. I have a topology that used both hBase and ElasticSearch 2.1. hBase depends on guava 1.3 and ElasticSearch 2.1 depends on guava 1.8. Can I include both versions of guava? I somehow need to tell maven to rewrite the Elastic byte code to refer to the relocated guava 1.8, but to NOT rewrite hBase byte code, so it continues to find guava 1.3.

Is this possible? How?

( It seems I could do this by breaking up my project into more pom files and isolating them that way, but that will break the more functional segmentation of my multi-pom project, so maybe there is a more direct way?)

[#10]

You can exclude certain sub dependencies with <exclude> tag, read the maven docs for that