Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

STORM : How do I fix the google.guava dependency while running mvn clean install -DskipTests=true ?

avatar
Expert Contributor

I'm getting this error :

The POM for com.google.guava:guava-testlib:jar:16.0.1 is invalid, transitive dependencies (if any) will not be available: 1 problem was encountered while building the effective model for com.google.guava:guava-testlib:[unknown-version]

1 ACCEPTED SOLUTION

avatar

Leave storm/lib vanilla and package the higher versions of those libs with your topology jar using maven shade to relocate the necessary packages to avoid conflict. Here is an example from a storm topology that relocates guava (com.google.common).

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>${version.shade}</version>
    <configuration>
        <relocations>
            <relocation>
                <pattern>com.google.common</pattern>
                <shadedPattern>com.cisco.com.google.common</shadedPattern>
            </relocation>
        </relocations>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
                <artifactSet>
                    <excludes>
                        <exclude>org.datanucleus</exclude>
                    </excludes>
                </artifactSet>
                <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
                <transformers>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"
/>
                </transformers>
            </configuration>
        </execution>
    </executions>
</plugin>


View solution in original post

9 REPLIES 9

avatar
Expert Contributor
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project storm-elasticsearch: Compilation failure: Compilation failure:
[ERROR] /root/storm/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/common/EsConfigTest.java:[23,33] package com.google.common.collect does not exist
[ERROR] /root/storm/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/common/EsConfigTest.java:[23,33] package com.google.common.collect does not exist
[ERROR] /root/storm/external/storm-elasticsearch/src/test/java/org/apache/storm/elasticsearch/common/EsConfigTest.java:[57,16] cannot find symbol
[ERROR] symbol:   variable ImmutableMap
[ERROR] location: class org.apache.storm.elasticsearch.common.EsConfigTest
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project storm-elasticsearch: Compilation failure
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: Compilation failure
        at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:858)
        at org.apache.maven.plugin.compiler.TestCompilerMojo.execute(TestCompilerMojo.java:152)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
        ... 19 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :storm-elasticsearch

avatar
Master Mentor
@keerthana gajarajakumar

what version of maven do you have and please explain what you're doing to reproduce.

avatar
Expert Contributor

Maven version is 3.2.5. I'm trying to execute storm-starter ExclamationTopology.

storm jar ./storm/examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar org.apache.storm.starter.ExclamationTopology

avatar

Leave storm/lib vanilla and package the higher versions of those libs with your topology jar using maven shade to relocate the necessary packages to avoid conflict. Here is an example from a storm topology that relocates guava (com.google.common).

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>${version.shade}</version>
    <configuration>
        <relocations>
            <relocation>
                <pattern>com.google.common</pattern>
                <shadedPattern>com.cisco.com.google.common</shadedPattern>
            </relocation>
        </relocations>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
                <artifactSet>
                    <excludes>
                        <exclude>org.datanucleus</exclude>
                    </excludes>
                </artifactSet>
                <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
                <transformers>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"
/>
                </transformers>
            </configuration>
        </execution>
    </executions>
</plugin>


avatar
Expert Contributor

@Paul Hargis -- Thanks for your time and suggestions. I was able to solve the guava dependency issue. Now I'm getting another error due to external Kafka dependency. Can you guide me?

Running org.apache.storm.kafka.bolt.KafkaBoltTest Tests run: 16, Failures: 0, Errors: 16, Skipped: 0, Time elapsed: 9.338 sec <<< FAILURE! - in org.apache.storm.kafka.bolt.KafkaBoltTest executeWithBrokerDown(org.apache.storm.kafka.bolt.KafkaBoltTest) Time elapsed: 1.65 sec <<< ERROR!

avatar

@keerthana gajarajakumar please post this as a new question on HCC

avatar
Expert Contributor

@Ancil McBarnett -- Hey I made this a seperate question. Thanks!

avatar
New Contributor

@Paul Hargis Follow up question. I have a topology that used both hBase and ElasticSearch 2.1. hBase depends on guava 1.3 and ElasticSearch 2.1 depends on guava 1.8. Can I include both versions of guava? I somehow need to tell maven to rewrite the Elastic byte code to refer to the relocated guava 1.8, but to NOT rewrite hBase byte code, so it continues to find guava 1.3.

Is this possible? How?

( It seems I could do this by breaking up my project into more pom files and isolating them that way, but that will break the more functional segmentation of my multi-pom project, so maybe there is a more direct way?)

avatar
Master Mentor

You can exclude certain sub dependencies with <exclude> tag, read the maven docs for that