I am getting InvalidClassException using storm on HDP 2.5

[#1]

Hi,

I am having HDP 2.5 and storm core version as storm-core-1.0.1.2.5.3.0-37.jar. My code was working with Local Cluster but its failing when using StormSubmitter.submitTopology. It gives me InvalidClassException. Below is complete stack.

One more observation that jar gives error while executing first time and run afterwards. I searched and implemented as below, but nothing worked. Please help me.

java.lang.RuntimeException: java.io.InvalidClassException: twitter4j.MediaEntityJSONImpl; local class incompatible: stream classdesc serialVersionUID = 1571961225214439778, local class serialVersionUID = 3609683338035442290
at org.apache.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:58) ~[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]

1. added serialVersionUID with value 1571961225214439778L, 3609683338035442290, 1L but nothing worked

2. Implemented Serializable, Externalizable in my POJOs

3. Since I am using storm-core-1.0.1.2.5.3.0-37.jar & HDP 2.5 so added maven dependency in pom with scope as provided which matches it, but did not worked.

4. Added no-arg constructors.

[#2 · Accepted solution]

@parag dharmadhikari

Looks like your class "twitter4j.MediaEntityJSONImpl" version is not compatible. So please check if you can get a different version of JAR that contains the "twitter4j.MediaEntityJSONImpl" class to see if it fixed the issue.

If you still face the issue then can you please share the list of JARs that your application is using? Better if you can share the whole "pom.xml" to see of the dependency is OK.

[#3]

thanks 🙂 It worked. I am using twitter4j 4.0.4 now.

[#4]

@Jay SenSharma

Thanks Jay for your response. I will try your suggestion. Mean while my POM is as below,

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>
<groupId>com.rolta</groupId>
<artifactId>stormtwitter</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.7</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.7</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.9.9</version>
</dependency>
<dependency>
<groupId>org.twitter4j</groupId>
<artifactId>twitter4j-core</artifactId>
<version>4.0.6</version>
</dependency>
<dependency>
<groupId>org.twitter4j</groupId>
<artifactId>twitter4j-stream</artifactId>
<version>4.0.6</version>
</dependency>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>1.0.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.opennlp</groupId>
<artifactId>opennlp-tools</artifactId>
<version>1.8.1</version>
</dependency>
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-solrj</artifactId>
<version>6.6.0</version>
</dependency>
<dependency>
<groupId>com.jolira</groupId>
<artifactId>onejar-maven-plugin</artifactId>
<version>1.4.4</version>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>org.slf4j:slf4j-log4j12:*</exclude>
<exclude>log4j:log4j:jar:</exclude>
<exclude>org.slf4j:slf4j-simple:jar</exclude>
<exclude>org.apache.storm:storm-core</exclude>
</excludes>
</artifactSet>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.rolta.storm.topology.Topology</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>

</plugins>
</build>

</project>