question Re: I am getting InvalidClassException using storm on HDP 2.5 in Archives of Support Questions (Read Only)

I am getting InvalidClassException using storm on HDP 2.5

parag_dharmadhi — Mon, 07 Aug 2017 18:13:22 GMT

Hi,

I am having HDP 2.5 and storm core version as storm-core-1.0.1.2.5.3.0-37.jar. My code was working with Local Cluster but its failing when using StormSubmitter.submitTopology. It gives me InvalidClassException. Below is complete stack.

One more observation that jar gives error while executing first time and run afterwards. I searched and implemented as below, but nothing worked. Please help me.

java.lang.RuntimeException: java.io.InvalidClassException: twitter4j.MediaEntityJSONImpl; local class incompatible: stream classdesc serialVersionUID = 1571961225214439778, local class serialVersionUID = 3609683338035442290
at org.apache.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:58) ~[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]

1. added serialVersionUID with value 1571961225214439778L, 3609683338035442290, 1L but nothing worked

2. Implemented Serializable, Externalizable in my POJOs

3. Since I am using storm-core-1.0.1.2.5.3.0-37.jar & HDP 2.5 so added maven dependency in pom with scope as provided which matches it, but did not worked.

4. Added no-arg constructors.

Re: I am getting InvalidClassException using storm on HDP 2.5

jsensharma — Mon, 07 Aug 2017 18:21:54 GMT

@parag dharmadhikari

Looks like your class "twitter4j.MediaEntityJSONImpl" version is not compatible. So please check if you can get a different version of JAR that contains the "twitter4j.MediaEntityJSONImpl" class to see if it fixed the issue.

If you still face the issue then can you please share the list of JARs that your application is using? Better if you can share the whole "pom.xml" to see of the dependency is OK.

Re: I am getting InvalidClassException using storm on HDP 2.5

parag_dharmadhi — Mon, 07 Aug 2017 18:57:55 GMT

@Jay SenSharma

Thanks Jay for your response. I will try your suggestion. Mean while my POM is as below,

<modelVersion>4.0.0</modelVersion>
<groupId>com.rolta</groupId>
<artifactId>stormtwitter</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.7</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.7</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.9.9</version>
</dependency>
<dependency>
<groupId>org.twitter4j</groupId>
<artifactId>twitter4j-core</artifactId>
<version>4.0.6</version>
</dependency>
<dependency>
<groupId>org.twitter4j</groupId>
<artifactId>twitter4j-stream</artifactId>
<version>4.0.6</version>
</dependency>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>1.0.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.opennlp</groupId>
<artifactId>opennlp-tools</artifactId>
<version>1.8.1</version>
</dependency>
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-solrj</artifactId>
<version>6.6.0</version>
</dependency>
<dependency>
<groupId>com.jolira</groupId>
<artifactId>onejar-maven-plugin</artifactId>
<version>1.4.4</version>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>org.slf4j:slf4j-log4j12:*</exclude>
<exclude>log4j:log4j:jar:</exclude>
<exclude>org.slf4j:slf4j-simple:jar</exclude>
<exclude>org.apache.storm:storm-core</exclude>
</excludes>
</artifactSet>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.rolta.storm.topology.Topology</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>

</plugins>
</build>

</project>

Re: I am getting InvalidClassException using storm on HDP 2.5

parag_dharmadhi — Tue, 08 Aug 2017 16:39:48 GMT

thanks 🙂 It worked. I am using twitter4j 4.0.4 now.