Deployment failed after upgrade from hadoop-client 2.6.0-cdh5.13.0 to 3.0.0-cdh6.3.2

[#1]

Hello everyone.

we have a third party application an jee application which uses some cdh jar to interact with our cluster cdh5.16.
We have a new cdh6.3.2 cluster. I am responsible to adapt the jee application to the cdh6 cluster.

Sadly the hadoop-client 2.6.0-cdh5.13.0 is embedded in the ear after the  build. 

when i switch the version in the pom.xml from 2.6.0-cdh5.13.0 to 3.0.0-cdh6.3.2 i get troubles. the ear generated seems to miss dependencies, the wildfly can't restart.

here are the logs errors:

2020-02-18 16:17:31,250 INFO [org.jboss.weld.Event] (MSC service thread 1-2) WELD-000411: Observer method [BackedAnnotatedMethod] com.sun.jersey.server.impl.cdi.CDIExtension.processAnnotatedType(@Observes ProcessAnnotatedType<T>) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds.
2020-02-18 16:17:31,259 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.deployment.unit."ossa-fault-2.0.0.0-SNAPSHOT.ear".WeldStartService: org.jboss.msc.service.StartException in service jboss.deployment.unit."ossa-fault-2.0.0.0-SNAPSHOT.ear".WeldStartService: Failed to start service
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_242]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_242]
Caused by: org.jboss.weld.exceptions.DefinitionException: Exception List with 1 exceptions:
Exception 0 :
javax.enterprise.event.ObserverException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40)
at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78)
at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96)
at org.jboss.weld.injection.MethodInvocationStrategy$SpecialParamPlusBeanManagerStrategy.invoke(MethodInvocationStrategy.java:144)
at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:313)
at org.jboss.weld.event.ExtensionObserverMethodImpl.sendEvent(ExtensionObserverMethodImpl.java:125)
at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:291)
at org.jboss.weld.event.ObserverMethodImpl.notify(ObserverMethodImpl.java:269)
at org.jboss.weld.event.ObserverNotifier.notifySyncObservers(ObserverNotifier.java:302)
at org.jboss.weld.event.ObserverNotifier.notify(ObserverNotifier.java:291)
at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:160)
at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:154)
at org.jboss.weld.bootstrap.events.AbstractContainerEvent.fire(AbstractContainerEvent.java:53)
at org.jboss.weld.bootstrap.events.AbstractDefinitionContainerEvent.fire(AbstractDefinitionContainerEvent.java:44)
at org.jboss.weld.bootstrap.events.BeforeBeanDiscoveryImpl.fire(BeforeBeanDiscoveryImpl.java:45)
at org.jboss.weld.bootstrap.WeldStartup.startInitialization(WeldStartup.java:370)
at org.jboss.weld.bootstrap.WeldBootstrap.startInitialization(WeldBootstrap.java:76)
at org.jboss.as.weld.WeldStartService.start(WeldStartService.java:94)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: com/sun/jersey/server/impl/InitialContextHelper
at com.sun.jersey.server.impl.cdi.CDIExtension.initialize(CDIExtension.java:195)
at com.sun.jersey.server.impl.cdi.CDIExtension.beforeBeanDiscovery(CDIExtension.java:302)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88)
... 20 more
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.server.impl.InitialContextHelper from [Module "deployment.ossa-fault-2.0.0.0-SNAPSHOT.ear:main" from Service Module Loader]
at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363)
at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351)
at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93)
... 27 more

at org.jboss.weld.bootstrap.events.AbstractDefinitionContainerEvent.fire(AbstractDefinitionContainerEvent.java:46)
at org.jboss.weld.bootstrap.events.BeforeBeanDiscoveryImpl.fire(BeforeBeanDiscoveryImpl.java:45)
at org.jboss.weld.bootstrap.WeldStartup.startInitialization(WeldStartup.java:370)
at org.jboss.weld.bootstrap.WeldBootstrap.startInitialization(WeldBootstrap.java:76)
at org.jboss.as.weld.WeldStartService.start(WeldStartService.java:94)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.6.Final.jar:1.2.6.Final]
... 3 more

2020-02-18 16:17:31,267 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "ossa-fault-2.0.0.0-SNAPSHOT.ear")]) - failure description: {
"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"ossa-fault-2.0.0.0-SNAPSHOT.ear\".WeldStartService" => "org.jboss.msc.service.StartException in service jboss.deployment.unit.\"ossa-fault-2.0.0.0-SNAPSHOT.ear\".WeldStartService: Failed to start service
Caused by: org.jboss.weld.exceptions.DefinitionException: Exception List with 1 exceptions:
Exception 0 :
javax.enterprise.event.ObserverException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40)
at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78)
at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96)
at org.jboss.weld.injection.MethodInvocationStrategy$SpecialParamPlusBeanManagerStrategy.invoke(MethodInvocationStrategy.java:144)
at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:313)
at org.jboss.weld.event.ExtensionObserverMethodImpl.sendEvent(ExtensionObserverMethodImpl.java:125)
at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:291)
at org.jboss.weld.event.ObserverMethodImpl.notify(ObserverMethodImpl.java:269)
at org.jboss.weld.event.ObserverNotifier.notifySyncObservers(ObserverNotifier.java:302)
at org.jboss.weld.event.ObserverNotifier.notify(ObserverNotifier.java:291)
at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:160)
at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:154)
at org.jboss.weld.bootstrap.events.AbstractContainerEvent.fire(AbstractContainerEvent.java:53)
at org.jboss.weld.bootstrap.events.AbstractDefinitionContainerEvent.fire(AbstractDefinitionContainerEvent.java:44)
at org.jboss.weld.bootstrap.events.BeforeBeanDiscoveryImpl.fire(BeforeBeanDiscoveryImpl.java:45)
at org.jboss.weld.bootstrap.WeldStartup.startInitialization(WeldStartup.java:370)
at org.jboss.weld.bootstrap.WeldBootstrap.startInitialization(WeldBootstrap.java:76)
at org.jboss.as.weld.WeldStartService.start(WeldStartService.java:94)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: com/sun/jersey/server/impl/InitialContextHelper
at com.sun.jersey.server.impl.cdi.CDIExtension.initialize(CDIExtension.java:195)
at com.sun.jersey.server.impl.cdi.CDIExtension.beforeBeanDiscovery(CDIExtension.java:302)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88)
... 20 more
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.server.impl.InitialContextHelper from [Module \"deployment.ossa-fault-2.0.0.0-SNAPSHOT.ear:main\" from Service Module Loader]
at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363)
at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351)
at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93)
... 27 more
"},
"WFLYCTL0412: Required services that are not installed:" => ["jboss.deployment.unit.\"ossa-fault-2.0.0.0-SNAPSHOT.ear\".WeldStartService"],
"WFLYCTL0180: Services with missing/unavailable dependencies" => undefined
}
2020-02-18 16:17:31,318 INFO [org.jboss.as.server] (ServerService Thread Pool -- 34) WFLYSRV0010: Deployed "ossa-fault-2.0.0.0-SNAPSHOT.ear" (runtime-name : "ossa-fault-2.0.0.0-SNAPSHOT.ear")
2020-02-18 16:17:31,319 INFO [org.jboss.as.server] (ServerService Thread Pool -- 34) WFLYSRV0010: Deployed "ossa-server.ear" (runtime-name : "ossa-server.ear")
2020-02-18 16:17:31,320 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
WFLYCTL0186: Services which failed to start: service jboss.deployment.unit."ossa-fault-2.0.0.0-SNAPSHOT.ear".WeldStartService: org.jboss.msc.service.StartException in service jboss.deployment.unit."ossa-fault-2.0.0.0-SNAPSHOT.ear".WeldStartService: Failed to start service

Here is the content of the pom.xml

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>3.0.0-cdh6.3.2</version>
<!-- <version>2.6.0-cdh5.13.0</version> -->
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>3.0.0-cdh6.3.2</version>
<!-- <version>2.6.0-cdh5.13.0</version> -->
<exclusions>
<exclusion> <!-- declare the exclusion here -->
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-bundle</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-server</artifactId>
<version>1.1.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.kudu</groupId>
<artifactId>kudu-client</artifactId>
<version>1.5.0-cdh6.3.2</version>
<!-- <version>2.6.0-cdh5.13.0</version> -->
</dependency>
<dependency>
<groupId>commons-dbcp</groupId>
<artifactId>commons-dbcp</artifactId>
<version>1.4</version>
<scope>test</scope>
</dependency>

Did i miss to add or remove some dependencies after the upgrade ?

[#2]

Ok guys, i guess i have to ask my question with another form.

is It safe to use the hadoop-client 2.6.0-cdh5.13.0 against my cdh6.3.2 platform ?

If not, in your point of view, what dependencies i probably must exclude or pay attention to when upgrading the client version inside my pom.xml ?

thanks