HDF 3.2 - jars not found in repo

[#1]

Hi,

I'm trying to deploy an appliction via SAM. The requested jars don't exist in repo:

http://repo.hortonworks.com/content/groups/public/org/apache/storm/storm-kafka-client/1.1.1.3.1.1.0-...

http://repo.hortonworks.com/content/groups/public/org/apache/storm/storm-hdfs/1.1.1.3.1.1.0-35/storm...

http://nexus-private.hortonworks.com/nexus/content/groups/public/org/apache/storm/storm-kafka-client...

http://nexus-private.hortonworks.com/nexus/content/groups/public/org/apache/storm/storm-hdfs/1.1.1.3...

[#2]

@Gunar Mueller

It seems the above mentioned jars are of 3.1.1.0-35 and not of HDF 3.2.

[#3]

Im using SAM 0.6.0.3.2.0.0-520 and during application deployment I can see, that SAM tries do reach these URLs.

[#4]

@Gunar Mueller

Version numbers are extracted form *component.json files under /usr/hdf/current/streamline/ (eg: /usr/hdf/current/streamline/bootstrap/components/sinks/hive-sink-topology-component.json)

Do check if /usr/hdf/current/streamline/ is pointing to the correct HDF version. below is for 3.1.1.0-35

localohost$ ls -lrth /usr/hdf/current/streamline
lrwxrwxrwx. 1 root root 30 Mar 29 12:32 /usr/hdf/current/streamline -> /usr/hdf/3.1.1.0-35/streamline

Also check if hdf-select is pointing to correct version

localohost$ hdf-select  | grep streamline
streamline - 3.1.1.0-35

[#5]

The version seems to be correct:

:~# ls -lrth /usr/hdf/current/streamline
lrwxrwxrwx 1 root root 31 Aug 17 11:25 /usr/hdf/current/streamline -> /usr/hdf/3.2.0.0-520/streamline

:~# hdf-select | grep streamline
streamline - 3.2.0.0-520

:/usr/hdf/current/streamline/bootstrap/components/sinks# cat kafka-sink-topology-component.json
{
"type": "SINK",
"name": "Kafka",
"subType": "KAFKA",
"streamingEngine": "STORM",
"builtin": true,
"fieldHintProviderClass": "com.hortonworks.streamline.streams.cluster.bundle.impl.KafkaSinkBundleHintProvider",
"transformationClass": "com.hortonworks.streamline.streams.layout.storm.KafkaBoltFluxComponent",
"mavenDeps": "org.apache.kafka:kafka-clients:0.10.2.1,org.apache.storm:storm-kafka-client:1.2.1.3.2.0.0-520^org.slf4j:slf4j-log4j12^log4j:log4j^org.apache.zookeeper:zookeeper^org.apache.kafka:kafka-clients",
"topologyComponentUISpecification": {
"fields": [

[#6]

I checked all symbolic links and can not find a link pointing to version 3.1.1.0.

The streamline.log shows the following output:

ERROR [2018-08-24 13:14:16.009] [ForkJoinPool-4-worker-9] c.h.s.s.a.t.s.TopologyStates - Trying to kill any running instance of topology 'test'
ERROR [2018-08-24 13:14:16.026] [ForkJoinPool-4-worker-9] c.h.s.c.u.ParallelStreamUtil - Got exception while running async task
java.lang.RuntimeException: java.lang.Exception: Topology could not be deployed successfully: storm deploy command failed with Exception in thread "main" java.lang.RuntimeException: org.eclipse.aether.resolution.DependencyResolutionException: Failed to collect dependencies at org.apache.kafka:kafka-clients:jar:0.10.2.1 -> org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Failed to collect dependencies at org.apache.kafka:kafka-clients:jar:0.10.2.1 -> org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at org.apache.kafka:kafka-clients:jar:0.10.2.1 -> org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.apache.storm:storm-kafka-client:pom:1.1.1.3.1.1.0-35 from/to central (http://repo1.maven.org/maven2/😞 Connect to repo1.maven.org:80 timed out, Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.apache.storm:storm-kafka-client:pom:1.1.1.3.1.1.0-35 from/to central (http://repo1.maven.org/maven2/😞 Connect to repo1.maven.org:80 timed out, Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to repo1.maven.org:80 timed out
at com.hortonworks.streamline.common.util.ParallelStreamUtil.lambda$runAsync$0(ParallelStreamUtil.java:58)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.Exception: Topology could not be deployed successfully: storm deploy command failed with Exception in thread "main" java.lang.RuntimeException: org.eclipse.aether.resolution.DependencyResolutionException: Failed to collect dependencies at org.apache.kafka:kafka-clients:jar:0.10.2.1 -> org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Failed to collect dependencies at org.apache.kafka:kafka-clients:jar:0.10.2.1 -> org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at org.apache.kafka:kafka-clients:jar:0.10.2.1 -> org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.apache.storm:storm-kafka-client:pom:1.1.1.3.1.1.0-35 from/to central (http://repo1.maven.org/maven2/😞 Connect to repo1.maven.org:80 timed out, Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.apache.storm:storm-kafka-client:pom:1.1.1.3.1.1.0-35 from/to central (http://repo1.maven.org/maven2/😞 Connect to repo1.maven.org:80 timed out, Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to repo1.maven.org:80 timed out
at com.hortonworks.streamline.streams.actions.storm.topology.StormTopologyActionsImpl.deploy(StormTopologyActionsImpl.java:288)
at com.hortonworks.streamline.streams.actions.topology.state.TopologyStates$5.deploy(TopologyStates.java:123)
at com.hortonworks.streamline.streams.actions.topology.state.TopologyContext.deploy(TopologyContext.java:87)
at com.hortonworks.streamline.streams.actions.topology.service.TopologyActionsService.lambda$deployTopology$1(TopologyActionsService.java:125)
at com.hortonworks.registries.storage.transaction.ManagedTransaction.lambda$executeConsumer$7(ManagedTransaction.java:165)
at com.hortonworks.registries.storage.transaction.ManagedTransaction.executeTransactionBlockInternal(ManagedTransaction.java:243)
at com.hortonworks.registries.storage.transaction.ManagedTransaction.executeConsumer(ManagedTransaction.java:164)
at com.hortonworks.streamline.streams.actions.topology.service.TopologyActionsService.deployTopology(TopologyActionsService.java:123)
at com.hortonworks.streamline.streams.service.TopologyActionResource.lambda$deploy$0(TopologyActionResource.java:161)
at com.hortonworks.streamline.common.util.ParallelStreamUtil.lambda$runAsync$0(ParallelStreamUtil.java:56)
... 6 common frames omitted

[#7]

Similar problem here. I changed the Kafka-Version-numbers in the *component.json files to the Kafka version associated with HDF 3.2.0.0-520 (1.1.1) but it seems like SAM still tries to download an old Kafka-Version (0.10.2.1).

[#8]

I am facing the same problem.

HDF is pointing to correct version.

localohost$ hdf-select  | grep streamline
streamline - 3.1.1.0-35

But, but app deployment is failing in SAM.

[#9]

I have solved the problem with a very uncomfortable work-around: Downgrading the cluster to HDF-3.1.2. Since I have time-pressure, this was the only reliable way to solve the problem in time.

[#10]

My solution was to setup a new one node cluster with HDF3.2, connect the envionment of the new streamline installation to the existing cluster, export SAM applications from existing SAM and import applications to new SAM.