Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDF 3.2 - jars not found in repo

9 REPLIES 9

Re: HDF 3.2 - jars not found in repo

@Gunar Mueller

It seems the above mentioned jars are of 3.1.1.0-35 and not of HDF 3.2.

Re: HDF 3.2 - jars not found in repo

New Contributor

Im using SAM 0.6.0.3.2.0.0-520 and during application deployment I can see, that SAM tries do reach these URLs.

Re: HDF 3.2 - jars not found in repo

@Gunar Mueller

Version numbers are extracted form *component.json files under /usr/hdf/current/streamline/ (eg: /usr/hdf/current/streamline/bootstrap/components/sinks/hive-sink-topology-component.json)

Do check if /usr/hdf/current/streamline/ is pointing to the correct HDF version. below is for 3.1.1.0-35

localohost$ ls -lrth /usr/hdf/current/streamline
lrwxrwxrwx. 1 root root 30 Mar 29 12:32 /usr/hdf/current/streamline -> /usr/hdf/3.1.1.0-35/streamline

Also check if hdf-select is pointing to correct version

localohost$ hdf-select  | grep streamline
streamline - 3.1.1.0-35

Re: HDF 3.2 - jars not found in repo

New Contributor

The version seems to be correct:

:~# ls -lrth /usr/hdf/current/streamline
lrwxrwxrwx 1 root root 31 Aug 17 11:25 /usr/hdf/current/streamline -> /usr/hdf/3.2.0.0-520/streamline

:~# hdf-select | grep streamline
streamline - 3.2.0.0-520

:/usr/hdf/current/streamline/bootstrap/components/sinks# cat kafka-sink-topology-component.json
{
"type": "SINK",
"name": "Kafka",
"subType": "KAFKA",
"streamingEngine": "STORM",
"builtin": true,
"fieldHintProviderClass": "com.hortonworks.streamline.streams.cluster.bundle.impl.KafkaSinkBundleHintProvider",
"transformationClass": "com.hortonworks.streamline.streams.layout.storm.KafkaBoltFluxComponent",
"mavenDeps": "org.apache.kafka:kafka-clients:0.10.2.1,org.apache.storm:storm-kafka-client:1.2.1.3.2.0.0-520^org.slf4j:slf4j-log4j12^log4j:log4j^org.apache.zookeeper:zookeeper^org.apache.kafka:kafka-clients",
"topologyComponentUISpecification": {
"fields": [

Re: HDF 3.2 - jars not found in repo

New Contributor

I checked all symbolic links and can not find a link pointing to version 3.1.1.0.

The streamline.log shows the following output:

ERROR [2018-08-24 13:14:16.009] [ForkJoinPool-4-worker-9] c.h.s.s.a.t.s.TopologyStates - Trying to kill any running instance of topology 'test'
ERROR [2018-08-24 13:14:16.026] [ForkJoinPool-4-worker-9] c.h.s.c.u.ParallelStreamUtil - Got exception while running async task
java.lang.RuntimeException: java.lang.Exception: Topology could not be deployed successfully: storm deploy command failed with Exception in thread "main" java.lang.RuntimeException: org.eclipse.aether.resolution.DependencyResolutionException: Failed to collect dependencies at org.apache.kafka:kafka-clients:jar:0.10.2.1 -> org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Failed to collect dependencies at org.apache.kafka:kafka-clients:jar:0.10.2.1 -> org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at org.apache.kafka:kafka-clients:jar:0.10.2.1 -> org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.apache.storm:storm-kafka-client:pom:1.1.1.3.1.1.0-35 from/to central (http://repo1.maven.org/maven2/): Connect to repo1.maven.org:80 timed out, Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.apache.storm:storm-kafka-client:pom:1.1.1.3.1.1.0-35 from/to central (http://repo1.maven.org/maven2/): Connect to repo1.maven.org:80 timed out, Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to repo1.maven.org:80 timed out
at com.hortonworks.streamline.common.util.ParallelStreamUtil.lambda$runAsync$0(ParallelStreamUtil.java:58)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.Exception: Topology could not be deployed successfully: storm deploy command failed with Exception in thread "main" java.lang.RuntimeException: org.eclipse.aether.resolution.DependencyResolutionException: Failed to collect dependencies at org.apache.kafka:kafka-clients:jar:0.10.2.1 -> org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Failed to collect dependencies at org.apache.kafka:kafka-clients:jar:0.10.2.1 -> org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at org.apache.kafka:kafka-clients:jar:0.10.2.1 -> org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.apache.storm:storm-kafka-client:jar:1.1.1.3.1.1.0-35, Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.apache.storm:storm-kafka-client:pom:1.1.1.3.1.1.0-35 from/to central (http://repo1.maven.org/maven2/): Connect to repo1.maven.org:80 timed out, Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.apache.storm:storm-kafka-client:pom:1.1.1.3.1.1.0-35 from/to central (http://repo1.maven.org/maven2/): Connect to repo1.maven.org:80 timed out, Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to repo1.maven.org:80 timed out
at com.hortonworks.streamline.streams.actions.storm.topology.StormTopologyActionsImpl.deploy(StormTopologyActionsImpl.java:288)
at com.hortonworks.streamline.streams.actions.topology.state.TopologyStates$5.deploy(TopologyStates.java:123)
at com.hortonworks.streamline.streams.actions.topology.state.TopologyContext.deploy(TopologyContext.java:87)
at com.hortonworks.streamline.streams.actions.topology.service.TopologyActionsService.lambda$deployTopology$1(TopologyActionsService.java:125)
at com.hortonworks.registries.storage.transaction.ManagedTransaction.lambda$executeConsumer$7(ManagedTransaction.java:165)
at com.hortonworks.registries.storage.transaction.ManagedTransaction.executeTransactionBlockInternal(ManagedTransaction.java:243)
at com.hortonworks.registries.storage.transaction.ManagedTransaction.executeConsumer(ManagedTransaction.java:164)
at com.hortonworks.streamline.streams.actions.topology.service.TopologyActionsService.deployTopology(TopologyActionsService.java:123)
at com.hortonworks.streamline.streams.service.TopologyActionResource.lambda$deploy$0(TopologyActionResource.java:161)
at com.hortonworks.streamline.common.util.ParallelStreamUtil.lambda$runAsync$0(ParallelStreamUtil.java:56)
... 6 common frames omitted

Re: HDF 3.2 - jars not found in repo

New Contributor

Similar problem here. I changed the Kafka-Version-numbers in the *component.json files to the Kafka version associated with HDF 3.2.0.0-520 (1.1.1) but it seems like SAM still tries to download an old Kafka-Version (0.10.2.1).

Re: HDF 3.2 - jars not found in repo

New Contributor

I am facing the same problem.

HDF is pointing to correct version.

localohost$ hdf-select  | grep streamline
streamline - 3.1.1.0-35

But, but app deployment is failing in SAM.

Re: HDF 3.2 - jars not found in repo

New Contributor

I have solved the problem with a very uncomfortable work-around: Downgrading the cluster to HDF-3.1.2. Since I have time-pressure, this was the only reliable way to solve the problem in time.

Re: HDF 3.2 - jars not found in repo

New Contributor

My solution was to setup a new one node cluster with HDF3.2, connect the envionment of the new streamline installation to the existing cluster, export SAM applications from existing SAM and import applications to new SAM.

Don't have an account?
Coming from Hortonworks? Activate your account here